Roger cleanup & counts

[max-zilla]

@jdmaloney the ROGER filesystem prep for today has 2 parts:

1. delete old /ua-mac/Level_1 test+deprecated+irrelevant data

EnvironmentLogger
demosaic
flir2tif
flirIrCamera
fullfield_OLD
hyperspectral
hyperspectral_manualcheck
hyperspectral_swir
scanner3DTop_heightmap
scanner3DTop_manualcheck
scanner3DTop_plant_height
stereoTop_canopyCover
stereoTop_mosaics
stereoTop_preview

2. generate summary counts for primary raw_data/Level_1 stuff

raw_data/
  EnvironmentLogger  (no timestamp directory)
  SWIR
  VNIR
  flirIrCamera
  irrigation  (no timestamp directory)
  lightning  (no timestamp directory)
  scanner3DTop
  stereoTop
  weather  (no timestamp directory)
Level_1/
  envlog_netcdf  (no timestamp directory)
  fullfield  (no timestamp directory)
  ir_geotiff
  laser3d_heightmap
  laser3d_ply2las
  rgb_geotiff
  scanner3DTop
  vnir_netcdf

...we can do the other stuff after, but this is key stuff I think.

@dlebauer @craig-willis @jterstriep FYI if there are any changes you want in the above.

[jdmaloney]

Deletes have been completed, running summary counts now

[jdmaloney]

Report given to David

[dlebauer]

@jdmaloney thanks - could you please

1. please check in the script to the scripts/ directory of https://github.com/terraref/reference-data
2. change it to be comma delimited (I can do this on the current copy using sed)
3. change it so that it writes to /gpfs/largeblockFS/projects/arpae/terraref/users/dlebauer?

[jdmaloney]

@dlebauer
[1-3] Sure thing

Is there a frequency this is to be run at in an automated fashion? It's 2-3 hours to run at this point, I'm not multithreading it yet though, if the frequency is more than once a week, I'll work on making it faster.

[jdmaloney]

2 and 3 are done; there is a Pull Request out there for 1

[dlebauer]

I think that for the short term, I can run it on demand (which should be < 1/week) ... in the longer term we should replace this with queries to one of the databases ... and eventually a faceted search interface

Thanks!