Add CReSIS Radar Depth Sounder (RDS) L2 data 2009-2017 #21
Assignees
Labels
Milestone
Comments
weiji14
added a commit
that referenced
this issue
Oct 4, 2018
Finally got to adding the bulky CReSIS Radar Depth Sounder (RDS) data from the University of Kansas for #21, bumping up training tile count to 2111! The csv point data is not exactly 'high' resolution, and I had to resort to using an IDW interpolation window_size of 2 in the pdal pipeline instead of 0 as in be0b77c. Three separate geotiffs groupings (Basler, DC8, TO) were used, with one (P3) only downloaded but not processed as the plane didn't fly in a grid pattern. Will use P3 for cross validation further down the track. Also refactored tiling function from the slow for-loop into a vectorized linear algebra implementation via scikit-image and numpy. This was cleaner and much faster than using a spatial index or parallel asyncio (which I've tried). Seconds instead of hours! This tiling code also has a more elegant UNIX philosophy vibe now. TODO: Move download links to a separate file, and have hash checksums too? Plus think hard about having unit tests in place.
weiji14
added a commit
that referenced
this issue
Oct 5, 2018
Rookie mistake, turns out the 'bottom' field in the CReSIS RDS csv data isn't actually the WGS84 elevation, but the range from the sensor. Used a python pdal filter to do `ELEVATION-BOTTOM` to get the actual bottom elevation as referenced to the WGS84 Ellipsoid. Also added a YUML diagram to visualize the pipeline. Part of #21. As an aside, may no longer need to use some fancy spline interpolation scheme to fix the crossover errors I found after 061242a. Was thinking of using Generic Mapping Tools (GMT) or some PDAL filter like 'median absolute deviation' to remove outliers.
4 tasks
weiji14
added a commit
that referenced
this issue
Nov 19, 2018
New CRESIS csv file was uploaded a few days ago, and the unit test caught it! Good ol' unintended consequences of unit testing :D Fix our unit test by choosing another blank file, and add this new dataset to our data_list for good measure. Relates to #21.
weiji14
added a commit
that referenced
this issue
Nov 19, 2018
New CRESIS csv file was uploaded a few days ago, and the unit test caught it! Good ol' unintended consequences of unit testing :D Fix our unit test by choosing another blank file, and add this new dataset to our data_list for good measure. Relates to #21.
weiji14
added a commit
that referenced
this issue
Nov 19, 2018
New CRESIS csv file was uploaded a few days ago, and the unit test caught it! Good ol' unintended consequences of unit testing :D Fix our unit test by choosing another blank file, and add this new dataset to our data_list for good measure. Relates to #21.
weiji14
added a commit
that referenced
this issue
Nov 19, 2018
New CRESIS csv file was uploaded a few days ago, and the unit test caught it! Good ol' unintended consequences of unit testing :D Fix our unit test by choosing another blank file, and add this new dataset to our data_list for good measure. Relates to #21.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add in Radar Depth Sounder (RDS) data from the Center for
the Remote Sensing of Ice Sheets (CReSIS) team based at the University of Kansas and other places (see here).
Acknowledgements:
We acknowledge the use of data and/or data products from CReSIS generated with support from the University of Kansas, NASA Operation IceBridge grant NNX16AH54G, and NSF grant ACI-1443054.
See also:
TODO:
The text was updated successfully, but these errors were encountered: