🍱 Add CReSIS RDS data ⚡ Fast vectorized tiling

Finally got to adding the bulky CReSIS Radar Depth Sounder (RDS) data from the University of Kansas for #21, bumping up training tile count to 2111! The csv point data is not exactly 'high' resolution, and I had to resort to using an IDW interpolation window_size of 2 in the pdal pipeline instead of 0 as in be0b77c. Three separate geotiffs groupings (Basler, DC8, TO) were used, with one (P3) only downloaded but not processed as the plane didn't fly in a grid pattern. Will use P3 for cross validation further down the track. Also refactored tiling function from the slow for-loop into a vectorized linear algebra implementation via scikit-image and numpy. This was cleaner and much faster than using a spatial index or parallel asyncio (which I've tried). Seconds instead of hours! This tiling code also has a more elegant UNIX philosophy vibe now. TODO: Move download links to a separate file, and have hash checksums too? Plus think hard about having unit tests in place.
weiji14 · Oct 4, 2018 · 061242a · 061242a
1 parent 329511a
commit 061242a
Show file tree

Hide file tree

Showing 5 changed files with 226 additions and 75 deletions.
diff --git a/data_prep.ipynb b/data_prep.ipynb
diff --git a/highres/201x_Antarctica_Basler.json b/highres/201x_Antarctica_Basler.json
@@ -0,0 +1,25 @@
+{
+    "pipeline":[
+        {
+          "type":"readers.text",
+          "filename":"201?_Antarctica_Basler.csv",
+          "separator":",",
+          "skip":1,
+          "header":"Y,X,TIME,THICK,ELEVATION,FRAME,SURFACE,BOTTOM,QUALITY"
+        },
+        {
+          "type":"filters.reprojection",
+          "in_srs":"EPSG:4326",
+          "out_srs":"EPSG:3031"
+        },
+        {
+          "type":"writers.gdal",
+          "filename":"201x_Antarctica_Basler.tif",
+          "resolution": 250,
+          "data_type": "float",
+          "dimension": "BOTTOM",
+          "output_type": "idw",
+          "window_size": 2
+        }
+    ]
+}
diff --git a/highres/20xx_Antarctica_DC8.json b/highres/20xx_Antarctica_DC8.json
@@ -0,0 +1,25 @@
+{
+    "pipeline":[
+        {
+          "type":"readers.text",
+          "filename":"20??_Antarctica_DC8.csv",
+          "separator":",",
+          "skip":1,
+          "header":"Y,X,TIME,THICK,ELEVATION,FRAME,SURFACE,BOTTOM,QUALITY"
+        },
+        {
+          "type":"filters.reprojection",
+          "in_srs":"EPSG:4326",
+          "out_srs":"EPSG:3031"
+        },
+        {
+          "type":"writers.gdal",
+          "filename":"20xx_Antarctica_DC8.tif",
+          "resolution": 250,
+          "data_type": "float",
+          "dimension": "BOTTOM",
+          "output_type": "idw",
+          "window_size": 2
+        }
+    ]
+}
diff --git a/highres/20xx_Antarctica_TO.json b/highres/20xx_Antarctica_TO.json
@@ -0,0 +1,25 @@
+{
+    "pipeline":[
+        {
+          "type":"readers.text",
+          "filename":"20??_Antarctica_TO*.csv",
+          "separator":",",
+          "skip":1,
+          "header":"Y,X,TIME,THICK,ELEVATION,FRAME,SURFACE,BOTTOM,QUALITY"
+        },
+        {
+          "type":"filters.reprojection",
+          "in_srs":"EPSG:4326",
+          "out_srs":"EPSG:3031"
+        },
+        {
+          "type":"writers.gdal",
+          "filename":"20xx_Antarctica_TO.tif",
+          "resolution": 250,
+          "data_type": "float",
+          "dimension": "BOTTOM",
+          "output_type": "idw",
+          "window_size": 2
+        }
+    ]
+}
diff --git a/highres/README.md b/highres/README.md
@@ -1,14 +1,18 @@
 # High Resolution Antarctic DEMs
 
-| Filename            | Location/Name           | Resolution | Literature Citation  | Data Citation/Link |
-|:--------------------|:----------------------- | ----------:|:--------------------:|:------------------:|
-| 2010tr.txt          | Pine Island Glacier     |        50m | [Bingham2018]        | [PIG2010-2011]     |
-| bed_WGS84_grid.txt  | Rutford Ice Stream      |     ~125m? | [King2016]           | [Rutford2007-2009] |
-| istar??.txt * 6     | Pine Island Glacier     |        50m | [Bingham2018]        | [PIG2013-2014]     |
+| Filename                   | Location/Name           | Resolution | Literature Citation  |      Data Citation/Link      |
+|:---------------------------|:----------------------- | ----------:|:--------------------:|:----------------------------:|
+| 20??_Antarctica_*.csv * 12 | Mainly West Antarctica  |         ?m | [Shi2010]            | [IRMCR2v1@NSIDC][RDS@CReSIS] |
+| 2010tr.txt                 | Pine Island Glacier     |        50m | [Bingham2018]        | [PIG2010-2011]               |
+| bed_WGS84_grid.txt         | Rutford Ice Stream      |     ~125m? | [King2016]           | [Rutford2007-2009]           |
+| istar??.txt * 6            | Pine Island Glacier     |        50m | [Bingham2018]        | [PIG2013-2014]               |
 
+[RDS@CReSIS]: https://data.cresis.ku.edu/data/rds/
+[IRMCR2v1@NSIDC]: https://doi.org/10.5067/GDQ0CUCVTE2Q
 [PIG2010-2011]: https://istar-gis-features.data.bas.ac.uk/Legacy/delores_2010_2011_DEM/
 [PIG2013-2014]: https://istar-gis-features.data.bas.ac.uk/glacial/delores_dems/
 [Rutford2007-2009]: https://doi.org/10.5285/54757cbe-0b13-4385-8b31-4dfaa1dab55e
 
 [Bingham2018]: https://doi.org/10.1038/s41467-017-01597-y
 [King2016]: https://doi.org/10.5194/essd-8-151-2016
+[Shi2010]: https://doi.org/10.1109/IGARSS.2010.5649518