TIFF: Consider Float32 as the data type

[mojodna]

GeoTIFFs are currently Int16, which means that the limit on vertical resolution is 1m. Higher resolution datasets (i.e. LiDAR) effectively lose resolution when converted to Int16. (NED is distributed as Float32, so this may already be occurring.)

[zerebubuth]

I did a highly unscientific single test case, but I think it's fairly representative of the whole; take NED13 n32w090, which is native Float32 data type:

• 359MB - Original .img compressed in a .zip file from USGS.
• 447MB - Uncompressed .img
• 266MB - GeoTiff as Float32 with COMPRESS=DEFLATE, PREDICTOR=3
• 28MB - GeoTIFF as Int16 with COMPRESS=LZW, PREDICTOR=2

The reduction in file size by almost an order of magnitude is a benefit that has to be weighed against the improvement in precision allowed by Float32. Just out of interest I tried:

• Using feet instead of meters, if we can accept some clipping in the deepest parts of the ocean: 47MB
• Using cm instead of meters, and Int32 data type to avoid clipping: 182MB

Of course, the files will be larger, as we're retaining more of the information from the original. Sub-meter information can be very useful, for example in flat areas to avoid quantization artefacts appearing as false ridges, and many other uses. But its presence in the file comes at a cost for users who don't need that level of precision. It's not clear to me where the balance point lies.

Side note

Although it should be noted that not all of the data in the sub-meter range is information. For example, in tile 13/1326/3123 which, in the "terrarium" format, retains sub-meter information down to 1/256th of a meter in the blue band:

The "starburst" pattern is clearly an artefact, perhaps from merging multiple DEMs around a reference point. So although there's precision in NED, the accuracy should be carefully evaluated. LIDAR, likewise, has the capability of being incredibly precise, but care should be taken to avoid assumptions about its accuracy.

[zerebubuth]

@nvkelso, what do you think about the precision / file size trade-off? We could, of course, make a separate format for Int16 and Float32 GeoTiffs?

[nvkelso]

What about generating the Int16 files for most zooms, but switching to Float32 at zoom 14/15+?

[mojodna]

I suspect that that will confuse the GDAL WMS driver, since it treats the whole thing as a single source with overviews.

[flitzi]

No, I tried, GDAL is fine with specifying a pixel type that is different from the data pixel type.
This works:

<GDAL_WMS> <Service name="TMS"> <ServerUrl>https://s3.amazonaws.com/elevation-tiles-prod/geotiff/${z}/${x}/${y}.tif</ServerUrl> </Service> <DataWindow> <UpperLeftX>-20037508.34</UpperLeftX> <UpperLeftY>20037508.34</UpperLeftY> <LowerRightX>20037508.34</LowerRightX> <LowerRightY>-20037508.34</LowerRightY> <TileLevel>14</TileLevel> <TileCountX>1</TileCountX> <TileCountY>1</TileCountY> <YOrigin>top</YOrigin> </DataWindow> <Projection>PROJCS["WGS_1984_Web_Mercator", GEOGCS["GCS_WGS_1984_Major_Auxiliary_Sphere", DATUM["WGS_1984_Major_Auxiliary_Sphere", SPHEROID["WGS_1984_Major_Auxiliary_Sphere",6378137.0,0.0]], PRIMEM["Greenwich",0.0], UNIT["Degree",0.0174532925199433]], PROJECTION["Mercator_1SP"], PARAMETER["False_Easting",0.0], PARAMETER["False_Northing",0.0], PARAMETER["Central_Meridian",0.0], PARAMETER["latitude_of_origin",0.0], UNIT["Meter",1.0], EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs"], AUTHORITY["EPSG","102113"]]</Projection> <BlockSizeX>512</BlockSizeX> <BlockSizeY>512</BlockSizeY> <BandsCount>1</BandsCount> <DataType>Float32</DataType> <UnsafeSSL>true</UnsafeSSL> </GDAL_WMS>

Regarding compression: Rounding the elevation values to one decimal for zoom 14, maybe 2 for zoom 15+ will decrease size quite a bit. I think this would be the better solution than switching to cm and int32.

[mojodna]

@flitzi different pixel types at varying zooms? That's fantastic.