Reading TIFF larger than 2GB

[yewunudt]

How to read a TIFF file larger than 2GB? In file geotrellis/spark/src/main/scala/geotrellis/spark/io/hadoop/HdfsUtils.scala: 165, it limits the length of a file to Int.MaxValue.toLong. The "Cannot read path $path because it's too big..." error will be reported if trying to read a large TIFF file.

[fosskers]

Hi there, there are two approaches to the "big GeoTIFF input problem" with GeoTrellis:

1. Cut up your large GeoTIFF ahead of time with a tool like gdal_retile.py
2. Use our recently-merged windowed GeoTIFF reading feature, which lets you read in a small section of a GeoTIFF, if you have an Extent that represents some subsection of it.

We haven't yet taken that new feature and implemented a kind of:

readBigGeoTiff[K]: GeoTiff => TileLayerRDD[K]

but it's coming soon.

[lossyrob]

The problem here is that we read files off of HDFS as an Array[Byte], which can't have more bytes than integers. In order to read large GeoTiffs (that aren't in BigTiff format, so < 4G), we'll have to change some things around in how we read off of HDFS and also implement the streaming reads that we have implemented for S3, for HDFS.

[jbouffard]

@yewunudt As of right now, you can read in files that are 4 GB or less either locally or through S3 via a LocalBytesStreamer and a S3BytesStreamer, respectively. Here is a quick overview on how to use S3BytesStreamer. The LocalBytesStreamer acts the same way. As for reading files from Hdfs, that is something that we are currently working on.

When a GeoTiff file is larger than 4 GB, then it's referred to as a BigTiff. These files actually have a different layout than a normal GeoTiff. We currently can't read BigTiffs from anywhere, but there's a PR in progress that'll allow to do so soon link.

[jbouffard]

BigTiff reading is now supported, there should be no issue ingesting GeoTiffs at any size.