RasterSource fails if selected bands are of two different resolutions.

[metasim]

Test:

val catalog = spark.read.l8Catalog.load()
val df = spark.read.raster
  .fromCatalog(catalog, "B1", "B8")
  .load()

Error:

19/07/01 09:28:00 ERROR RasterSourceToRasterRefs: Error fetching data for one of: GDALRasterSource(https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/003/048/LC08_L1TP_003048_20180805_20180814_01_T1/LC08_L1TP_003048_20180805_20180814_01_T1_B1.TIF), GDALRasterSource(https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/003/048/LC08_L1TP_003048_20180805_20180814_01_T1/LC08_L1TP_003048_20180805_20180814_01_T1_B8.TIF)
java.lang.IllegalArgumentException: transpose requires all collections have the same size
	at scala.collection.generic.GenericTraversableTemplate$class.fail$1(GenericTraversableTemplate.scala:213)
	at scala.collection.generic.GenericTraversableTemplate$$anonfun$transpose$1$$anonfun$apply$1.apply(GenericTraversableTemplate.scala:220)
	at scala.collection.generic.GenericTraversableTemplate$$anonfun$transpose$1$$anonfun$apply$1.apply(GenericTraversableTemplate.scala:219)
	at scala.collection.immutable.Stream.foreach(Stream.scala:594)
	at scala.collection.generic.GenericTraversableTemplate$$anonfun$transpose$1.apply(GenericTraversableTemplate.scala:219)
	at scala.collection.generic.GenericTraversableTemplate$$anonfun$transpose$1.apply(GenericTraversableTemplate.scala:217)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at scala.collection.generic.GenericTraversableTemplate$class.transpose(GenericTraversableTemplate.scala:217)
	at scala.collection.AbstractTraversable.transpose(Traversable.scala:104)
	at org.locationtech.rasterframes.expressions.transformers.RasterSourceToRasterRefs.eval(RasterSourceToRasterRefs.scala:77)

[metasim]

@vpipkt Need to figure out what's the appropriate way of handling this B1 is 30m, B8 is 15m.

[vpipkt]

Excellent but yikes. I think desired result here may vary by use case. Need to think that through...

[vpipkt]

Hmm i just duplicated this #326

But I also thought about some solutions.

[vpipkt]

From #326 dupe, some thoughts on how to address this.

A current work around would be to make individual catalogs for each resolution, making careful use of the tileDimensions param, then try to join on extents.

A proposed solution:

In the RasterSource reader, use the raster with the finest resolution to set the gridding and extents to read for all the other columns. This will result in the same number of rows generated from each catalog row.

A further nice to have may be a way for the user to express if / how they want resampling so all tile columns are at the same resolution. A sensible enough default might be for nearest neighbor upsampling to the finest resolution column.

[vpipkt]

A further point related to this.

If there is a value in one of the catalogColumns that is null the same error occurs, instead of just returning a null value for the projected raster... ?

[metasim]

In the RasterSource reader, use the raster with the finest resolution to set the gridding and extents to read for all the other columns. This will result in the same number of rows generated from each catalog row.

👍

[vpipkt]

@JenniferYingyiWu2020 i believe your comment is not relevant to this particular issue. Please take a look at this response