Skip to content

[GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL - #3312

Open
james-willis wants to merge 3 commits into
apache:masterfrom
james-willis:fix/rs-setbandnodatavalue-null-clear
Open

[GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL#3312
james-willis wants to merge 3 commits into
apache:masterfrom
james-willis:fix/rs-setbandnodatavalue-null-clear

Conversation

@james-willis

@james-willis james-willis commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Did you read the Contributor Guide?

Is this PR related to a ticket?

What changes were proposed in this PR?

The docs and the common Java implementation say a null noDataValue removes the band's no-data value, but from Spark SQL that path was unreachable: the catalyst expression wrapped RasterBandEditors.setBandNoDataValue with inferrableFunction2/3/4, which null-propagate as soon as any argument is null, so RS_SetBandNoDataValue(raster, 1, NULL) returned a NULL raster instead. Fixing this matches PostGIS ST_SetBandNoDataValue NULL-clears semantics and the behavior SedonaDB adopted in apache/sedona-db#1158.

  • Added InferrableFunction.allowRightNull3, an arity-3 counterpart of the existing allowRightNull pattern (allows the third argument to be null; still null-propagates when the raster or band index is null). It is a separate name rather than an overload because overloading allowRightNull breaks eta-expansion of the overloaded functions passed to it at existing call sites.
  • Wired the 2- and 3-argument overloads of RS_SetBandNoDataValue through allowRightNull/allowRightNull3. The 4-argument replace overload still null-propagates, since replacing pixels with a null no-data value is meaningless; the docs now note that clearing requires the 2-/3-argument form.
  • Noted in RS_SetBandNoDataValue.md that the replace variant requires a non-null noDataValue; the existing intro already documents NULL-clears.

How was this patch tested?

Strengthened the existing RS_SetBandNoDataValue test in rasteralgebraTest.scala so it distinguishes clearing from null-propagation: it sets a no-data value, clears it with NULL (both the 2- and 3-argument forms), and asserts the resulting raster is non-null while RS_BandNoDataValue on it is null; it also asserts a NULL raster input still yields NULL. Before this change the previous test passed under both semantics because RS_BandNoDataValue returns null for a null raster too. Verified the new assertions fail without the expression change and pass with it (rasteralgebraTest, Spark 3.5 / Scala 2.12 / JDK 17).

Did this PR include necessary documentation updates?

  • Yes, I have updated the documentation.

Review follow-up

The second commit addresses the review findings:

  • The null path now consults the target band's no-data value (rasterNoData) instead of getBandNoDataValue(raster), which checked band 1 and made clearing band N a no-op whenever band 1 had no no-data value. Covered by new asymmetric multi-band tests at both the common and SQL levels.
  • Clearing drops the GC_NODATA sentinel from the coverage properties and copies the pixels into a property-free image — the sentinel also rides on the rendered image itself through serialization, so cloning the image would let it survive a shuffle. Covered by a common-level test that asserts the property is gone from the coverage, the image, and a serde round trip.
  • While reproducing the GeoTIFF round trip, it turned out GeoTiffWriter writes a default GDAL_NODATA of 0 for any coverage without a no-data value — an unmodified raster read from a plain GeoTIFF (no nodata) already came back with 0.0 after RS_AsGeoTiffRS_FromGeoTiff, independent of this PR. RS_AsGeoTiff now sets GeoTiffFormat.WRITE_NODATA to whether the raster actually has a no-data value, which fixes both the cleared case and that pre-existing resurrection. Covered by the new round-trip test on raster_with_no_data/test5.tiff.

Second review round

  • Non-zero image origin. Confirmed: copyData(null) keeps the source origin and BufferedImage rejects a raster whose minX/minY are non-zero. Clearing is back on a RenderedImage path via a new PropertyMaskedRenderedImage, which delegates everything to the source image but reports GC_NODATA as Image.UndefinedProperty. (RenderedImageAdapter cannot do this — it declares getProperty final and answers from the source, so removeProperty does not mask an inherited value.) A translated-raster regression test covers it.
  • Streaming cost. The same change removes the pixel copy entirely, so a lazily decoded GeoTIFF stays undecoded; masking is a wrapper allocation.
  • Coverage vs image property. Worth recording: masking the image alone is not sufficient, because GC_NODATA is also a key in the coverage property map; filtering the map alone is not sufficient either, because GridCoverage2D.getProperty falls through to the image. Both are now handled, and the map is filtered on a defensive copy so the input raster is no longer mutated (the previous revision mutated it). The test asserts GeoTools' own CoverageUtilities.getNoDataProperty reports null for a cleared raster and that the input still reports its original value.
  • WRITE_NODATA opt-out. Fixed: the flag is only forced to false when no band has a no-data value, so an explicit -Dgeotiff.writenodata=false is left alone.

Tests: common 1339 passed, rasteralgebraTest 168 passed, rasterIOTest 21 passed.

…e in Spark SQL

The docs and the common Java implementation say a null noDataValue removes
the band's no-data value, but the catalyst expression null-propagated on any
null argument, so RS_SetBandNoDataValue(raster, 1, NULL) returned a null
raster and the documented clear path was unreachable from SQL. Route the
2- and 3-arg overloads through the allowRightNull pattern (adding an arity-3
variant) so a null noDataValue reaches the Java implementation. The 4-arg
replace overload still null-propagates since replacing pixels with a null
no-data value is meaningless.
@james-willis
james-willis force-pushed the fix/rs-setbandnodatavalue-null-clear branch from 3768356 to ab5f6a3 Compare September 2, 2026 17:26
@james-willis
james-willis marked this pull request as ready for review September 2, 2026 17:26
…hrough GeoTIFF round trips

The null path consulted band 1's no-data value instead of the target
band's, so clearing band N was a no-op whenever band 1 had none.

Clearing now also drops the GC_NODATA sentinel from the coverage
properties and copies the pixels into a property-free image, since the
sentinel rides on the rendered image itself through serialization.

RS_AsGeoTiff only writes GDAL_NODATA when the raster actually has a
no-data value: GeoTiffWriter otherwise writes a default of 0, which
resurrected cleared (and never-set) no-data values on re-read.
Comment thread common/src/main/java/org/apache/sedona/common/raster/RasterBandEditors.java Outdated
Comment thread common/src/main/java/org/apache/sedona/common/raster/RasterBandEditors.java Outdated
Comment thread common/src/main/java/org/apache/sedona/common/raster/RasterOutputs.java Outdated
…the GeoTIFF nodata opt-out

Clearing no longer copies pixels. RenderedImageAdapter cannot mask an
inherited property (it declares getProperty final and answers from the
source), so PropertyMaskedRenderedImage wraps the image and hides
GC_NODATA while delegating everything else. That keeps a lazily decoded
raster undecoded and preserves a non-zero image origin, which the
previous copyData path lost -- BufferedImage rejects a raster whose
minX/minY are not zero.

GridCoverage2D.getProperty falls through to the rendered image, so the
sentinel is dropped from the coverage property map as well, on a
defensive copy so the input raster is left untouched. GeoTools' own
CoverageUtilities.getNoDataProperty now reports null for a cleared
raster.

RS_AsGeoTiff only forces WRITE_NODATA off when no band has a no-data
value; forcing it on otherwise would override an explicit
-Dgeotiff.writenodata=false.
if (source.getProperty(propertyName) == Image.UndefinedProperty) {
return source;
}
return new PropertyMaskedRenderedImage(source, propertyName);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we preserve WritableRenderedImage when the source is writable? Clearing NoData currently makes an editable coverage read-only. I reproduced this through the Java API:

import java.awt.image.WritableRenderedImage;
import javax.media.jai.PlanarImage;
import it.geosolutions.jaiext.range.NoDataContainer;
import org.apache.sedona.common.raster.*;

var raster = RasterConstructors.makeEmptyRaster(1, "d", 4, 3, 0, 0, 1);
raster = RasterBandEditors.setBandNoDataValue(raster, 1, -9999.0);
((PlanarImage) raster.getRenderedImage()).setProperty(
    NoDataContainer.GC_NODATA, new NoDataContainer(-9999.0));

System.out.println(raster.isDataEditable()); // true
var cleared = RasterBandEditors.setBandNoDataValue(raster, 1, null);
System.out.println(cleared.isDataEditable()); // false
((WritableRenderedImage) cleared.getRenderedImage()).getWritableTile(0, 0);
// ClassCastException

This affects Java/GeoTools callers using writable tiles; I haven't found a Spark SQL failure from it. A writable subclass selected only for writable sources, forwarding the eight WritableRenderedImage methods, looks like a small way to preserve that capability while keeping the lazy wrapper. A test that acquires, writes, and releases a tile after clearing would cover it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RS_SetBandNoDataValue: documented "NULL removes the no-data value" behavior is unreachable from Spark SQL

2 participants