[GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL - #3312
Open
james-willis wants to merge 3 commits into
Open
Conversation
…e in Spark SQL The docs and the common Java implementation say a null noDataValue removes the band's no-data value, but the catalyst expression null-propagated on any null argument, so RS_SetBandNoDataValue(raster, 1, NULL) returned a null raster and the documented clear path was unreachable from SQL. Route the 2- and 3-arg overloads through the allowRightNull pattern (adding an arity-3 variant) so a null noDataValue reaches the Java implementation. The 4-arg replace overload still null-propagates since replacing pixels with a null no-data value is meaningless.
james-willis
force-pushed
the
fix/rs-setbandnodatavalue-null-clear
branch
from
September 2, 2026 17:26
3768356 to
ab5f6a3
Compare
james-willis
marked this pull request as ready for review
September 2, 2026 17:26
…hrough GeoTIFF round trips The null path consulted band 1's no-data value instead of the target band's, so clearing band N was a no-op whenever band 1 had none. Clearing now also drops the GC_NODATA sentinel from the coverage properties and copies the pixels into a property-free image, since the sentinel rides on the rendered image itself through serialization. RS_AsGeoTiff only writes GDAL_NODATA when the raster actually has a no-data value: GeoTiffWriter otherwise writes a default of 0, which resurrected cleared (and never-set) no-data values on re-read.
jiayuasu
reviewed
Sep 3, 2026
jiayuasu
reviewed
Sep 3, 2026
jiayuasu
reviewed
Sep 3, 2026
jiayuasu
reviewed
Sep 3, 2026
…the GeoTIFF nodata opt-out Clearing no longer copies pixels. RenderedImageAdapter cannot mask an inherited property (it declares getProperty final and answers from the source), so PropertyMaskedRenderedImage wraps the image and hides GC_NODATA while delegating everything else. That keeps a lazily decoded raster undecoded and preserves a non-zero image origin, which the previous copyData path lost -- BufferedImage rejects a raster whose minX/minY are not zero. GridCoverage2D.getProperty falls through to the rendered image, so the sentinel is dropped from the coverage property map as well, on a defensive copy so the input raster is left untouched. GeoTools' own CoverageUtilities.getNoDataProperty now reports null for a cleared raster. RS_AsGeoTiff only forces WRITE_NODATA off when no band has a no-data value; forcing it on otherwise would override an explicit -Dgeotiff.writenodata=false.
1 task
jiayuasu
reviewed
Sep 5, 2026
| if (source.getProperty(propertyName) == Image.UndefinedProperty) { | ||
| return source; | ||
| } | ||
| return new PropertyMaskedRenderedImage(source, propertyName); |
Member
There was a problem hiding this comment.
Could we preserve WritableRenderedImage when the source is writable? Clearing NoData currently makes an editable coverage read-only. I reproduced this through the Java API:
import java.awt.image.WritableRenderedImage;
import javax.media.jai.PlanarImage;
import it.geosolutions.jaiext.range.NoDataContainer;
import org.apache.sedona.common.raster.*;
var raster = RasterConstructors.makeEmptyRaster(1, "d", 4, 3, 0, 0, 1);
raster = RasterBandEditors.setBandNoDataValue(raster, 1, -9999.0);
((PlanarImage) raster.getRenderedImage()).setProperty(
NoDataContainer.GC_NODATA, new NoDataContainer(-9999.0));
System.out.println(raster.isDataEditable()); // true
var cleared = RasterBandEditors.setBandNoDataValue(raster, 1, null);
System.out.println(cleared.isDataEditable()); // false
((WritableRenderedImage) cleared.getRenderedImage()).getWritableTile(0, 0);
// ClassCastExceptionThis affects Java/GeoTools callers using writable tiles; I haven't found a Spark SQL failure from it. A writable subclass selected only for writable sources, forwarding the eight WritableRenderedImage methods, looks like a small way to preserve that capability while keeping the lazy wrapper. A test that acquires, writes, and releases a tile after clearing would cover it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Closes RS_SetBandNoDataValue: documented "NULL removes the no-data value" behavior is unreachable from Spark SQL #3311What changes were proposed in this PR?
The docs and the common Java implementation say a
nullnoDataValueremoves the band's no-data value, but from Spark SQL that path was unreachable: the catalyst expression wrappedRasterBandEditors.setBandNoDataValuewithinferrableFunction2/3/4, which null-propagate as soon as any argument is null, soRS_SetBandNoDataValue(raster, 1, NULL)returned a NULL raster instead. Fixing this matches PostGISST_SetBandNoDataValueNULL-clears semantics and the behavior SedonaDB adopted in apache/sedona-db#1158.InferrableFunction.allowRightNull3, an arity-3 counterpart of the existingallowRightNullpattern (allows the third argument to be null; still null-propagates when the raster or band index is null). It is a separate name rather than an overload because overloadingallowRightNullbreaks eta-expansion of the overloaded functions passed to it at existing call sites.RS_SetBandNoDataValuethroughallowRightNull/allowRightNull3. The 4-argumentreplaceoverload still null-propagates, since replacing pixels with a null no-data value is meaningless; the docs now note that clearing requires the 2-/3-argument form.RS_SetBandNoDataValue.mdthat thereplacevariant requires a non-nullnoDataValue; the existing intro already documents NULL-clears.How was this patch tested?
Strengthened the existing
RS_SetBandNoDataValuetest inrasteralgebraTest.scalaso it distinguishes clearing from null-propagation: it sets a no-data value, clears it with NULL (both the 2- and 3-argument forms), and asserts the resulting raster is non-null whileRS_BandNoDataValueon it is null; it also asserts a NULL raster input still yields NULL. Before this change the previous test passed under both semantics becauseRS_BandNoDataValuereturns null for a null raster too. Verified the new assertions fail without the expression change and pass with it (rasteralgebraTest, Spark 3.5 / Scala 2.12 / JDK 17).Did this PR include necessary documentation updates?
Review follow-up
The second commit addresses the review findings:
rasterNoData) instead ofgetBandNoDataValue(raster), which checked band 1 and made clearing band N a no-op whenever band 1 had no no-data value. Covered by new asymmetric multi-band tests at both the common and SQL levels.GC_NODATAsentinel from the coverage properties and copies the pixels into a property-free image — the sentinel also rides on the rendered image itself through serialization, so cloning the image would let it survive a shuffle. Covered by a common-level test that asserts the property is gone from the coverage, the image, and a serde round trip.GeoTiffWriterwrites a defaultGDAL_NODATAof0for any coverage without a no-data value — an unmodified raster read from a plain GeoTIFF (no nodata) already came back with0.0afterRS_AsGeoTiff→RS_FromGeoTiff, independent of this PR.RS_AsGeoTiffnow setsGeoTiffFormat.WRITE_NODATAto whether the raster actually has a no-data value, which fixes both the cleared case and that pre-existing resurrection. Covered by the new round-trip test onraster_with_no_data/test5.tiff.Second review round
copyData(null)keeps the source origin andBufferedImagerejects a raster whoseminX/minYare non-zero. Clearing is back on aRenderedImagepath via a newPropertyMaskedRenderedImage, which delegates everything to the source image but reportsGC_NODATAasImage.UndefinedProperty. (RenderedImageAdaptercannot do this — it declaresgetPropertyfinal and answers from the source, soremovePropertydoes not mask an inherited value.) A translated-raster regression test covers it.GC_NODATAis also a key in the coverage property map; filtering the map alone is not sufficient either, becauseGridCoverage2D.getPropertyfalls through to the image. Both are now handled, and the map is filtered on a defensive copy so the input raster is no longer mutated (the previous revision mutated it). The test asserts GeoTools' ownCoverageUtilities.getNoDataPropertyreports null for a cleared raster and that the input still reports its original value.WRITE_NODATAopt-out. Fixed: the flag is only forced tofalsewhen no band has a no-data value, so an explicit-Dgeotiff.writenodata=falseis left alone.Tests:
common1339 passed,rasteralgebraTest168 passed,rasterIOTest21 passed.