Skip to content

[GH-3218] Support integer values in Moran's I - #3220

Merged
jiayuasu merged 2 commits into
apache:masterfrom
jiayuasu:fix/moran-integer-values
Aug 1, 2026
Merged

[GH-3218] Support integer values in Moran's I#3220
jiayuasu merged 2 commits into
apache:masterfrom
jiayuasu:fix/moran-integer-values

Conversation

@jiayuasu

@jiayuasu jiayuasu commented Jul 31, 2026

Copy link
Copy Markdown
Member

What changed

  • Normalize the selected Moran value column to DOUBLE before computing the mean and centered-value statistics.
  • Build mean, centering, and neighbor expressions with Spark's column API instead of interpolated SQL, including a typed literal for the mean.
  • Treat configured ID and value column names as literal top-level and nested field names, including names containing dots.
  • Reject all-null value columns and non-finite means with clear IllegalArgumentException messages.
  • Add regression coverage for BIGINT values, dotted column names, all-null values, and NaN means.

Why

Moran's I returns its intermediate aggregates through a typed tuple of Double values. With an integral input column, Spark inferred z * z and its sum as decimal values, then rejected the attempted upcast from DECIMAL(38, scale) to DOUBLE.

Normalizing the value column at the calculation boundary keeps the intermediate expressions consistent with the existing double-based result contract. Callers can now pass integer counts without adding their own cast.

The same calculation interpolated the mean and configured column names into SQL expressions. That made null or non-finite means fragile and caused dots in literal column names to be parsed as nested paths. Column expressions preserve the intended types and field names throughout the calculation.

Validation

  • MoranTest on Spark 3.4 / Scala 2.12: 8 tests passed.
  • MoranTest on Spark 3.5.8 / Scala 2.12: 8 tests passed.
  • MoranTest on Spark 4.1.1 / Scala 2.13: 8 tests passed.
  • Repository pre-commit hooks passed.

Fixes #3218.

@jiayuasu jiayuasu added this to the sedona-1.9.1 milestone Jul 31, 2026
@jiayuasu
jiayuasu requested a review from james-willis July 31, 2026 14:27
@jiayuasu
jiayuasu marked this pull request as ready for review August 1, 2026 02:16
@jiayuasu
jiayuasu merged commit 547565d into apache:master Aug 1, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Moran.getGlobal fails with CANNOT_UP_CAST_DATATYPE when the value column is an integer type

1 participant