feat: add 40 native geospatial SQL functions (ST_ expressions)

[sp-202]

Summary

• Adds 40 geospatial SQL functions registered as Spark SQL extensions via
  SparkSessionExtensions.injectFunction. All functions execute natively in
  the Rust/DataFusion engine when Comet is enabled (spark.comet.exec.enabled=true).
• Geometries are represented as WKT strings, consistent with the existing Comet
  geo UDF convention.
• Includes optional Sedona interop: if Apache Sedona is on the classpath, Sedona's
  ST_* expression classes are transparently mapped to the same Comet native UDFs
  via the serde layer, so Sedona queries also benefit from native acceleration.

New files

• spark/src/main/scala/org/apache/comet/expressions/GeoExpressions.scala
  — 40 Spark expression case classes and their SparkSessionExtensions registration descriptors.
• spark/src/main/scala/org/apache/comet/expressions/CometGeoFallback.scala
  — JVM fallback stubs. Called only when Comet native execution is disabled and
  Sedona is not on the classpath; all methods throw UnsupportedOperationException.
• spark/src/main/scala/org/apache/comet/serde/geo.scala
  — Serde map wiring each expression class to its named DataFusion scalar function.
• docs/geo-functions.md
  — Reference documentation covering all 40 functions with signatures, parameter
  types, return types, descriptions, and SQL examples.

Functions added (40)

Category	Functions
Constructors	st_geomfromwkt, st_geomfromgeojson, st_point, st_makeenvelope, st_makeline
Serializers	st_astext, st_asgeojson
Predicates	st_contains, st_intersects, st_within, st_covers, st_coveredby, st_equals, st_touches, st_crosses, st_disjoint, st_overlaps
Measurements	st_area, st_length, st_perimeter, st_distance, st_distancesphere, st_hausdorffdistance, st_numpoints, st_x, st_y
Accessors	st_isempty, st_geometrytype
Transformations	st_centroid, st_envelope, st_convexhull, st_buffer, st_simplify, st_simplifypreservetopology, st_flipcoordinates, st_boundary
Set operations	st_union, st_intersection, st_difference, st_symdifference

Test plan

• ./mvn scalastyle:check passes on all changed Scala files
• ./mvn spotless:check passes (scalafmt formatting)
• ./mvn compile succeeds with no errors or warnings
• Verified all 40 functions are registered in a live spark-shell session
  (spark.sessionState.functionRegistry.lookupFunction)
• Ran a stress test with 10 000 parquet rows across 10 partitions: per-row
  transforms (18 functions), aggregation + shuffle (CometHashAggregate),
  broadcast self-join (CometBroadcastHashJoin), and window functions all
  produced correct results
• Query plans confirm CometNativeScan + CometProject throughout —
  no fallback to JVM evaluation

[andygrove]

Thanks for the contribution @sp-202! I'm not very familiar with geo functions in Spark. Are the functions in this PR intended to accelerate existing Spark geo functions? If so, could you add Comet SQL tests so that we can ensure Spark and Comet produce identical results?

[andygrove]

@paleolimbot fyi

[andygrove]

Thanks for the contribution @sp-202! I'm not very familiar with geo functions in Spark. Are the functions in this PR intended to accelerate existing Spark geo functions? If so, could you add Comet SQL tests so that we can ensure Spark and Comet produce identical results?

If the goal is to accelerate Sedona, I think we need to have a discussion on the mailing list first since this would be a shift in scope for Comet.

[sp-202]

Hi @andygrove — thanks for the clarification, and great to have @paleolimbot's eyes on this

To clarify the intent: this PR is not about accelerating Sedona. The goal is to add native geospatial SQL functions directly into Comet the same way Comet has native math, string, and datetime functions today. No Sedona dependency, no external library coupling. Comet compiles and runs completely without Sedona on the classpath.

So the core ask of this PR is simply:

• Users add CometSparkSessionExtensions (which they already do to enable Comet)
• They get ST_Contains, ST_Intersects, ST_Distance etc. natively in Spark SQL
• No Sedona, no extra JARs, no config changes

On build dependencies — the native geo layer requires the following additions to native/core/Cargo.toml:

geo = "0.28"
geoarrow = "0.8"
geojson = { version = "0.24", features = ["geo-types"] }
geos = { version = "8.3", features = ["static"] }
wkt = "0.11"

And on the build machine, libgeos-dev needs to be installed:

sudo apt install -y libgeos-dev

Since geos is built with features = ["static"], the GEOS C library is statically linked into libcomet.so — so end users don't need libgeos-dev at runtime. The build machine needs it, but the final JAR is self-contained. For CI, I'm happy to submit a follow-up PR updating the CI workflow once this PR direction is agreed.

I am working on the SQL tests now — will push them shortly. Happy to proceed however you think is best

[andygrove]

I don't think we should be adding dependencies on C++ libraries, especially when they are LGPL

[sp-202]

@andygrove — completely agree, thank you for catching this. I will remove the geos dependency entirely. The LGPL license and C++ dependency are both good reasons to avoid it. I'll reimplement ST_Buffer, ST_Union, ST_Intersection, and ST_Difference using the pure Rust geo

[andygrove]

Let's have a discussion on the mailing list about this work.

[paleolimbot]

Cool!

A note that you can achieve roughly this as a user by using Wherobots (which I believe has 100ish functions + join + Parquet IO fully implemented on the Rust side). Of course, it's great to have this in Comet proper as well and I'm happy to help. Full UDT support would help us quite a bit...as I understand it there are a lot of DataType usages that would have to be FieldRef usages if UDTs or true Spark geometry were to be implemented (Spark geometry has a type parameter that is dropped when represented as a Utf8 here).

In its current form, I wonder if this could be implemented in such a way that the function maintenance is not Comet's responsibility...we have many functions implemented, tested, and benchmarked in SedonaDB that could be wrapped (such that only the wrapper code is a Comet maintenance burden).

Geometries are represented as WKT strings

I didn't know about this...Spark and Parquet and SedonaDB are all using WKB here, which is much faster (what Comet is currently doing is the equivalent of passing around doubles as strings...very slow).

cc @zhangfengcdt and @jiayuasu who are more familiar with the Spark side 🙂

[paleolimbot]

Just a note that statically linking GEOS is often cited as not being compatible with a non-(L)GPL license (although in practice it is frequently done). I believe the general Apache line on this is that you can have (L)GPL dependencies as long as they are optional (which in this case I think would mean a non-default feature flag).

[paleolimbot]

I believe the latest version is 0.33?

[paleolimbot]

We have ~150 of these defined in SedonaDB ( https://github.com/apache/sedona-db ) with thousands of lines of tests against a live PostGIS instance. These are exportable/importable from C so you don't necessarily need the DataFusion/arrow-rs versions to align ( https://github.com/apache/sedona-db/blob/d286f9af8164a48889fd0d6fc82bc2bc274d687e/c/sedona-extension/src/scalar_kernel.rs#L39-L46 ). You would need a wrapper on this side to handle the to/from WKT for as long as you are still using it (WKB is what Spark and SedonaDB are using and it's much faster).

[paleolimbot]

Both Apache Sedona ( https://sedona.apache.org/latest/api/sql/Geometry-Constructors/ST_GeomFromWKT/ ) and SedonaDB ( https://sedona.apache.org/sedonadb/latest/reference/sql/st_geomfromwkt/ ) maintain copies of approximately these you can link to to avoid maintaining these yourselves. (We will probably converge those two docs at some point)

[sp-202]

@paleolimbot — thanks, really helpful direction!

1. Is sedona-extension on crates.io or would it be a git dependency?
2. Which arrow-rs version does SedonaDB pin? Comet is on 58.3.0 they need to match.
3. Direct Cargo dep or the C ABI path you mentioned — which do you prefer?

[paleolimbot]

A Cargo dependency would be great, but we're a few versions behind Comet and you're right that they have to align for the non-FFI case. We can figure out a way to make this work on our end if Comet is interested.