Skip to content

test: add sql-file test confirming fallback on parquet variant reads#4021

Merged
andygrove merged 1 commit intoapache:mainfrom
andygrove:variant-fallback
Apr 21, 2026
Merged

test: add sql-file test confirming fallback on parquet variant reads#4021
andygrove merged 1 commit intoapache:mainfrom
andygrove:variant-fallback

Conversation

@andygrove
Copy link
Copy Markdown
Member

@andygrove andygrove commented Apr 21, 2026

Which issue does this PR close?

Part of #1637

Rationale for this change

Comet does not support Spark 4.0's VARIANT data type, so scans exposing a VariantType column must fall back to Spark. There is no regression test pinning this behavior today. If a future change silently lets a VariantType column through the scan, we would only notice through runtime errors or wrong results. A targeted sql-file test locks in the fallback contract.

What changes are included in this PR?

  • New file spark/src/test/resources/sql-tests/expressions/misc/variant.sql
  • Gated with -- MinSparkVersion: 4.0 so it is skipped on Spark 3.4 / 3.5 via CometSqlFileTestSuite
  • Creates parquet-backed tables with a VARIANT column (directly and nested inside STRUCT<v: VARIANT>), populated via parse_json
  • Uses query expect_fallback(Unsupported v of type VariantType) on five queries covering projection, variant_get extraction, variant_get in a predicate, a COUNT(*) with a not-null filter, and a struct-containing-variant projection

How are these changes tested?

Ran the new test under the Spark 4.0 profile:

./mvnw test -Pspark-4.0 -Dsuites="org.apache.comet.CometSqlFileTestSuite variant" -Dtest=none

All five queries in the file fall back with the expected reason and the suite reports Tests: succeeded 1, failed 0.

Adds a Spark 4.0+ sql-file test that creates parquet-backed tables with a VARIANT column (both directly and nested inside a struct) and asserts Comet falls back to Spark with reason "Unsupported v of type VariantType" for projection, filter, and aggregation queries. Gated by MinSparkVersion: 4.0 so older Spark profiles skip it.
Copy link
Copy Markdown
Contributor

@kazuyukitanimura kazuyukitanimura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.
We can add more cases later but good start

@andygrove andygrove merged commit cae101e into apache:main Apr 21, 2026
169 of 171 checks passed
@andygrove andygrove deleted the variant-fallback branch April 21, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants