feat: add experimental native support for in-memory cache, disabled by default - #5051
Open
andygrove wants to merge 8 commits into
Open
feat: add experimental native support for in-memory cache, disabled by default#5051andygrove wants to merge 8 commits into
andygrove wants to merge 8 commits into
Conversation
# Conflicts: # spark/src/main/scala/org/apache/spark/Plugins.scala
InMemoryRelation resolves spark.sql.cache.serializer once per JVM and memoizes the instance in a static field. Test suites share a forked JVM, so whichever suite caches a table first pins the serializer for every suite that follows. CometInMemoryCacheSuite runs after CometExecSuite in the exec group, so its configured serializer was ignored, every cached batch was a DefaultCachedBatch, and 9 of its 11 tests failed on all Spark profiles. Reset the memoized serializer around the suite, via a test-only shim for the private[columnar] clearSerializer. Also address review feedback: - Fall through to the SparkToColumnar path when the native cache is enabled but the relation was cached by a foreign serializer, so enabling the feature is never worse for a scan than leaving it off. Covered by a new CometExecSuite test. - Explain on CometInMemoryTableScanExec.output why the declared output can be narrower than the emitted batch width for an empty projection. - Drop the import made redundant by the org.apache.spark.sql.comet._ wildcard.
andygrove
marked this pull request as ready for review
July 27, 2026 15:03
andygrove
changed the title
Member
Author
|
I found some issues with data type coverage and will push some fixes soon. |
…serializer Three defects found while reviewing type coverage, each with a regression test that fails without the corresponding fix. Pruning dropped every batch for columns without bounds. `tracksBounds` only computes bounds for the types it lists, so a collated `StringType` on Spark 4.x left them null. Spark still builds a partition filter for such a column because a collated string literal is an `AtomicType`, and comparing against null bounds yields null, which the generated predicate treats as false. A filtered query over a collated string column returned zero rows instead of the matching ones, with no error. `buildFilter` now drops predicates over columns that have no bounds, keeping `IsNull` and `IsNotNull` which only read null and row counts. Interval columns broke `cache()`. `Utils.getFieldVector` has no case for `DurationVector` or `IntervalYearVector`, so materializing a cached relation containing a `DayTimeIntervalType` or `YearMonthIntervalType` column threw `Unsupported Arrow Vector for serialize`. The serializer now reports the schemas its Arrow writer supports and delegates the rest to Spark's default cache serializer. Reading a `DefaultCachedBatch` failed for any non-primitive column. `supportsColumnarOutput` returned true unconditionally while `decodeDefaultCachedBatch` decoded through `ColumnAccessor.decompress`, whose `PassThrough` decoder only handles the seven primitive physical types. Spark's own serializer gates `supportsColumnarOutput` on exactly those types to avoid this. Reading a cached string column threw `scala.MatchError: PhysicalStringType`. The cached format is now decided by schema alone rather than by a runtime config, so a relation is either entirely Comet format or entirely delegated to Spark, and the hand-rolled `DefaultCachedBatch` decoder is gone. `spark.sql.cache.serializer` is a static conf, so a format that could flip mid-session was never readable back reliably. `CometExecRule` checks the schema before choosing the native scan. Also add coverage for all supported types, the row read path over `CometCachedBatch`, and a reordered full-width projection.
Member
Author
|
@0lai0 @manuzhang @sandugood @peterxcli could you help with reviews on this PR? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #2391.
Rationale for this change
This PR continues the work started by @pchintar in #4591, who is no longer able to work on it. All credit for the original implementation goes to them. This branch is based on their branch with latest
apache/mainmerged in, and picks up the remaining review feedback and CI failures.Comet currently has limited support for Spark's in-memory cache.
When a table is cached and later read, the cached data cannot be consumed directly by Comet operators. Instead, the execution plan falls back to Spark's cache scan path and introduces an additional
CometSparkColumnarToColumnarconversion before execution can continue in Comet.This extra conversion adds overhead to cached table scans and prevents cached data from remaining on a native Comet execution path.
What changes are included in this PR?
A native cache path for in-memory cached tables behind a new configuration:
spark.comet.exec.inMemoryCache.enabledWhen enabled:
CometCachedBatch.CometInMemoryTableScanExec.CometSparkColumnarToColumnarconversion.SimpleMetricsCachedBatchSerializer, so Spark'sbuildFiltercan prune cached batches before they are decoded.When disabled:
How are these changes tested?
CometInMemoryCacheSuitecovers:CometCachedBatchSELECT count(*))DefaultCachedBatchCometInMemoryCacheBenchmarkcompares the native cache path against the existing fallback path for repeated cached table scans and selective-filter scans exercising cache pruning.Benchmark results (release build, Apple M3 Ultra, JDK 17, Spark 3.5 profile, 5M-row cached table):
Repeated full scan (
SELECT sum(id), sum(k), sum(v))Selective filter (
WHERE id >= 4500000 AND id < 4750000)Performance follow-ups are tracked in #4781.