Gluten functional sweep#18890
Closed
vamsikarnika wants to merge 3 commits into
Closed
Conversation
…test harness Wires the prebuilt quanton-velox-bundle (Onehouse's rebranded Gluten) into the Hudi functional-test harness for a Spark data-type offload sweep. - HoodieClientTestUtils.getSparkConfForTest: inject Quanton plugin + columnar configs, gated behind -Dhudi.test.gluten.enabled=true (default off, so the harness stays vanilla otherwise). Config keys use the bundle's actual ai.onehouse.quanton / spark.quanton.* namespace, NOT upstream org.apache.gluten / spark.gluten.* (the bundle relocates everything via shading). - hudi-spark-client/pom.xml + hudi-spark/pom.xml: add quanton-velox-bundle as a test-scope dependency (installed into the local repo via install:install-file). - hudi-spark/pom.xml scalatest argLine: add the JNI --add-opens/--add-exports the native runtime needs (surefire-run tests already get these from the root pom). NOTE: the sweep does not yet run -- the bundle's native libgluten.so aborts the JVM at plugin init with "Unable to find method rtHandle within signature()J" (native/Java ABI mismatch inside the prebuilt bundle). Tracking separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ack reporter Switch the sweep harness from the quanton-rebranded bundle (whose libgluten.so hardcodes org/apache/gluten/* JNI class paths that shading relocated to ai/onehouse/quanton/*, aborting the JVM at init) to the non-rebranded gluten-velox-bundle, whose native .so and Java classes agree (all 10 JNI bootstrap classes resolve). Gluten now initializes and runs full suites. - HoodieClientTestUtils: org.apache.gluten.GlutenPlugin + spark.gluten.* config. Drop the fallback thresholds (threshold=1 forced whole-query fallback and masked per-operator reasons); keep GlutenFallbackReporter on so each non-offloadable operator logs its concrete reason (the sweep signal). - poms: depend on org.apache.gluten:gluten-velox-bundle (install:install-file'd). Smoke (TestParquetReaderCompatibility): 20/20 pass under Gluten, BUILD SUCCESS with -DskipUTs=true (skips the unrelated scalatest run). Dominant fallback so far is "Unsupported file format UnknownFormat" at Scan HudiFileGroup -- a Hudi scan-path gap, not a type gap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ire argLine The first COW sweep errored 158/230 with "DirectByteBuffer.<init>(long,int) not available" from netty during Gluten columnar-shuffle deserialization. Root cause: the JNI flags were only on the scalatest plugin, but the target tests are JUnit @test and run under surefire (which inherited an argLine lacking io.netty.tryReflectionSetAccessible + the jdk.internal.misc open/export). Add a maven-surefire-plugin override that appends those to @{argLine}. Validated: TestBasicSchemaEvolution 12/12 pass (was 12/12 error), 0 netty errors, fallback reporter shows the real signal ("Unsupported file format UnknownFormat"). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## master #18890 +/- ##
=============================================
- Coverage 57.27% 44.89% -12.38%
+ Complexity 18650 8573 -10077
=============================================
Files 1956 1203 -753
Lines 107076 62990 -44086
Branches 13255 6854 -6401
=============================================
- Hits 61329 28281 -33048
+ Misses 39943 31579 -8364
+ Partials 5804 3130 -2674
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Summary and Changelog
Impact
Risk Level
Documentation Update
Contributor's checklist