HBASE-30179: Define spark4 module and redefine common modules#155
Open
wchevreuil wants to merge 4 commits into
Open
HBASE-30179: Define spark4 module and redefine common modules#155wchevreuil wants to merge 4 commits into
wchevreuil wants to merge 4 commits into
Conversation
Author
|
Disclaimer: I've used cursor's composer model to generate the pom files for the new modules, and the lifecycle changes. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Pull request overview
Phase-0 scaffolding to introduce a parallel spark4 Maven reactor (Apache Spark 4.x / Scala 2.13 / JDK 17+) alongside the existing Spark 3 connector, and to factor SQL pushdown out into a shared hbase-spark-pushdown module that both reactors will share.
Changes:
- Adds a new top-level
spark4reactor (spark4/pom.xml,spark4/hbase-spark4/,spark4/hbase-spark-pushdown/) gated by a-Pspark4profile in the root pom, with a JDK-17 enforcer and a placeholder Scala object. - Introduces a new
spark/hbase-spark-pushdownmodule under the existing Spark 3 reactor and wireshbase-sparkto depend on it. - Updates the binary assembly to exclude the Spark 4 modules until the distribution layout is finalized.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| pom.xml | Adds opt-in spark4 profile that overrides the reactor module list. |
| spark/pom.xml | Registers new hbase-spark-pushdown module and adds it to dependencyManagement. |
| spark/hbase-spark/pom.xml | Declares dependency on the new shared pushdown module. |
| spark/hbase-spark-pushdown/pom.xml | New shared pushdown module for Spark 3 (contains only a pom — no sources moved). |
| spark4/pom.xml | New Spark 4 reactor pom, JDK 17+ enforcer, Scala 2.13.14, Spark 4.0.0. |
| spark4/hbase-spark4/pom.xml | New hbase-spark4 module pom with Spark 4 dependencies and Scala build wiring. |
| spark4/hbase-spark4/src/main/scala/.../Spark4Placeholder.scala | Placeholder Scala object so the module is non-empty. |
| spark4/hbase-spark-pushdown/pom.xml | Spark 4 build of pushdown that pulls sources via build-helper from the Spark 3 module path. |
| hbase-connectors-assembly/src/main/assembly/hbase-connectors-bin.xml | Excludes hbase-spark4 and hbase-spark-pushdown_2.13 from the binary tarball. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+116
to
+149
| <plugin> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-compiler-plugin</artifactId> | ||
| </plugin> | ||
| <plugin> | ||
| <groupId>org.codehaus.mojo</groupId> | ||
| <artifactId>build-helper-maven-plugin</artifactId> | ||
| <executions> | ||
| <execution> | ||
| <id>add-java-source-dir</id> | ||
| <goals> | ||
| <goal>add-source</goal> | ||
| </goals> | ||
| <phase>generate-sources</phase> | ||
| <configuration> | ||
| <sources> | ||
| <source>src/main/java</source> | ||
| </sources> | ||
| </configuration> | ||
| </execution> | ||
| <execution> | ||
| <id>add-scala-source-dir</id> | ||
| <goals> | ||
| <goal>add-source</goal> | ||
| </goals> | ||
| <phase>validate</phase> | ||
| <configuration> | ||
| <sources> | ||
| <source>src/main/scala</source> | ||
| </sources> | ||
| </configuration> | ||
| </execution> | ||
| </executions> | ||
| </plugin> |
Author
There was a problem hiding this comment.
Yes, this was intentional. The goal with this PR was just to re-structure the modules. Related ports and class moves will come on subsequent PRs.
Comment on lines
+134
to
+135
| <source>${project.basedir}/../../spark/hbase-spark-pushdown/src/main/java</source> | ||
| <source>${project.basedir}/../../spark/hbase-spark-pushdown/src/main/scala</source> |
Author
There was a problem hiding this comment.
Will sort it on another PR, when we move SparkSQLPushDownFilter and related classes.
|
|
||
| /** | ||
| * Phase-0 build marker for the Spark 4 connector line (`hbase-spark4`). | ||
| * Replaced incrementally during feature porting; see ../../SPARK4-ROADMAP.md at repo root. |
| <module>kafka</module> | ||
| <module>spark</module> | ||
| <module>spark4</module> | ||
| <module>hbase-connectors-assembly</module> |
| <exclude>org.apache.hbase.connectors.spark:hbase-spark-it</exclude> | ||
| <!-- Not yet packaged when assembly runs (reactor lists spark4 after this module). | ||
| Exclude until dist layout includes Spark 4; see SPARK4-ROADMAP.md --> | ||
| <exclude>org.apache.hbase.connectors.spark:hbase-spark4</exclude> |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.