-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-55105][SS] Add Integration Test for Join Operator #53874
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
JIRA Issue Information=== New Feature SPARK-55105 === This comment was automatically generated by GitHub Actions |
48cef31 to
0ed1512
Compare
aaa642b to
aa2e31b
Compare
...org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionIntegrationSuite.scala
Outdated
Show resolved
Hide resolved
...org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionIntegrationSuite.scala
Outdated
Show resolved
Hide resolved
...apache/spark/sql/execution/streaming/state/OfflineStateRepartitionIntegrationJoinSuite.scala
Outdated
Show resolved
Hide resolved
...org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionIntegrationSuite.scala
Show resolved
Hide resolved
9c5c1d1 to
dd2c949
Compare
|
|
||
| import testImplicits._ | ||
| import OfflineStateRepartitionTestUtils._ | ||
| abstract class OfflineStateRepartitionIntegrationSuiteBase extends StateDataSourceTestBase { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changing this to abstract class instead of trait because here is the inheritance chain
1. OfflineStateRepartitionIntegrationSuite (class) extends OfflineStateRepartitionIntegrationSuiteBase (trait)
2. OfflineStateRepartitionIntegrationSuiteBase (trait) extends StateDataSourceTestBase (trait)
3. StateDataSourceTestBase (trait) extends StreamTest (trait)
4. StreamTest (trait) extends QueryTest (abstract class)
5. QueryTest (abstract class) extends PlanTest (trait)
6. PlanTest (trait) extends SparkFunSuite (class)
which means that there is a concrete class → trait → trait → abstract class → trait → class.
In Scala, traits cannot safely hide or defer concrete superclasses for concrete subclasses; the concrete superclass must be resolved at a class boundary. Making this an abstract class allows it to legally “anchor” the concrete superclass in the inheritance chain, ensuring correct class linearization and avoiding illegal inheritance errors during compilation
refactor code support column family
dd2c949 to
c5eb04d
Compare
What changes were proposed in this pull request?
Adding integration test for join operator, including V1, V2 and V3.
V1 and V2 join operator use multiple state stores with single column family, while V3 uses a single state store with multiple column families
Why are the changes needed?
Adding integration test for all operators to ensure repartition correctness
Does this PR introduce any user-facing change?
No
How was this patch tested?
Add added test
Was this patch authored or co-authored using generative AI tooling?
yes. Sonnet 4.5