Skip to content

Conversation

@zifeif2
Copy link
Contributor

@zifeif2 zifeif2 commented Jan 20, 2026

What changes were proposed in this pull request?

Adding integration test for join operator, including V1, V2 and V3.
V1 and V2 join operator use multiple state stores with single column family, while V3 uses a single state store with multiple column families

Why are the changes needed?

Adding integration test for all operators to ensure repartition correctness

Does this PR introduce any user-facing change?

No

How was this patch tested?

Add added test

Was this patch authored or co-authored using generative AI tooling?

yes. Sonnet 4.5

@github-actions
Copy link

JIRA Issue Information

=== New Feature SPARK-55105 ===
Summary: Test repartition for Join
Assignee: None
Status: Open
Affected: ["4.1.1"]


This comment was automatically generated by GitHub Actions

@zifeif2 zifeif2 force-pushed the repartition-test-join branch from 48cef31 to 0ed1512 Compare January 20, 2026 23:03
@zifeif2 zifeif2 marked this pull request as ready for review January 21, 2026 18:10
@zifeif2 zifeif2 force-pushed the repartition-test-join branch 5 times, most recently from aaa642b to aa2e31b Compare January 26, 2026 20:35
@zifeif2 zifeif2 force-pushed the repartition-test-join branch from 9c5c1d1 to dd2c949 Compare January 27, 2026 18:33

import testImplicits._
import OfflineStateRepartitionTestUtils._
abstract class OfflineStateRepartitionIntegrationSuiteBase extends StateDataSourceTestBase {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing this to abstract class instead of trait because here is the inheritance chain

  1. OfflineStateRepartitionIntegrationSuite (class) extends OfflineStateRepartitionIntegrationSuiteBase (trait)                                                                                                                         
  2. OfflineStateRepartitionIntegrationSuiteBase (trait) extends StateDataSourceTestBase (trait)                                                                                                                                         
  3. StateDataSourceTestBase (trait) extends StreamTest (trait)                                                                                                                                                                          
  4. StreamTest (trait) extends QueryTest (abstract class)                                                                                                                                                                               
  5. QueryTest (abstract class) extends PlanTest (trait)                                                                                                                                                                                 
  6. PlanTest (trait) extends SparkFunSuite (class) 

which means that there is a concrete class → trait → trait → abstract class → trait → class.
In Scala, traits cannot safely hide or defer concrete superclasses for concrete subclasses; the concrete superclass must be resolved at a class boundary. Making this an abstract class allows it to legally “anchor” the concrete superclass in the inheritance chain, ensuring correct class linearization and avoiding illegal inheritance errors during compilation

@zifeif2 zifeif2 force-pushed the repartition-test-join branch from dd2c949 to c5eb04d Compare January 28, 2026 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants