Skip to content

[SPARK-56839][SS] Remove dev loop config for new state format of stream-stream join #55837

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HeartSaVioR wants to merge 2 commits into from
Closed

[SPARK-56839][SS] Remove dev loop config for new state format of stream-stream join #55837

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
250049a
[SS] Remove the gating config for stream-stream join with state forma…
HeartSaVioR May 12, 2026
a3e3c85
empty commit
HeartSaVioR May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 4 additions & 13 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3327,24 +3327,15 @@ object SQLConf {
.doc("State format version used by streaming join operations in a streaming query. " +
"State between versions are tend to be incompatible, so state format version shouldn't " +
"be modified after running. Version 3 uses a single state store with virtual column " +
"families instead of four stores and is only supported with RocksDB. NOTE: version " +
"1 is DEPRECATED and should not be explicitly set by users. " +
"Version 4 is under development and only available for testing.")
"families instead of four stores and is only supported with RocksDB. Version 4 optimizes " +
"watermark-based eviction, also using a single state store with virtual " +
"column families and only supported with RocksDB. NOTE: version 1 is DEPRECATED and " +
"should not be explicitly set by users.")
.version("3.0.0")
.intConf
.checkValue(v => Set(1, 2, 3, 4).contains(v), "Valid versions are 1, 2, 3, and 4")
.createWithDefault(2)

val STREAMING_JOIN_STATE_FORMAT_V4_ENABLED =
buildConf("spark.sql.streaming.join.stateFormatV4.enabled")
.internal()
.doc("When true, enables state format version 4 for stream-stream joins. " +
"This config will be removed once V4 is complete.")
.version("4.2.0")
.withBindingPolicy(ConfigBindingPolicy.SESSION)
.booleanConf
.createWithDefaultFunction(() => Utils.isTesting)

val STREAMING_SESSION_WINDOW_MERGE_SESSIONS_IN_LOCAL_PARTITION =
buildConf("spark.sql.streaming.sessionWindow.merge.sessions.in.local.partition")
.doc("When true, streaming session window sorts and merge sessions in local partition " +
Expand Down
2 changes: 0 additions & 2 deletions ...spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2045,8 +2045,6 @@ object SymmetricHashJoinStateManager {
joinStoreGenerator: JoinStateManagerStoreGenerator,
joinKeyOrdinalForWatermark: Option[Int] = None): SymmetricHashJoinStateManager = {
if (stateFormatVersion == 4) {
require(SQLConf.get.getConf(SQLConf.STREAMING_JOIN_STATE_FORMAT_V4_ENABLED),
"State format version 4 is under development.")
new SymmetricHashJoinStateManagerV4(
joinSide, inputValueAttributes, joinKeys, stateInfo, storeConf, hadoopConf,
partitionId, keyToNumValuesStateStoreCkptId, keyWithIndexToValueStateStoreCkptId,
Expand Down