[SPARK-58030][SQL][TESTS] Make FileDataSourceV2FallBackSuite robust to extra listener events#57097
Closed
HyukjinKwon wants to merge 1 commit into
Closed
[SPARK-58030][SQL][TESTS] Make FileDataSourceV2FallBackSuite robust to extra listener events#57097HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
HyukjinKwon
force-pushed
the
DO-NOT-MERGE/deflake-master
branch
from
12fd8f8 to
6e5afb3
Compare
HyukjinKwon
changed the title
…o extra listener events The 'Fallback Parquet V2 to V1' test asserted that exactly one query execution event landed on the listener bus (assert(commands.length == 1)). An additional eager 'collect' callback on the analyzed UnresolvedDataSource can race onto the bus, making the count 2 and failing the test nondeterministically (seen on the JDK 21 macOS-26 Maven run). Filter the recorded events to the write 'command' entry and assert on that, rather than requiring the bus to contain exactly one event.
HyukjinKwon
force-pushed
the
DO-NOT-MERGE/deflake-master
branch
from
6e5afb3 to
867b691
Compare
HyukjinKwon
changed the title
dongjoon-hyun
approved these changes
Jul 8, 2026
HyukjinKwon
added a commit
that referenced
this pull request
Jul 8, 2026
… extra listener events ### What changes were proposed in this pull request? Fixes a flaky failure in scheduled CI on `master` (`build_maven_java21_macos26`, `sql#core - other tests`). `FileDataSourceV2FallBackSuite` "Fallback Parquet V2 to V1" asserted `commands.length == 1` on a `QueryExecutionListener`. An extra eager `collect` callback on the analyzed `UnresolvedDataSource` can race onto the listener bus, making the count `2` and failing the test non-deterministically. Fixed by filtering the recorded events to the write `"command"` entry and asserting on that, instead of requiring the bus to contain exactly one event. ### Why are the changes needed? The flake intermittently reds scheduled builds without indicating a real product regression. ### Does this PR introduce _any_ user-facing change? No. Test-only change. ### How was this patch tested? - Passes locally (`Tests: succeeded 5, failed 0`, including "Fallback Parquet V2 to V1"). - Fork CI (`build_maven_java21_macos26`, `sql#core - other tests`): ✅ **PASS** — https://github.com/HyukjinKwon/spark/actions/runs/28906196443 `Tests: succeeded 19657, failed 0` — "All tests passed", with "Fallback Parquet V2 to V1" running clean. (The run's overall status shows a failure only from an unrelated `TransformWithStateAvroSuite` state-store schema-evolution flake in the separate `sql#core - slow tests` shard, which shares no code path with this test-only change.) JIRA: https://issues.apache.org/jira/browse/SPARK-58030 ### Related-but-not-fixed (documented for reviewers) - **`DynamicPartitionPruningHiveScanSuiteAEOff` "broadcast a single key in a HashedRelation"** (`sql#hive - slow tests` on the same workflow): an embedded-Derby internal `ClassCastException` (`ReferencedColumnsDescriptorImpl` → `ExecRowBuilder`) in the DataNucleus metastore layer — Derby-internal state corruption under the slow macOS-26 JDK21 run, with no Spark product line to fix. It passed on this run, consistent with a non-deterministic flake. Not addressed here. Closes #57097 from HyukjinKwon/DO-NOT-MERGE/deflake-master. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> (cherry picked from commit 8dcb844) Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
HyukjinKwon
added a commit
that referenced
this pull request
Jul 8, 2026
… extra listener events ### What changes were proposed in this pull request? Fixes a flaky failure in scheduled CI on `master` (`build_maven_java21_macos26`, `sql#core - other tests`). `FileDataSourceV2FallBackSuite` "Fallback Parquet V2 to V1" asserted `commands.length == 1` on a `QueryExecutionListener`. An extra eager `collect` callback on the analyzed `UnresolvedDataSource` can race onto the listener bus, making the count `2` and failing the test non-deterministically. Fixed by filtering the recorded events to the write `"command"` entry and asserting on that, instead of requiring the bus to contain exactly one event. ### Why are the changes needed? The flake intermittently reds scheduled builds without indicating a real product regression. ### Does this PR introduce _any_ user-facing change? No. Test-only change. ### How was this patch tested? - Passes locally (`Tests: succeeded 5, failed 0`, including "Fallback Parquet V2 to V1"). - Fork CI (`build_maven_java21_macos26`, `sql#core - other tests`): ✅ **PASS** — https://github.com/HyukjinKwon/spark/actions/runs/28906196443 `Tests: succeeded 19657, failed 0` — "All tests passed", with "Fallback Parquet V2 to V1" running clean. (The run's overall status shows a failure only from an unrelated `TransformWithStateAvroSuite` state-store schema-evolution flake in the separate `sql#core - slow tests` shard, which shares no code path with this test-only change.) JIRA: https://issues.apache.org/jira/browse/SPARK-58030 ### Related-but-not-fixed (documented for reviewers) - **`DynamicPartitionPruningHiveScanSuiteAEOff` "broadcast a single key in a HashedRelation"** (`sql#hive - slow tests` on the same workflow): an embedded-Derby internal `ClassCastException` (`ReferencedColumnsDescriptorImpl` → `ExecRowBuilder`) in the DataNucleus metastore layer — Derby-internal state corruption under the slow macOS-26 JDK21 run, with no Spark product line to fix. It passed on this run, consistent with a non-deterministic flake. Not addressed here. Closes #57097 from HyukjinKwon/DO-NOT-MERGE/deflake-master. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> (cherry picked from commit 8dcb844) Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
HyukjinKwon
added a commit
that referenced
this pull request
Jul 8, 2026
… extra listener events ### What changes were proposed in this pull request? Fixes a flaky failure in scheduled CI on `master` (`build_maven_java21_macos26`, `sql#core - other tests`). `FileDataSourceV2FallBackSuite` "Fallback Parquet V2 to V1" asserted `commands.length == 1` on a `QueryExecutionListener`. An extra eager `collect` callback on the analyzed `UnresolvedDataSource` can race onto the listener bus, making the count `2` and failing the test non-deterministically. Fixed by filtering the recorded events to the write `"command"` entry and asserting on that, instead of requiring the bus to contain exactly one event. ### Why are the changes needed? The flake intermittently reds scheduled builds without indicating a real product regression. ### Does this PR introduce _any_ user-facing change? No. Test-only change. ### How was this patch tested? - Passes locally (`Tests: succeeded 5, failed 0`, including "Fallback Parquet V2 to V1"). - Fork CI (`build_maven_java21_macos26`, `sql#core - other tests`): ✅ **PASS** — https://github.com/HyukjinKwon/spark/actions/runs/28906196443 `Tests: succeeded 19657, failed 0` — "All tests passed", with "Fallback Parquet V2 to V1" running clean. (The run's overall status shows a failure only from an unrelated `TransformWithStateAvroSuite` state-store schema-evolution flake in the separate `sql#core - slow tests` shard, which shares no code path with this test-only change.) JIRA: https://issues.apache.org/jira/browse/SPARK-58030 ### Related-but-not-fixed (documented for reviewers) - **`DynamicPartitionPruningHiveScanSuiteAEOff` "broadcast a single key in a HashedRelation"** (`sql#hive - slow tests` on the same workflow): an embedded-Derby internal `ClassCastException` (`ReferencedColumnsDescriptorImpl` → `ExecRowBuilder`) in the DataNucleus metastore layer — Derby-internal state corruption under the slow macOS-26 JDK21 run, with no Spark product line to fix. It passed on this run, consistent with a non-deterministic flake. Not addressed here. Closes #57097 from HyukjinKwon/DO-NOT-MERGE/deflake-master. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> (cherry picked from commit 8dcb844) Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
HyukjinKwon
added a commit
that referenced
this pull request
Jul 8, 2026
… extra listener events ### What changes were proposed in this pull request? Fixes a flaky failure in scheduled CI on `master` (`build_maven_java21_macos26`, `sql#core - other tests`). `FileDataSourceV2FallBackSuite` "Fallback Parquet V2 to V1" asserted `commands.length == 1` on a `QueryExecutionListener`. An extra eager `collect` callback on the analyzed `UnresolvedDataSource` can race onto the listener bus, making the count `2` and failing the test non-deterministically. Fixed by filtering the recorded events to the write `"command"` entry and asserting on that, instead of requiring the bus to contain exactly one event. ### Why are the changes needed? The flake intermittently reds scheduled builds without indicating a real product regression. ### Does this PR introduce _any_ user-facing change? No. Test-only change. ### How was this patch tested? - Passes locally (`Tests: succeeded 5, failed 0`, including "Fallback Parquet V2 to V1"). - Fork CI (`build_maven_java21_macos26`, `sql#core - other tests`): ✅ **PASS** — https://github.com/HyukjinKwon/spark/actions/runs/28906196443 `Tests: succeeded 19657, failed 0` — "All tests passed", with "Fallback Parquet V2 to V1" running clean. (The run's overall status shows a failure only from an unrelated `TransformWithStateAvroSuite` state-store schema-evolution flake in the separate `sql#core - slow tests` shard, which shares no code path with this test-only change.) JIRA: https://issues.apache.org/jira/browse/SPARK-58030 ### Related-but-not-fixed (documented for reviewers) - **`DynamicPartitionPruningHiveScanSuiteAEOff` "broadcast a single key in a HashedRelation"** (`sql#hive - slow tests` on the same workflow): an embedded-Derby internal `ClassCastException` (`ReferencedColumnsDescriptorImpl` → `ExecRowBuilder`) in the DataNucleus metastore layer — Derby-internal state corruption under the slow macOS-26 JDK21 run, with no Spark product line to fix. It passed on this run, consistent with a non-deterministic flake. Not addressed here. Closes #57097 from HyukjinKwon/DO-NOT-MERGE/deflake-master. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> (cherry picked from commit 8dcb844) Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
HyukjinKwon
added a commit
that referenced
this pull request
Jul 8, 2026
… extra listener events ### What changes were proposed in this pull request? Fixes a flaky failure in scheduled CI on `master` (`build_maven_java21_macos26`, `sql#core - other tests`). `FileDataSourceV2FallBackSuite` "Fallback Parquet V2 to V1" asserted `commands.length == 1` on a `QueryExecutionListener`. An extra eager `collect` callback on the analyzed `UnresolvedDataSource` can race onto the listener bus, making the count `2` and failing the test non-deterministically. Fixed by filtering the recorded events to the write `"command"` entry and asserting on that, instead of requiring the bus to contain exactly one event. ### Why are the changes needed? The flake intermittently reds scheduled builds without indicating a real product regression. ### Does this PR introduce _any_ user-facing change? No. Test-only change. ### How was this patch tested? - Passes locally (`Tests: succeeded 5, failed 0`, including "Fallback Parquet V2 to V1"). - Fork CI (`build_maven_java21_macos26`, `sql#core - other tests`): ✅ **PASS** — https://github.com/HyukjinKwon/spark/actions/runs/28906196443 `Tests: succeeded 19657, failed 0` — "All tests passed", with "Fallback Parquet V2 to V1" running clean. (The run's overall status shows a failure only from an unrelated `TransformWithStateAvroSuite` state-store schema-evolution flake in the separate `sql#core - slow tests` shard, which shares no code path with this test-only change.) JIRA: https://issues.apache.org/jira/browse/SPARK-58030 ### Related-but-not-fixed (documented for reviewers) - **`DynamicPartitionPruningHiveScanSuiteAEOff` "broadcast a single key in a HashedRelation"** (`sql#hive - slow tests` on the same workflow): an embedded-Derby internal `ClassCastException` (`ReferencedColumnsDescriptorImpl` → `ExecRowBuilder`) in the DataNucleus metastore layer — Derby-internal state corruption under the slow macOS-26 JDK21 run, with no Spark product line to fix. It passed on this run, consistent with a non-deterministic flake. Not addressed here. Closes #57097 from HyukjinKwon/DO-NOT-MERGE/deflake-master. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> (cherry picked from commit 8dcb844) Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fixes a flaky failure in scheduled CI on
master(build_maven_java21_macos26,sql#core - other tests).FileDataSourceV2FallBackSuite"Fallback Parquet V2 to V1" assertedcommands.length == 1on aQueryExecutionListener. An extra eagercollectcallback on the analyzedUnresolvedDataSourcecan race onto the listener bus, making the count2and failing the test non-deterministically. Fixed by filtering the recorded events to the write"command"entry and asserting on that, instead of requiring the bus to contain exactly one event.Why are the changes needed?
The flake intermittently reds scheduled builds without indicating a real product regression.
Does this PR introduce any user-facing change?
No. Test-only change.
How was this patch tested?
Tests: succeeded 5, failed 0, including "Fallback Parquet V2 to V1").build_maven_java21_macos26,sql#core - other tests): ✅ PASS — https://github.com/HyukjinKwon/spark/actions/runs/28906196443Tests: succeeded 19657, failed 0— "All tests passed", with "Fallback Parquet V2 to V1" running clean. (The run's overall status shows a failure only from an unrelatedTransformWithStateAvroSuitestate-store schema-evolution flake in the separatesql#core - slow testsshard, which shares no code path with this test-only change.)JIRA: https://issues.apache.org/jira/browse/SPARK-58030
Related-but-not-fixed (documented for reviewers)
DynamicPartitionPruningHiveScanSuiteAEOff"broadcast a single key in a HashedRelation" (sql#hive - slow testson the same workflow): an embedded-Derby internalClassCastException(ReferencedColumnsDescriptorImpl→ExecRowBuilder) in the DataNucleus metastore layer — Derby-internal state corruption under the slow macOS-26 JDK21 run, with no Spark product line to fix. It passed on this run, consistent with a non-deterministic flake. Not addressed here.