test(amber, 1.1): drop stale Thread.sleep(1000) after COMPLETED state#4874
Merged
Yicong-Huang merged 1 commit intoMay 3, 2026
Conversation
…che#4680) ### What changes were proposed in this PR? Remove the `Thread.sleep(1000)` workaround inside the result-reading block of `DataProcessingSpec` and `ReconfigurationSpec`. The sleep was annotated with a TODO calling it a workaround for "the issue of reporting `completed` status too early" — the current engine path is synchronous end-to-end, so the workaround is dead code. ```diff .map(terminalOpId => { - //TODO: remove the delay after fixing the issue of reporting "completed" status too early. - Thread.sleep(1000) val uri = getResultUriByLogicalPortId(...) ... }) ``` The synchronous path that emits `COMPLETED`: 1. `DataProcessor.outputOneTuple` handles `FinalizePort` → `outputManager.closeOutputStorageWriterIfNeeded(portId)` 2. `closeOutputStorageWriterIfNeeded` puts a terminate signal on the writer thread's queue and `join()`s it 3. The writer thread, on terminate, calls `IcebergTableWriter.close()` → `flushBuffer()` → `table.newAppend().appendFile(dataFile).commit()` 4. Only then does `DataProcessor` fire `portCompleted` to the controller 5. After all ports complete, controller emits `ExecutionStateUpdate(COMPLETED)` to the client 6. Read side does `IcebergDocument.get()` → `seekToUsableFile()` → `table.refresh()` before scan ### Any related issues, documentation, discussions? Closes apache#4679. ### How was this PR tested? Ran the two specs together with `-T 1` (sequential) on a clean local checkout, 5 consecutive runs: ``` sbt 'WorkflowExecutionService / Test / testOnly \ org.apache.texera.amber.engine.e2e.DataProcessingSpec \ org.apache.texera.amber.engine.e2e.ReconfigurationSpec' run 1: 21/21 passed in 39.2 s run 2: 21/21 passed in 39.5 s run 3: 21/21 passed in 43.2 s run 4: 21/21 passed in 37.3 s run 5: 21/21 passed in 36.9 s ``` Compared to the same two specs with the sleep on the same machine: | spec | before | after | delta | |---|---|---|---| | DataProcessingSpec | 36.5 s | 16.9 s | -19.6 s | | ReconfigurationSpec | 27.2 s | 18.7 s | -8.5 s | DataProcessingSpec saves more than 16 × 1 s because the sleep was inside a per-terminal-port `.map(...)` block — workflows with multiple terminal ports paid the cost more than once per test. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7, 1M context) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Yicong-Huang
changed the title
Yicong-Huang
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## release/v1.1.0-incubating #4874 +/- ##
============================================================
Coverage 41.50% 41.51%
- Complexity 1985 1987 +2
============================================================
Files 952 952
Lines 33796 33796
Branches 3713 3713
============================================================
+ Hits 14028 14029 +1
Misses 19002 19002
+ Partials 766 765 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
aglinxinyuan
approved these changes
May 3, 2026
Yicong-Huang
merged commit 4867fc0
into
apache:release/v1.1.0-incubating
20 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Backport #4680 (commit
598ed0f7f037b96ebe5549759158e46ba736b372onmain) ontorelease/v1.1.0-incubating.Removes the dead
Thread.sleep(1000)+ TODO inside the result-reading block inReconfigurationSpec.scalaandDataProcessingSpec.scala, matching what already landed onmain.Any related issues, documentation, discussions?
Backports #4680. Unblocks the backport leg of #4871.
How was this PR tested?
Cherry-pick of an already-reviewed and merged commit; applies cleanly. CI on this PR exercises the change against the release branch's full Python matrix.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-7)