feat(pyamber): skip duplicate output writers - #8278
Conversation
Backport auto-label reportThis
|
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8278 +/- ##
=========================================
Coverage 93.93% 93.93%
Complexity 4761 4761
=========================================
Files 1191 1191
Lines 48308 48308
Branches 5377 5377
=========================================
Hits 45378 45378
Misses 1495 1495
Partials 1435 1435
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 377 | 0.23 | 25,787/39,870/39,870 us | 🔴 +11.5% / 🔴 +155.9% |
| 🔴 | bs=100 sw=10 sl=64 | 932 | 0.569 | 106,397/147,293/147,293 us | 🔴 +19.3% / 🔴 +37.3% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,093 | 0.667 | 903,589/1,058,874/1,058,874 us | ⚪ within ±5% / 🟢 -8.0% |
Baseline details
Latest main 50321e4 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 377 tuples/sec | 424 tuples/sec | 787.98 tuples/sec | -11.1% | -52.2% |
| bs=10 sw=10 sl=64 | MB/s | 0.23 MB/s | 0.259 MB/s | 0.481 MB/s | -11.2% | -52.2% |
| bs=10 sw=10 sl=64 | p50 | 25,787 us | 23,133 us | 12,593 us | +11.5% | +104.8% |
| bs=10 sw=10 sl=64 | p95 | 39,870 us | 36,374 us | 15,579 us | +9.6% | +155.9% |
| bs=10 sw=10 sl=64 | p99 | 39,870 us | 36,374 us | 18,786 us | +9.6% | +112.2% |
| bs=100 sw=10 sl=64 | throughput | 932 tuples/sec | 987 tuples/sec | 1,008 tuples/sec | -5.6% | -7.5% |
| bs=100 sw=10 sl=64 | MB/s | 0.569 MB/s | 0.603 MB/s | 0.615 MB/s | -5.6% | -7.5% |
| bs=100 sw=10 sl=64 | p50 | 106,397 us | 100,973 us | 100,701 us | +5.4% | +5.7% |
| bs=100 sw=10 sl=64 | p95 | 147,293 us | 123,444 us | 107,244 us | +19.3% | +37.3% |
| bs=100 sw=10 sl=64 | p99 | 147,293 us | 123,444 us | 116,122 us | +19.3% | +26.8% |
| bs=1000 sw=10 sl=64 | throughput | 1,093 tuples/sec | 1,085 tuples/sec | 1,042 tuples/sec | +0.7% | +4.9% |
| bs=1000 sw=10 sl=64 | MB/s | 0.667 MB/s | 0.663 MB/s | 0.636 MB/s | +0.6% | +4.9% |
| bs=1000 sw=10 sl=64 | p50 | 903,589 us | 927,852 us | 981,959 us | -2.6% | -8.0% |
| bs=1000 sw=10 sl=64 | p95 | 1,058,874 us | 1,018,444 us | 1,023,080 us | +4.0% | +3.5% |
| bs=1000 sw=10 sl=64 | p99 | 1,058,874 us | 1,018,444 us | 1,051,697 us | +4.0% | +0.7% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,531.03,200,128000,377,0.230,25786.88,39869.54,39869.54
1,100,10,64,20,2146.44,2000,1280000,932,0.569,106396.95,147293.44,147293.44
2,1000,10,64,20,18294.20,20000,12800000,1093,0.667,903588.86,1058873.61,1058873.61
What changes were proposed in this PR?
Start output storage writers only when the output port is first registered. Duplicate registration keeps the existing port and writer set.
Any related issues, documentation, discussions?
Closes #8277
How was this PR tested?
Added coverage proving that duplicate port registration does not start replacement writers, alongside the existing positive writer-setup coverage.
C:\Users\carlo\texera\texera\.venv312\Scripts\python.exe -c "import sys,pytest; sys.path[:0]=[r'C:\Users\carlo\texera\texera-worktrees\investigate-bug74\amber\src\main\python',r'C:\Users\carlo\texera\texera\amber\src\main\python']; raise SystemExit(pytest.main([r'amber\src\test\python\core\architecture\packaging\test_output_manager.py','-p','no:cacheprovider','-q']))"C:\Users\carlo\texera\texera\.venv312\Scripts\ruff.exe check amber/src/main/python/core/architecture/packaging/output_manager.py amber/src/test/python/core/architecture/packaging/test_output_manager.pyC:\Users\carlo\texera\texera\.venv312\Scripts\ruff.exe format --check amber/src/main/python/core/architecture/packaging/output_manager.py amber/src/test/python/core/architecture/packaging/test_output_manager.pyAll 40 tests passed. Ruff checks passed.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Codex