feat(pyamber): hash epoch timestamps on Windows - #8212
Conversation
Backport auto-label reportThis
|
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8212 +/- ##
=========================================
Coverage 93.94% 93.94%
Complexity 4759 4759
=========================================
Files 1185 1185
Lines 48069 48070 +1
Branches 5359 5359
=========================================
+ Hits 45159 45160 +1
Misses 1480 1480
Partials 1430 1430
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 430 | 0.262 | 23,146/29,737/29,737 us | 🔴 +9.1% / 🔴 +90.9% |
| 🔴 | bs=100 sw=10 sl=64 | 920 | 0.561 | 107,112/148,528/148,528 us | 🔴 +10.4% / 🔴 +38.5% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,092 | 0.667 | 911,018/998,858/998,858 us | ⚪ within ±5% / 🟢 -7.2% |
Baseline details
Latest main 70c2114 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 430 tuples/sec | 461 tuples/sec | 787.98 tuples/sec | -6.7% | -45.4% |
| bs=10 sw=10 sl=64 | MB/s | 0.262 MB/s | 0.281 MB/s | 0.481 MB/s | -6.8% | -45.5% |
| bs=10 sw=10 sl=64 | p50 | 23,146 us | 21,212 us | 12,593 us | +9.1% | +83.8% |
| bs=10 sw=10 sl=64 | p95 | 29,737 us | 32,632 us | 15,579 us | -8.9% | +90.9% |
| bs=10 sw=10 sl=64 | p99 | 29,737 us | 32,632 us | 18,786 us | -8.9% | +58.3% |
| bs=100 sw=10 sl=64 | throughput | 920 tuples/sec | 949 tuples/sec | 1,008 tuples/sec | -3.1% | -8.7% |
| bs=100 sw=10 sl=64 | MB/s | 0.561 MB/s | 0.579 MB/s | 0.615 MB/s | -3.1% | -8.8% |
| bs=100 sw=10 sl=64 | p50 | 107,112 us | 103,694 us | 100,701 us | +3.3% | +6.4% |
| bs=100 sw=10 sl=64 | p95 | 148,528 us | 134,479 us | 107,244 us | +10.4% | +38.5% |
| bs=100 sw=10 sl=64 | p99 | 148,528 us | 134,479 us | 116,122 us | +10.4% | +27.9% |
| bs=1000 sw=10 sl=64 | throughput | 1,092 tuples/sec | 1,086 tuples/sec | 1,042 tuples/sec | +0.6% | +4.8% |
| bs=1000 sw=10 sl=64 | MB/s | 0.667 MB/s | 0.663 MB/s | 0.636 MB/s | +0.6% | +4.9% |
| bs=1000 sw=10 sl=64 | p50 | 911,018 us | 928,345 us | 981,959 us | -1.9% | -7.2% |
| bs=1000 sw=10 sl=64 | p95 | 998,858 us | 998,661 us | 1,023,080 us | +0.0% | -2.4% |
| bs=1000 sw=10 sl=64 | p99 | 998,858 us | 998,661 us | 1,051,697 us | +0.0% | -5.0% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,465.12,200,128000,430,0.262,23145.78,29736.74,29736.74
1,100,10,64,20,2174.46,2000,1280000,920,0.561,107112.28,148527.94,148527.94
2,1000,10,64,20,18308.72,20000,12800000,1092,0.667,911017.53,998857.61,998857.61
What changes were proposed in this PR?
Attach the system local timezone before converting naive timestamps for tuple hashing. This avoids the Windows pre-epoch failure while preserving aware timestamps and Java-compatible epoch seconds.
Any related issues, documentation, discussions?
Closes #8211
How was this PR tested?
The broader existing test_hash now passes its epoch assertion but later fails while its Windows fixture calls datetime.fromtimestamp with a negative value, before Texera hashing runs.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Codex