Skip to content

feat(pyamber): hash epoch timestamps on Windows - #8212

Open
carloea2 wants to merge 1 commit into
apache:mainfrom
carloea2:fix/windows-timestamp-hash
Open

feat(pyamber): hash epoch timestamps on Windows#8212
carloea2 wants to merge 1 commit into
apache:mainfrom
carloea2:fix/windows-timestamp-hash

Conversation

@carloea2

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Attach the system local timezone before converting naive timestamps for tuple hashing. This avoids the Windows pre-epoch failure while preserving aware timestamps and Java-compatible epoch seconds.

Any related issues, documentation, discussions?

Closes #8211

How was this PR tested?

python -c "import sys,pytest; sys.path[:0]=[r'<worktree>ambersrcmainpython',r'<main-checkout>ambersrcmainpython']; raise SystemExit(pytest.main([r'amber/src/test/python/core/models/test_tuple.py::TestTuple::test_hash_supports_local_epoch_timestamp','-q','-p','no:cacheprovider']))"
1 passed

python -c "import sys,pytest; sys.path[:0]=[r'<worktree>ambersrcmainpython',r'<main-checkout>ambersrcmainpython']; raise SystemExit(pytest.main([r'amber/src/test/python/core/models/test_tuple.py','-q','-p','no:cacheprovider','-k','not test_hash']))"
101 passed, 3 deselected

ruff check amber/src/main/python amber/src/test/python
All checks passed

ruff format --check amber/src/main/python amber/src/test/python
213 files already formatted

The broader existing test_hash now passes its epoch assertion but later fails while its Windows fixture calls datetime.fromtimestamp with a negative value, before Texera hashing runs.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Codex

@Yicong-Huang Yicong-Huang added the release/v1.2 back porting to release/v1.2 label Aug 31, 2026
@github-actions
github-actions Bot requested a review from xuang7 August 31, 2026 03:19
@github-actions

Copy link
Copy Markdown
Contributor

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. release/* labels drive the post-merge backport, so add or remove one to change where this fix lands.

Release branch Analysis
⚠️ release/v1.3 Not labeled automatically — none of the files this PR modifies exist on this branch (amber/src/main/python/core/models/tuple.py, amber/src/test/python/core/models/test_tuple.py). The fix may target code that isn't on this release, or the files were moved/renamed after the branch was cut. Please check and add release/v1.3 by hand if this fix should be backported here.
release/v1.2 Change detected on this branch — label added; this fix is queued to backport here. Requested review from @xuang7.

Auto-label run.

@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @eugenegujing, @aglinxinyuan
    You can notify them by mentioning @eugenegujing, @aglinxinyuan in a comment.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.94%. Comparing base (70c2114) to head (2129adf).

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #8212   +/-   ##
=========================================
  Coverage     93.94%   93.94%           
  Complexity     4759     4759           
=========================================
  Files          1185     1185           
  Lines         48069    48070    +1     
  Branches       5359     5359           
=========================================
+ Hits          45159    45160    +1     
  Misses         1480     1480           
  Partials       1430     1430           
Flag Coverage Δ *Carryforward flag
access-control-service 81.00% <ø> (ø) Carriedforward from 70c2114
agent-service 99.32% <ø> (ø) Carriedforward from 70c2114
amber 89.99% <ø> (ø) Carriedforward from 70c2114
computing-unit-managing-service 73.67% <ø> (ø) Carriedforward from 70c2114
config-service 86.86% <ø> (ø) Carriedforward from 70c2114
file-service 87.91% <ø> (ø) Carriedforward from 70c2114
frontend 96.49% <ø> (ø) Carriedforward from 70c2114
notebook-migration-service 79.31% <ø> (ø) Carriedforward from 70c2114
pyamber 98.87% <100.00%> (+<0.01%) ⬆️
workflow-compiling-service 77.19% <ø> (ø) Carriedforward from 70c2114

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 5 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 70c2114 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 430 0.262 23,146/29,737/29,737 us 🔴 +9.1% / 🔴 +90.9%
🔴 bs=100 sw=10 sl=64 920 0.561 107,112/148,528/148,528 us 🔴 +10.4% / 🔴 +38.5%
bs=1000 sw=10 sl=64 1,092 0.667 911,018/998,858/998,858 us ⚪ within ±5% / 🟢 -7.2%
Baseline details

Latest main 70c2114 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 430 tuples/sec 461 tuples/sec 787.98 tuples/sec -6.7% -45.4%
bs=10 sw=10 sl=64 MB/s 0.262 MB/s 0.281 MB/s 0.481 MB/s -6.8% -45.5%
bs=10 sw=10 sl=64 p50 23,146 us 21,212 us 12,593 us +9.1% +83.8%
bs=10 sw=10 sl=64 p95 29,737 us 32,632 us 15,579 us -8.9% +90.9%
bs=10 sw=10 sl=64 p99 29,737 us 32,632 us 18,786 us -8.9% +58.3%
bs=100 sw=10 sl=64 throughput 920 tuples/sec 949 tuples/sec 1,008 tuples/sec -3.1% -8.7%
bs=100 sw=10 sl=64 MB/s 0.561 MB/s 0.579 MB/s 0.615 MB/s -3.1% -8.8%
bs=100 sw=10 sl=64 p50 107,112 us 103,694 us 100,701 us +3.3% +6.4%
bs=100 sw=10 sl=64 p95 148,528 us 134,479 us 107,244 us +10.4% +38.5%
bs=100 sw=10 sl=64 p99 148,528 us 134,479 us 116,122 us +10.4% +27.9%
bs=1000 sw=10 sl=64 throughput 1,092 tuples/sec 1,086 tuples/sec 1,042 tuples/sec +0.6% +4.8%
bs=1000 sw=10 sl=64 MB/s 0.667 MB/s 0.663 MB/s 0.636 MB/s +0.6% +4.9%
bs=1000 sw=10 sl=64 p50 911,018 us 928,345 us 981,959 us -1.9% -7.2%
bs=1000 sw=10 sl=64 p95 998,858 us 998,661 us 1,023,080 us +0.0% -2.4%
bs=1000 sw=10 sl=64 p99 998,858 us 998,661 us 1,051,697 us +0.0% -5.0%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,465.12,200,128000,430,0.262,23145.78,29736.74,29736.74
1,100,10,64,20,2174.46,2000,1280000,920,0.561,107112.28,148527.94,148527.94
2,1000,10,64,20,18308.72,20000,12800000,1092,0.667,911017.53,998857.61,998857.61

@carloea2 carloea2 changed the title fix(pyamber): hash epoch timestamps on Windows feat(pyamber): hash epoch timestamps on Windows Aug 31, 2026
@mengw15 mengw15 removed the release/v1.2 back porting to release/v1.2 label Aug 31, 2026
@xuang7 xuang7 removed the fix label Sep 1, 2026
@xuang7
xuang7 removed their request for review September 1, 2026 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tuple hashing fails for local epoch timestamps on Windows

5 participants