feat(arrow): store and read a timestamp as the wall clock it is - #7672
Open
kz930 wants to merge 1 commit into
Open
feat(arrow): store and read a timestamp as the wall clock it is#7672kz930 wants to merge 1 commit into
kz930 wants to merge 1 commit into
Conversation
Every other source hands the engine a wall clock. CSV and JSONL carry no zone at all, so what the file states is what the operator gets, and a Texera TIMESTAMP has no zone to record one either. Arrow was the exception: its fields carry a zone, and both directions reconciled that through the value's own epoch, which reads the wall clock in the JVM's zone and puts the machine's setting where the file cannot record it. Reading was the visible half. A zoned vector hands back epoch milliseconds, and `new Timestamp(millis)` renders those in the local zone, so one file read out 00:00 in Los Angeles, 08:00 in UTC and 17:00 in Tokyo. Nothing in the file accounted for the difference and nothing reported it. Writing was the same mismatch from the other end: the number stored was the local instant of the wall clock while the label beside it said UTC, so every reader other than a Texera in that same zone saw the value moved. Both now go through the label the field already carries, which puts Arrow on the same footing as the rest: what the file states is what the engine gets, wherever it runs. Zoneless columns are untouched, those handing back a LocalDateTime that is already the wall clock itself. The spec asserted the old identity, that the stored long equals the value's own epoch. It states the wall clock explicitly now, so what it expects no longer depends on where it runs. Note for anyone with existing files: bytes written before this are read by the new rule, so a timestamp in them shifts by the offset of the zone that wrote it. Closes apache#7666 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7672 +/- ##
============================================
- Coverage 90.14% 90.13% -0.01%
+ Complexity 4415 4414 -1
============================================
Files 1176 1176
Lines 47133 47139 +6
Branches 5285 5288 +3
============================================
+ Hits 42487 42488 +1
- Misses 2879 2881 +2
- Partials 1767 1770 +3
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 437 | 0.267 | 21,780/33,886/33,886 us | 🔴 -5.2% / 🔴 +107.3% |
| 🟢 | bs=100 sw=10 sl=64 | 990 | 0.605 | 100,103/112,899/112,899 us | 🟢 -31.9% / 🔴 +6.0% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,100 | 0.671 | 907,962/952,820/952,820 us | ⚪ within ±5% / 🟢 -9.6% |
Baseline details
Latest main f5017f7 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 437 tuples/sec | 461 tuples/sec | 779.28 tuples/sec | -5.2% | -43.9% |
| bs=10 sw=10 sl=64 | MB/s | 0.267 MB/s | 0.281 MB/s | 0.476 MB/s | -5.0% | -43.9% |
| bs=10 sw=10 sl=64 | p50 | 21,780 us | 21,440 us | 12,712 us | +1.6% | +71.3% |
| bs=10 sw=10 sl=64 | p95 | 33,886 us | 35,068 us | 16,345 us | -3.4% | +107.3% |
| bs=10 sw=10 sl=64 | p99 | 33,886 us | 35,068 us | 19,050 us | -3.4% | +77.9% |
| bs=100 sw=10 sl=64 | throughput | 990 tuples/sec | 882 tuples/sec | 1,017 tuples/sec | +12.2% | -2.7% |
| bs=100 sw=10 sl=64 | MB/s | 0.605 MB/s | 0.538 MB/s | 0.621 MB/s | +12.5% | -2.6% |
| bs=100 sw=10 sl=64 | p50 | 100,103 us | 103,664 us | 100,048 us | -3.4% | +0.1% |
| bs=100 sw=10 sl=64 | p95 | 112,899 us | 165,900 us | 106,477 us | -31.9% | +6.0% |
| bs=100 sw=10 sl=64 | p99 | 112,899 us | 165,900 us | 114,739 us | -31.9% | -1.6% |
| bs=1000 sw=10 sl=64 | throughput | 1,100 tuples/sec | 1,105 tuples/sec | 1,048 tuples/sec | -0.5% | +5.0% |
| bs=1000 sw=10 sl=64 | MB/s | 0.671 MB/s | 0.675 MB/s | 0.639 MB/s | -0.6% | +4.9% |
| bs=1000 sw=10 sl=64 | p50 | 907,962 us | 907,635 us | 976,350 us | +0.0% | -7.0% |
| bs=1000 sw=10 sl=64 | p95 | 952,820 us | 927,940 us | 1,022,084 us | +2.7% | -6.8% |
| bs=1000 sw=10 sl=64 | p99 | 952,820 us | 927,940 us | 1,053,520 us | +2.7% | -9.6% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,457.86,200,128000,437,0.267,21780.17,33886.28,33886.28
1,100,10,64,20,2019.31,2000,1280000,990,0.605,100103.40,112898.73,112898.73
2,1000,10,64,20,18184.94,20000,12800000,1100,0.671,907961.53,952819.64,952819.64
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Every other source hands the engine a wall clock. CSV and JSONL carry no zone at all, so what the file states is what the operator gets, and
TypeCastingand the time-series plot both parse with a plainpd.to_datetime. A Texera TIMESTAMP has no zone to record one either. Arrow was the exception: its fields carry a zone, andArrowUtilsreconciled that through the value's own epoch, which reads the wall clock in the JVM's zone and puts the machine's setting where the file cannot record it.So one file read out different values on servers in different places, with nothing to account for the difference:
Writing was the same mismatch from the other end: the number stored was the local instant of the wall clock while the label beside it said UTC, so every reader other than a Texera in that same zone saw the value moved. Reading that file with
pandas.read_featherreported 08:00 before and reports 00:00 now, which is what Texera itself shows.Both directions now go through the label the field already carries, which puts Arrow on the same footing as the rest: what the file states is what the engine gets, wherever it runs. Zoneless columns, which is what an ordinary
.arrowfile written by pandas holds, are untouched. Those hand back aLocalDateTime, already the wall clock itself.One thing for anyone holding existing files: bytes written before this are read by the new rule, so a timestamp in them shifts by the offset of the zone that wrote it.
Any related issues, documentation, discussions?
Closes #7666
How was this PR tested?
ArrowUtilsSpeccovers both halves and one assertion moved with the change: it stated the identity between the stored long and the value's own epoch, which is machine-dependent, and now states the wall clock explicitly. The round-trip case needed no change, preserving the wall clock being the property both before and after.Beyond the suite, the table above is one file written once and read by four JVMs started with different
-Duser.timezone, and a zoneless file written by pandas was read by three more, reporting the wall clock pandas wrote in each.workflow-core(759 tests) andworkflow-operator(2214) pass, as do the Python-channel specs that share this conversion,PythonProxyServerSpecandPythonProxyClientSpec.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)