feat(storage): add datasets resource-type prefix to logical paths#6502
feat(storage): add datasets resource-type prefix to logical paths#6502tanishqgandhi1908 wants to merge 6 commits into
Conversation
… tests Review fixes on apache#5911: use dataset owner email in retrieveLatestDatasetVersion, replace brittle Option.get/.head with headOption, strip datasets prefix on selection-modal reopen, clarify FileResolver docs; add FileResolver/DatasetFileNode and frontend path-helper tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
|
/request-review @aicam |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #6502 +/- ##
============================================
- Coverage 70.44% 70.42% -0.03%
+ Complexity 3404 3403 -1
============================================
Files 1142 1142
Lines 44876 44887 +11
Branches 4955 4957 +2
============================================
- Hits 31613 31610 -3
- Misses 11619 11629 +10
- Partials 1644 1648 +4
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 445 | 0.272 | 21,561/32,802/32,802 us | 🔴 -9.0% / 🔴 +128.4% |
| ⚪ | bs=100 sw=10 sl=64 | 948 | 0.578 | 105,133/130,351/130,351 us | ⚪ within ±5% / 🔴 +31.5% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,118 | 0.682 | 889,048/988,623/988,623 us | 🔴 +5.4% / ⚪ within ±5% |
Baseline details
Latest main f722c26 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 445 tuples/sec | 489 tuples/sec | 831.47 tuples/sec | -9.0% | -46.5% |
| bs=10 sw=10 sl=64 | MB/s | 0.272 MB/s | 0.298 MB/s | 0.507 MB/s | -8.7% | -46.4% |
| bs=10 sw=10 sl=64 | p50 | 21,561 us | 21,371 us | 11,844 us | +0.9% | +82.0% |
| bs=10 sw=10 sl=64 | p95 | 32,802 us | 31,001 us | 14,359 us | +5.8% | +128.4% |
| bs=10 sw=10 sl=64 | p99 | 32,802 us | 31,001 us | 18,888 us | +5.8% | +73.7% |
| bs=100 sw=10 sl=64 | throughput | 948 tuples/sec | 923 tuples/sec | 1,089 tuples/sec | +2.7% | -12.9% |
| bs=100 sw=10 sl=64 | MB/s | 0.578 MB/s | 0.563 MB/s | 0.665 MB/s | +2.7% | -13.0% |
| bs=100 sw=10 sl=64 | p50 | 105,133 us | 106,212 us | 92,503 us | -1.0% | +13.7% |
| bs=100 sw=10 sl=64 | p95 | 130,351 us | 136,400 us | 99,121 us | -4.4% | +31.5% |
| bs=100 sw=10 sl=64 | p99 | 130,351 us | 136,400 us | 112,422 us | -4.4% | +15.9% |
| bs=1000 sw=10 sl=64 | throughput | 1,118 tuples/sec | 1,121 tuples/sec | 1,117 tuples/sec | -0.3% | +0.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.682 MB/s | 0.684 MB/s | 0.682 MB/s | -0.3% | +0.0% |
| bs=1000 sw=10 sl=64 | p50 | 889,048 us | 888,167 us | 910,046 us | +0.1% | -2.3% |
| bs=1000 sw=10 sl=64 | p95 | 988,623 us | 937,725 us | 950,716 us | +5.4% | +4.0% |
| bs=1000 sw=10 sl=64 | p99 | 988,623 us | 937,725 us | 985,832 us | +5.4% | +0.3% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,449.37,200,128000,445,0.272,21560.50,32801.82,32801.82
1,100,10,64,20,2110.49,2000,1280000,948,0.578,105133.35,130351.05,130351.05
2,1000,10,64,20,17892.39,20000,12800000,1118,0.682,889048.22,988622.69,988622.69The datasets logical-path prefix strips four leading segments (datasets/owner/dataset/version); update the cover-image test's input path to include the prefix so the extracted relative path is the file name, not an empty string. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
I see that it has no behavioral change now, but the intention is to later introduce other source types prefixes such as models right? when we do so, it will eventually have a collision on the path without the prefix. how are we going to solve that data migration issue later? A possible solution:
If you can have a better solution or data migration plan that would be better. |
| * Expected format: /ownerEmail/datasetName/versionName/fileRelativePath | ||
| * | ||
| * Two path shapes are accepted: | ||
| * - Prefixed: /datasets/ownerEmail/datasetName/versionName/fileRelativePath |
There was a problem hiding this comment.
In the new design, we will have a new table named "model", in this regard, an unprefixed path can not be resolved because the prefix indicate the table, please update implementation.
|
Makes sense. Sure let me research a bit and get back |
What changes were proposed in this PR?
Adds an explicit resource-type prefix to asset logical file paths, changing the format from
/<owner>/<name>/<version>/<file>to/datasets/<owner>/<name>/<version>/<file>.Paths are namespaced under a resource-type segment so different resource kinds can be distinguished from a single logical path. The resolver strips a recognized prefix and parses paths without one exactly as before, so there is no behavior
change for existing assets. A leading segment that isn't a recognized type is left untouched (so an unregistered segment like
modelsis not stripped yet).Any related issues, documentation, discussions?
Closes #6495.
How was this PR tested?
New and updated unit tests, all passing locally:
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)