Skip to content

feat(storage): add datasets resource-type prefix to logical paths#6502

Open
tanishqgandhi1908 wants to merge 6 commits into
apache:mainfrom
tanishqgandhi1908:feat/dataset-logical-path-prefix
Open

feat(storage): add datasets resource-type prefix to logical paths#6502
tanishqgandhi1908 wants to merge 6 commits into
apache:mainfrom
tanishqgandhi1908:feat/dataset-logical-path-prefix

Conversation

@tanishqgandhi1908

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Adds an explicit resource-type prefix to asset logical file paths, changing the format from

/<owner>/<name>/<version>/<file> to
/datasets/<owner>/<name>/<version>/<file>.

Paths are namespaced under a resource-type segment so different resource kinds can be distinguished from a single logical path. The resolver strips a recognized prefix and parses paths without one exactly as before, so there is no behavior
change
for existing assets. A leading segment that isn't a recognized type is left untouched (so an unregistered segment like models is not stripped yet).

  • Path resolver: strips a known resource-type prefix in one place before parsing; unprefixed (legacy) paths still parse for backward compatibility.
  • File tree: the tree is rooted at a resource-type node, with owner/name/version nested underneath; owner-email lookup and empty-tree handling are hardened.
  • Frontend: file-tree relative-path handling and the selection modal account for the prefix; an unused client-side path parser with no callers is removed.
  • Python file API: strips the prefix on parse and re-emits it when requesting a presigned download URL.

Any related issues, documentation, discussions?

Closes #6495.

How was this PR tested?

New and updated unit tests, all passing locally:

  • Resolver: an unprefixed path resolves identically to the prefixed path; an unregistered prefix is not stripped; a too-few-segments path is rejected.
  • File tree: the tree is rooted at the resource-type node with owner/name/version nested underneath.
  • Frontend: relative-path extraction accounts for the new leading segment.
  • Python file API: the prefix is stripped on parse and re-emitted on presign.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

aicam and others added 5 commits July 1, 2026 13:57
… tests

Review fixes on apache#5911: use dataset owner email in retrieveLatestDatasetVersion,
replace brittle Option.get/.head with headOption, strip datasets prefix on
selection-modal reopen, clarify FileResolver docs; add FileResolver/DatasetFileNode
and frontend path-helper tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions github-actions Bot added feature pyamber frontend Changes related to the frontend GUI common platform Non-amber Scala service paths labels Jul 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Yicong-Huang, @Ma77Ball, @kunwp1
    You can notify them by mentioning @Yicong-Huang, @Ma77Ball, @kunwp1 in a comment.

@tanishqgandhi1908

Copy link
Copy Markdown
Contributor Author

/request-review @aicam

@github-actions
github-actions Bot requested a review from aicam July 17, 2026 21:10
@tanishqgandhi1908 tanishqgandhi1908 changed the title Feat/dataset logical path prefix feat(storage): add datasets resource-type prefix to asset logical paths Jul 17, 2026
@tanishqgandhi1908 tanishqgandhi1908 changed the title feat(storage): add datasets resource-type prefix to asset logical paths feat(storage): add datasets resource-type prefix to logical paths Jul 17, 2026
@codecov-commenter

codecov-commenter commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 47.36842% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.42%. Comparing base (f722c26) to head (a31c897).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...lection-modal/dataset-selection-modal.component.ts 0.00% 5 Missing ⚠️
...ache/texera/service/resource/DatasetResource.scala 0.00% 4 Missing ⚠️
...pache/texera/amber/core/storage/FileResolver.scala 75.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #6502      +/-   ##
============================================
- Coverage     70.44%   70.42%   -0.03%     
+ Complexity     3404     3403       -1     
============================================
  Files          1142     1142              
  Lines         44876    44887      +11     
  Branches       4955     4957       +2     
============================================
- Hits          31613    31610       -3     
- Misses        11619    11629      +10     
- Partials       1644     1648       +4     
Flag Coverage Δ *Carryforward flag
access-control-service 70.00% <ø> (ø)
agent-service 76.76% <ø> (ø) Carriedforward from e4df21c
amber 66.65% <75.00%> (-0.01%) ⬇️
computing-unit-managing-service 17.72% <ø> (ø)
config-service 52.30% <ø> (ø)
file-service 66.53% <0.00%> (-0.28%) ⬇️
frontend 69.25% <37.50%> (-0.04%) ⬇️
notebook-migration-service 78.94% <ø> (ø)
pyamber 91.53% <100.00%> (+<0.01%) ⬆️
workflow-compiling-service 55.14% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 6 worse · ⚪ 9 noise (<±5%) · 0 without baseline

Compared against main f722c26 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 445 0.272 21,561/32,802/32,802 us 🔴 -9.0% / 🔴 +128.4%
bs=100 sw=10 sl=64 948 0.578 105,133/130,351/130,351 us ⚪ within ±5% / 🔴 +31.5%
🔴 bs=1000 sw=10 sl=64 1,118 0.682 889,048/988,623/988,623 us 🔴 +5.4% / ⚪ within ±5%
Baseline details

Latest main f722c26 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 445 tuples/sec 489 tuples/sec 831.47 tuples/sec -9.0% -46.5%
bs=10 sw=10 sl=64 MB/s 0.272 MB/s 0.298 MB/s 0.507 MB/s -8.7% -46.4%
bs=10 sw=10 sl=64 p50 21,561 us 21,371 us 11,844 us +0.9% +82.0%
bs=10 sw=10 sl=64 p95 32,802 us 31,001 us 14,359 us +5.8% +128.4%
bs=10 sw=10 sl=64 p99 32,802 us 31,001 us 18,888 us +5.8% +73.7%
bs=100 sw=10 sl=64 throughput 948 tuples/sec 923 tuples/sec 1,089 tuples/sec +2.7% -12.9%
bs=100 sw=10 sl=64 MB/s 0.578 MB/s 0.563 MB/s 0.665 MB/s +2.7% -13.0%
bs=100 sw=10 sl=64 p50 105,133 us 106,212 us 92,503 us -1.0% +13.7%
bs=100 sw=10 sl=64 p95 130,351 us 136,400 us 99,121 us -4.4% +31.5%
bs=100 sw=10 sl=64 p99 130,351 us 136,400 us 112,422 us -4.4% +15.9%
bs=1000 sw=10 sl=64 throughput 1,118 tuples/sec 1,121 tuples/sec 1,117 tuples/sec -0.3% +0.1%
bs=1000 sw=10 sl=64 MB/s 0.682 MB/s 0.684 MB/s 0.682 MB/s -0.3% +0.0%
bs=1000 sw=10 sl=64 p50 889,048 us 888,167 us 910,046 us +0.1% -2.3%
bs=1000 sw=10 sl=64 p95 988,623 us 937,725 us 950,716 us +5.4% +4.0%
bs=1000 sw=10 sl=64 p99 988,623 us 937,725 us 985,832 us +5.4% +0.3%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,449.37,200,128000,445,0.272,21560.50,32801.82,32801.82
1,100,10,64,20,2110.49,2000,1280000,948,0.578,105133.35,130351.05,130351.05
2,1000,10,64,20,17892.39,20000,12800000,1118,0.682,889048.22,988622.69,988622.69

The datasets logical-path prefix strips four leading segments
(datasets/owner/dataset/version); update the cover-image test's input
path to include the prefix so the extracted relative path is the file
name, not an empty string.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Yicong-Huang

Copy link
Copy Markdown
Contributor

I see that it has no behavioral change now, but the intention is to later introduce other source types prefixes such as models right? when we do so, it will eventually have a collision on the path without the prefix. how are we going to solve that data migration issue later?

A possible solution:

  • use "no-prefix" for dataset,
  • and require exact prefix for future assets/resources.

If you can have a better solution or data migration plan that would be better.

* Expected format: /ownerEmail/datasetName/versionName/fileRelativePath
*
* Two path shapes are accepted:
* - Prefixed: /datasets/ownerEmail/datasetName/versionName/fileRelativePath

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the new design, we will have a new table named "model", in this regard, an unprefixed path can not be resolved because the prefix indicate the table, please update implementation.

@tanishqgandhi1908

Copy link
Copy Markdown
Contributor Author

Makes sense. Sure let me research a bit and get back

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common feature frontend Changes related to the frontend GUI platform Non-amber Scala service paths pyamber

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a resource-type prefix to logical paths

4 participants