Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support car-scope param #33

Closed
4 tasks done
Tracked by #754 ...
olizilla opened this issue Apr 12, 2023 · 1 comment
Closed
4 tasks done
Tracked by #754 ...

Support car-scope param #33

olizilla opened this issue Apr 12, 2023 · 1 comment

Comments

@olizilla
Copy link
Contributor

olizilla commented Apr 12, 2023

?car-scope=block|file|all lets the user state if they want a single block , or the subset of blocks needed for a file or complete dir listing (could be single unixfs directory block or all hamt shards required to construct dir list) or all the blocks for the dag (traditional flavour, how freeway works today for CAR responses)... note that #32 still applies and CARs should also contain all additional blocks required to verify any path provided, regardless of car-scope.

?car-scope=block|file|all supersedes ?depth=0|1|all. Both may need to be supported in the short term but car-scope is where the work is going.

see: ipfs/bifrost-gateway#80
see: filecoin-saturn/L1-node#324 (comment) (note lassie has implemented block as root currently.

For context, from the Lassie car spec (which then goes on to describe the params in terms of depthType which seems to be another previous iteration on this)

All traversals will include the content at the termination of the path specifier, as well as all blocks from the root CID to the path terminus where a path is provided. At the termination of a provided path, or from the root CID where no path is provided, the depth of the DAG returned may vary depending on the request or available data.
https://github.com/filecoin-project/lassie/blob/main/docs/CAR.md#dag-depth

Tasks

olizilla added a commit to web3-storage/dagula that referenced this issue Apr 17, 2023
add getPath method as a generator that returns blocks for the targeted dag and all blocks traversed while resolving a cid+path string

supports carScope to specify what blocks to return for the resolved dag
- 'all': return the entire dag starting at path. (default)
- 'block': return the block identified by the path.
- 'file': Mimic gateway semantics: Return All blocks for a multi-block file or just enough blocks to enumerate a dir/map but not the dir contents.

see: web3-storage/freeway#33
see: web3-storage/freeway#34

TODO:
- [] find out how to identify the boundaries of a unixfs hamt (unixfs-exported seems to define it as "not having an empty or null Link.Name after the first 2 chars are stripped, which seems risky... what happens if the actual dir listing has 2 char long link names? see: https://github.com/ipfs/js-ipfs-unixfs/blob/e853049bd63d6773442e1540ae49b6a443ca8672/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/hamt-sharded-directory.ts#L20-L42

License: MIT
Signed-off-by: Oli Evans <oli@protocol.ai>
olizilla added a commit to web3-storage/dagula that referenced this issue May 1, 2023
add getPath method as a generator that returns blocks for the targeted
dag and all blocks traversed while resolving a cid+path string

supports carScope to specify what blocks to return for the resolved dag
- `'all'`: return the entire dag starting at path. (default)
- `'block'`: return the block identified by the path.
- `'file'`: Mimic gateway semantics: Return All blocks for a multi-block
file or just enough blocks to enumerate a dir/map but not the dir
contents.

see: web3-storage/freeway#33
see: web3-storage/freeway#34
see: ipfs/specs#402

TODO:
- [x] find out how to identify the boundaries of a unixfs hamt 

...unixfs-exporter seems to define it as "not having an empty or null
Link.Name after the first 2 chars are stripped, which seems loose...
what happens if the actual dir listing has 2 char long link names? see:
https://github.com/ipfs/js-ipfs-unixfs/blob/e853049bd63d6773442e1540ae49b6a443ca8672/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/hamt-sharded-directory.ts#L20-L42

License: MIT

---------

Signed-off-by: Oli Evans <oli@protocol.ai>
Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
olizilla added a commit to web3-storage/gateway-lib that referenced this issue May 1, 2023
- update dagula to get `getPath` with carScope support https://github.com/web3-storage/dagula/releases/tag/v6.0.0
- update handleCar to extract ?car-scope query and use `dagula.getPath`

BREAKING CHANGE: CARs returned for cid+path will now be rooted at the root cid rather than the resovled cid for the end of the path and include all blocks needed to verify the path was traveresed correctly.

see: web3-storage/freeway#33
see: web3-storage/freeway#34

License: MIT
Signed-off-by: Oli Evans <oli@protocol.ai>
olizilla added a commit to web3-storage/gateway-lib that referenced this issue May 2, 2023
- update dagula to get `getPath` with carScope support
https://github.com/web3-storage/dagula/releases/tag/v6.0.0
- update handleCar to extract ?car-scope query and use `dagula.getPath`

BREAKING CHANGE: CARs returned for cid+path will now be rooted at the
root cid rather than the resovled cid for the end of the path and
include all blocks needed to verify the path was traveresed correctly.

see: web3-storage/freeway#33
see: web3-storage/freeway#34

License: MIT

---------

Signed-off-by: Oli Evans <oli@protocol.ai>
@olizilla
Copy link
Contributor Author

olizilla commented May 3, 2023

Completed in #35
Deployed as #36

@olizilla olizilla closed this as completed May 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant