Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: switch to DAG-CBOR #93

Merged
merged 7 commits into from
Oct 19, 2022
Merged

refactor: switch to DAG-CBOR #93

merged 7 commits into from
Oct 19, 2022

Conversation

lidel
Copy link
Member

@lidel lidel commented Sep 6, 2022

This PR aims to close #85 and switch b-tree away from expensive JSON stored in Data field of dag-pb and deprecated object RPC API.

TLDR: ~30% DAG size reduction

DAG-PB was 399424 bytes (~400MiB)
DAG-CBOR is 281089143 bytes (~281MiB)

馃憠 this means ~30% reduction 馃憣

TODO

  • switch from ipfs.object.get to ipfs.block.get
  • switch from expensive ipfs.object.put to CAR writer
    • create a CAR with entire b-tree
    • mark final root node as the CAR root
  • remove use of DAG-PB and JSON serialized as string
  • create DAG-CBOR nodes
    • use CID CBOR tag in links so b-tree can be traversed by generic IPLD libraries and ipfs dag stat in Kubo
    • remove string fields that indicate leaf nodes, use presence of data and links to reason about node type (storage savings)
  • bump aegir and switch to ESM
  • switch from legacy JS libs to modern https://github.com/multiformats/js-multiformats/
  • regenerate dataset and confirm node example/lookup.js 9.9.9.9 returns same result
  • detect when ipfs is not CoreAPI instance, but a string, and if it is a valid gateway URL, use it for fetching raw blocks as application/vnd.ipld.raw
  • update source dataset and regenerate b-tree, so users not only get storage savings thanks to DAG-CBOR, but also get updated GeoIP database
  • cleanup add retry if gateway timeouts
  • solve issue with ip dependency requiring native node module

Tests

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
We now can fetch raw blocks from a public gateway.
This will use public gateway if no API is provided.

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
@lidel lidel force-pushed the refactor/dag-cbor branch 7 times, most recently from 84802ca to 76f1c2a Compare September 19, 2022 00:20
@lidel lidel force-pushed the refactor/dag-cbor branch 2 times, most recently from de35eff to 51c9e99 Compare September 28, 2022 01:27
License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
* Using async-await
* feat: support IPFS_GATEWAY env

ipfs/specs#280

Signed-off-by: Marcin Rataj <lidel@lidel.org>

Signed-off-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
@lidel lidel marked this pull request as ready for review October 19, 2022 17:50
@lidel
Copy link
Member Author

lidel commented Oct 19, 2022

I'm merging this to unblock @whizzzkid to clean up remaining items before we can release:

  1. remove dependency on os by doing the build step
  2. fix npm test in retry PR

When we have that, I'll regenerate dataset and make the release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch to dag-cbor
2 participants