Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exporting a CAR with @helia/car of a @helia/json encoded CID throws an error #375

Closed
2color opened this issue Jan 10, 2024 · 3 comments · Fixed by #381 or #386
Closed

Exporting a CAR with @helia/car of a @helia/json encoded CID throws an error #375

2color opened this issue Jan 10, 2024 · 3 comments · Fixed by #381 or #386

Comments

@2color
Copy link
Member

2color commented Jan 10, 2024

As per @achingbrain recommendation, I tried the @helia/json library to try out some simple utf-8 encoded JSON as an alternative to UnixFS.

The APIs are indeed nicer to work with(adding and getting CIDs is an async operation rather than an async iterator), but exporting a CAR fails with the following error:

Uncaught (in promise) Error: No dag walker found for cid codec 512
    at #walkDag (index.js:129:19)
    at eval (index.js:109:36)
    at eval (index.js:187:37)
    at #tryToStartAnother (index.js:125:17)
    at eval (index.js:211:36)
    at new Promise (<anonymous>)
    at PQueue.add (index.js:181:16)
    at DefaultCar.export (index.js:108:24)
    at createCarBlob (ipfs.tsx:118:20)
    at async eval (sign-page.tsx:51:18)

The source that produces this looks as follows

import { json as heliaJson } from '@helia/json'

export async function createCidForSignedMessage(
  helia: Helia,
  { message, signature, address }: SignedMessagePayload,
) {
  const j = heliaJson(helia)

  const cid = await j.add({
    address,
    message,
    signature,
  })

  return cid
}

export async function createCarBlob(helia: Helia, cid: CID) {
  const { writer, out } = await CarWriter.create(cid)
  car(helia).export(cid, writer)

  const carBlob = await carWriterOutToBlob(out)
  return carBlob
}

I tested this using @helia/dag-json and this works. So for now that's what I'm using since it gives the same return type.

@achingbrain
Copy link
Member

achingbrain commented Jan 10, 2024

In order to export a complete DAG as a CAR file, we need to know how to get a list of blocks that are linked to from the current block. To do this we need to know how to interpret the current block in order to extract and CIDs contained therein. @helia/car does this through DAGWalkers.

From the error message there's a CID with the codec 512 (0x200) in a block being parsed but no DAGWalker has been configured for that codec.

Helia comes with DAGWalkers for common codecs preconfigured (used for pinning operations), @helia/car uses it's own set of default DAGWalkers and doesn't include one for the JSON codec.

A short term fix would be to copy the JSON codec DAGWalker from Helia to @helia/car but really Helia should expose it's configured DAGWalkers and @helia/car should take those as the default set instead of having it's own.

@2color
Copy link
Member Author

2color commented Jan 10, 2024

but really Helia should expose it's configured DAGWalkers and @helia/car should take those as the default set instead of having it's own.

I had a look at the code, and it seems that the default dag walkers are instantiated by the PinsImpl on the main Helia object. To avoid duplication of the dag walkers, how would you suggest organising the code?

  • To extract the dag walkers into a separate module that would be imported by both the pins class and the CAR module?
  • Or perhaps to move the dag walkes to the root Helia object and for the CAR module to use it from the helia object?

Sorry for the novice questions, I'd love to help with a PR, but wanna make sure I follow the established practices for the repo.

@achingbrain
Copy link
Member

achingbrain commented Jan 12, 2024

Or perhaps to move the dag walkers to the root Helia object and for the CAR module to use it from the helia object?

This, I think - that way people can augment the list with their own configurations.

Sorry for the novice questions, I'd love to help with a PR, but wanna make sure I follow the established practices for the repo.

No problem!

achingbrain added a commit that referenced this issue Jan 16, 2024
To allow modules like `@helia/car` to re-use configured dag walkers
instead of having to bundle their own, expose configured dag walkers
and hashers on the helia interface and have them consume those.

Fixes #375
achingbrain added a commit that referenced this issue Jan 17, 2024
To allow modules like `@helia/car` to re-use configured dag walkers
instead of having to bundle their own, expose configured dag walkers
and hashers on the helia interface and have them consume those.

Fixes #375
achingbrain added a commit that referenced this issue Jan 17, 2024
)

To allow modules like `@helia/car` to re-use configured dag walkers
instead of having to bundle their own, expose configured dag walkers
and hashers on the helia interface and have them consume those.

Fixes #375
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Archived in project
2 participants