Compilation error setting up dag-jose ipld format in newest js-ipfs

[stbrody]

• Version:
  ipfs-core: "^0.5.2"

• Platform:
  Linux Sonny 5.8.0-43-generic config #49~20.04.1-Ubuntu SMP Fri Feb 5 09:57:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

• Subsystem:
  ipld

Severity:

High

Description:

I'm trying to upgrade our code from using ipfs-core 0.3.0 to 0.5.2, but I'm having compilation failure when trying to instantiate a new IPFS instance with the dagJose IPLD format.

The original version of our code that worked before the upgrade was:

import multiformats from 'multiformats/basics'
import legacy from 'multiformats/legacy'
import dagJose from 'dag-jose'
import IPFS from 'ipfs-core'
import getPort from 'get-port'
import tmp from 'tmp-promise'

multiformats.multicodec.add(dagJose)
const format = legacy(multiformats, dagJose.name)

const tmpFolder = await tmp.dir({ unsafeCleanup: true });
const port = await getPort();

const config = {
  ipld: { formats: [format] },
  repo: `${tmpFolder.path}/ipfs${port}/`,
  config: {
    Addresses: { Swarm: [`/ip4/127.0.0.1/tcp/${port}`] },
    Bootstrap: [],
  },
};

const instance = await IPFS.create(config);

After the upgrade, that code started failing to compile with an error:

src/__tests__/ipfs-util.ts(29,38): error TS2345: Argument of type '{ ipld: { formats: any[]; }; repo: string; config: { Addresses: { Swarm: string[]; }; Bootstrap: any[]; }; }' is not assignable to parameter of 
type 'Options'.
  Type '{ ipld: { formats: any[]; }; repo: string; config: { Addresses: { Swarm: string[]; }; Bootstrap: any[]; }; }' is not assignable to type 'StorageOptions'.
    The types of 'ipld.formats' are incompatible between these types.
      Type 'any[]' is not assignable to type 'Record<string, Format<any>>'.
        Index signature is missing in type 'any[]'.

So I went to the ipfs docs and found this page on how to use a custom ipld format. So I tried to mimic that approach to setting the format. My new code looks like:

import dagJose from 'dag-jose'
import IPFS from 'ipfs-core'
import getPort from 'get-port'
import tmp from 'tmp-promise'
import multicodec from 'multicodec'
import multihashing from 'multihashing-async'

const format = {
  codec: dagJose.code,
  defaultHashAlg: multicodec.SHA2_256,
  util: {
    serialize: dagJose.encode,
    deserialize: dagJose.decode,
    async cid(buf) {
      const multihash = await multihashing(buf, format.defaultHashAlg)
      return new CID(1, format.codec, multihash)
    }
  }
}

const tmpFolder = await tmp.dir({ unsafeCleanup: true });
const port = await getPort();

const config = {
  ipld: { formats: [format] },
  repo: `${tmpFolder.path}/ipfs${port}/`,
  config: {
    Addresses: { Swarm: [`/ip4/127.0.0.1/tcp/${port}`] },
    Bootstrap: [],
  },
};

const instance = await IPFS.create(config);

Trying to compile that, however, I get the similar error message:

src/__tests__/ipfs-util.ts(52,38): error TS2345: Argument of type '{ ipld: { formats: { codec: number; defaultHashAlg: CodecCode; util: { serialize: (obj: string | DagJWS | DagJWE) => Uint8Array; deserialize: (d
ata: Uint8Array) => DagJWS | DagJWE; cid(buf: any): Promise<...>; }; }[]; }; repo: string; config: { ...; }; }' is not assignable to parameter of type 'Options'.
  Type '{ ipld: { formats: { codec: number; defaultHashAlg: CodecCode; util: { serialize: (obj: string | DagJWS | DagJWE) => Uint8Array; deserialize: (data: Uint8Array) => DagJWS | DagJWE; cid(buf: any): Promise
<...>; }; }[]; }; repo: string; config: { ...; }; }' is not assignable to type 'StorageOptions'.
    The types of 'ipld.formats' are incompatible between these types.
      Type '{ codec: number; defaultHashAlg: CodecCode; util: { serialize: (obj: string | DagJWS | DagJWE) => Uint8Array; deserialize: (data: Uint8Array) => DagJWS | DagJWE; cid(buf: any): Promise<...>; }; }[]' 
is not assignable to type 'Record<string, Format<any>>'.
        Index signature is missing in type '{ codec: number; defaultHashAlg: CodecCode; util: { serialize: (obj: string | DagJWS | DagJWE) => Uint8Array; deserialize: (data: Uint8Array) => DagJWS | DagJWE; cid(b
uf: any): Promise<...>; }; }[]'.

Any suggestions on what I can do to get this to compile again?

[achingbrain]

I think you need to add additional methods in your shim to meet the Format definition - looks like the resolver methods are missing.

[stbrody]

I think you need to add additional methods in your shim to meet the Format definition - looks like the resolver methods are missing.

Ah, thank you @achingbrain! Are there any examples or documentation of what those resolver methods are supposed to do/look like? They don't show up in the example for adding a custom IPLD format

[stbrody]

Nevermind, I found https://github.com/ipld/interface-ipld-format#local-resolver-methods for docs and https://github.com/ipld/js-ipld-dag-cbor/blob/master/src/resolver.js as an example implementation. I can probably use those to get started.

@oed looks like we probably want to add these new resolver functions directly to https://github.com/ceramicnetwork/js-dag-jose

[stbrody]

@oed and I synced up and we think that multiformats/legacy should be able to convert our dagJose format into the form that ipfs expects. My previous example was trying to manually coerce the dagJose format into the legacy format that IPFS expects. Here's what we have now utilizing multiformats/legacy from the most up to date version of the multiformats package.

import dagJose from 'dag-jose';
import { sha256 } from 'multiformats/hashes/sha2'
import legacy from 'multiformats/legacy'
import IPFS from 'ipfs';

async function createIPFS() {
  const hasher = {}
  hasher[sha256.code] = sha256
  const format = legacy(dagJose, {hashes: hasher})

  const conf = {
    ipld: { formats: [format] },
  };

  const instance = await IPFS.create(conf);

  return instance
}

When we compile this we get the error:

src/__tests__/ipfs-util.ts:15:38 - error TS2345: Argument of type '{ ipld: { formats: { defaultHashAlg: string; codec: number; util: { serialize: (o: string | DagJWS | DagJWE) => Buffer; deserialize: (b: Uint8Ar
ray) => string | DagJWS | DagJWE; cid: (buff: Buffer, opts?: { ...; }) => Promise<...>; }; resolver: { ...; }; }[]; }; repo: string; config: { ...; }; }' is not assignable to parameter of type 'Options'.
  Type '{ ipld: { formats: { defaultHashAlg: string; codec: number; util: { serialize: (o: string | DagJWS | DagJWE) => Buffer; deserialize: (b: Uint8Array) => string | DagJWS | DagJWE; cid: (buff: Buffer, opts?
: { ...; }) => Promise<...>; }; resolver: { ...; }; }[]; }; repo: string; config: { ...; }; }' is not assignable to type 'StorageOptions'.
    The types of 'ipld.formats' are incompatible between these types.
      Type '{ defaultHashAlg: string; codec: number; util: { serialize: (o: string | DagJWS | DagJWE) => Buffer; deserialize: (b: Uint8Array) => string | DagJWS | DagJWE; cid: (buff: Buffer, opts?: { ...; }) => 
Promise<...>; }; resolver: { ...; }; }[]' is not assignable to type 'Record<string, Format<any>>'.
        Index signature is missing in type '{ defaultHashAlg: string; codec: number; util: { serialize: (o: string | DagJWS | DagJWE) => Buffer; deserialize: (b: Uint8Array) => string | DagJWS | DagJWE; cid: (bu
ff: Buffer, opts?: { ...; }) => Promise<...>; }; resolver: { ...; }; }[]'.

15   const instance = await IPFS.create(defaultConfig);

I tried suppressing the compilation error with @ts-ignore and am now fighting errors trying to get our tests to pass, which I am still investigating, but I wanted to update you on what we're doing and see if you have any ideas about the compilation error when using multiformats/legacy to get the ipld format

[stbrody]

So to give a final update here, we were able to successfully complete the IPFS upgrade and everything seems to be working. We did, however, have to disable the compilation warning I posted above by adding // @ts-ignore before the await IPFS.create(conf) line. So please consider this a bug report that the typescript bindings for IPFS.create() are reporting a compilation error when in fact the config is actually well-formed and works fine.

[achingbrain]

@oed and I synced up and we think that multiformats/legacy should be able to convert our dagJose format into the form that ipfs expects

This may be the root cause of the problem here. The type IPFS is expecting is here but it looks like multiformats/legacy is not returning a similarly shaped object.

The next js-IPFS release will use the types from interface-ipld-format among other things which may make the above error go away, but it will also be a lot stricter so it may expose other incompatibilities.

[achingbrain]

I've opened multiformats/js-multiformats#67 to cover the conversion to the correct legacy IPLD format type.

Closing this as that will fix the issue for the next js-IPFS release without the need for the // @ts-ignore