Awesome Endeavour: repo interoperability

[vasco-santos]

Awesome Endeavour: Repo interoperability 🃏

This issue has the purpose of gathering and tracking everything related to the repo interoperability.
All the necessary steps for having a full interoperable repo betweem js-ipfs and go-ipfs are listed below.

✅ Tasks checklist

1) ipfs/specs

• 1. Update the specs with the current repo specs (go-ipfs)

2) ipfs/js-ipfs

• 1. Send the default/provided config (with datastore.spec)
• 2. Present errors provided by js-ipfs-repo (have tests)
  • INIT: if the configuration is not compatible to what repo knows
  • START: if the repo is different from the spec
• 3. js-ipfs should not test the repo version, it should be ipfs-repo who takes care about this. This is more problematic, as in JS land, repo v6 and repo v7 are the same.
• 4. Create an example

3) ipfs/js-ipfs-repo

• 1. Throw an error on invalid / inconsistent configs
  • INIT: create the repo based on the config received (verify if the storageBackends exists and if so, guarantee that it matches the spec). If is is not possible to create the repo an error should be thrown to inform that the options received are not valid.
  • Start: verify if the repo that exists is compliant with the configuration. If not, throw an error
• 2. transition for using spec instead of the current programatic approach (support both to avoid breaking existing projects, but deprecate the current format)
• 3. have configurations of each possible datastore listed clearly in their individual repos (allowing the users to get them to their config in a really easy manner)
• 4. remove storageBackends option in the future (deprecate)
• 5. go-ipfs uses Keystore and js-ipfs uses Keys

4) ipfs/interop

• 1. Have current interop tests passing and not skipped
• 2. Have more specific interop tests, in order to prevent future break of specs

Iterations

We will divide this endeavour in several iterations, as follows:

1) Unblock IPNS interoperability

• Have datastore and blocks mounting points fully interop

• Have mounting points configurable through the config file

• Have datastore_spec file with the mounting points specified

• Guarantee support for the datastore spec and the current storageBackends format to avoid breaking existing projects

• Handle possible errors regarding inconsistent / invalid configurations provided

• Tasks involved: 1.1, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1

2) Make repo configuration friendly

• Create an example like the existant one (storageBackends) to the config datastore usage

• Have configurations of each possible datastore listed clearly in their individual repos (allowing the users to get them to their config in a really easy manner)

• Tasks involved: 2.4, 3.3

3) Have a full interoperable repo

• Have an interop keys implementation

• Deprecate storageBackends

• Tasks involved: 3.4, 3.5, 4.2

Repo migration tool

As we arrive to the end of this endeavour, we must also have a repo migration tool for the repo, in order to allow a easy migration between older repo versions and the newer (and future) versions of the repo.

Useful links:

• fs-repo-migrations
• js-ipfs#1115

[vasco-santos]

cc @diasdavid @jacobheun

[achingbrain]

go-ipfs is adopting badger as a datastore and there's no js implementation (yet, I suppose) so I don't think this is going to get done. We can probably close this.

[daviddias]

Agreed. Repo interoperability became a non-goal. Instead, reuse the energy to optimize js-ipfs datastore