Create js-kubo-rpc-client library #9125

lidel · 2022-07-19T13:41:52Z

ETA: 2022-12-16

Part of #8959 – based on discussions from IPFS Thing 2022
GO counterpart: #9124

Current state

Kubo shares RPC client with JS-IPFS:

https://www.npmjs.com/package/ipfs-http-client

After renaming GO-IPFS to Kubo, this state of things is arguably a bad user experience:

js-ipfs and Kubo are joined at the RPC hip – unable to improve their respective RPC interfaces, and pretending to be interoperable:
- js-ipfs provides gRPC endpoint while Kubo does not
- we don't test all RPC command because implementations diverged already
- making change to RPC in Kubo forces us to do the same change in js-ipfs, and makes it extremely hard to coordinate releases to minimize breaking changes to rpc client library users (example: fix: multibase in pubsub http rpc #8183 & fix(pubsub): multibase in pubsub http rpc js-ipfs#3922)

Desired state

Ideally, we should have one Kubo client at https://github.com/ipfs/js-kubo-rpc-client and kubo-rpc-client on NPM.

Keeping go- prefix is important because we want consistency across languages,
and Kubo team will maintain go-kubo-rpc-client too.

TODO

Below is a broad strokes plan, feel free to adjust if something else makes more sense:

Followups

Below are good improvements, but since we didn't have them with ipfs-http-client, we're not requiring them here.

CI test that fails when a new command is added to Kubo and is not supported by kubo-rpc-client yet
- trackled separately in Automation to detect new Kubo commands js-kubo-rpc-client#47

cc @2color @BigLep @achingbrain @SgtPooki

The text was updated successfully, but these errors were encountered:

SgtPooki · 2022-08-18T19:37:14Z

Created https://github.com/ipfs/js-kubo-rpc-client.
Used https://support.github.com/contact?tags=rr-forks&subject=Detach%20Fork&flow=detach_fork to detach the forked repo from ipfs/js-ipfs

SgtPooki · 2022-08-18T23:15:01Z

Trying to generate documentation, but I can't seem to do so and can't find instructions at https://github.com/ipfs/js-ipfs

It seems like aegir doesn't support generating docs anymore (aegir docs command no longer exists), and they used to use typedoc, so I'm going to use https://github.com/TypeStrong/typedoc directly

Action is running right now: https://github.com/ipfs/js-kubo-rpc-client/actions/runs/2886171949

Updated the action, ipfs/js-kubo-rpc-client@cc2823b, and the site is now visible at https://ipfs.github.io/js-kubo-rpc-client/

SgtPooki · 2022-08-19T19:17:13Z

Created PR to add js-kubo-rpc-client to ipfs/github-mgmt#43

SgtPooki · 2022-08-19T19:26:32Z

Using personal NPM_TOKEN for publish js-kubo-rpc-client to npm automatically with github action via aegir until I get access to npm service account to create more permanent NPM_TOKEN

SgtPooki · 2022-08-19T19:39:16Z

Unified CI should be enabled for js-kubo-rpc-client once protocol/.github#371 and ipfs/github-mgmt#43 are merged

SgtPooki · 2022-08-19T19:47:02Z

Updated NPM_TOKEN to use automated token from service account

SgtPooki · 2022-08-19T19:51:00Z

update https://github.com/ipfs/js-ipfsd-ctl to support both ipfsHttpModule and kuboRpcModule
(we will also rename binary package, but that is tracked separately in ipfs/npm-kubo#51)

Starting work on this now.

SgtPooki · 2022-08-19T20:10:20Z

published https://www.npmjs.com/package/js-kubo-rpc-client so I can bring the lib into https://github.com/ipfs/js-ipfsd-ctl

see ipfs/kubo#9125

SgtPooki · 2022-08-23T18:37:27Z

decide if js-kubo-rpc-client CI should reuse tests against Core APIs interface-ipfs-core

@lidel @BigLep @aschmahmann, who should I talk to about this? Which stakeholders make the final call for this item?

SgtPooki · 2022-08-23T18:49:30Z

Created milestone for tracking each task: https://github.com/ipfs/js-kubo-rpc-client/milestone/1

BigLep · 2022-08-23T18:58:23Z

decide if js-kubo-rpc-client CI should reuse tests against Core APIs interface-ipfs-core

@lidel @BigLep @aschmahmann, who should I talk to about this? Which stakeholders make the final call for this item?

@SgtPooki : We need to make sure that kubo (RPC server) and the js-kupo-rpc-client can interopoerate and that we don't break compatibility between them. We should take whatever tests we had in the past to bootstrap our efforts, but we no longer need to adhere to the "Core API" concept. Basically, I would write tests that invoke the js-kubo-rpc client directly, rather than through legacy "Core API" interfaces.

SgtPooki · 2022-08-23T23:51:26Z

So it sounds like Kubo RPC Client should have it's own tests, stripped out from interface-ipfs-core. And we should only bring over the tests that are applicable for kubo's supported RPC calls.

I can make sense of that I think. Thanks

BigLep · 2022-09-07T20:00:31Z

@SgtPooki and @lidel : question on ipfs/interop when this work is done. Will ipfs/interop assert that a kubo node and a js-ipfs node can continue to interact with each other (e.g., discover each other, transport data)? Will the way that ipfs/interop accomplish this by orchestrating the kubo node using the js-kubo-rpc-client and the js-ipfs node using ipfs-http-client?

lidel · 2022-09-07T21:03:08Z

I think for now, we want to do a surgical swap: drop-in replace of ipfs-http-client with kubo-rpc-client when the backend node is Kubo, but all existing tests remain and should pass.

lidel · 2022-09-22T20:14:40Z

Seems we are back to the underlying question if both ipfs-http-client and kubo-rpc-client should conform to the same JS Interface (Core API) or not.

My understanding is:

Iroh does not implement Kubo RPC at all, and there are no plans to do that.
- The only interface other implementations use for benchmarking is HTTP Gateway.
One of the reasons why we were asked to rename go-ipfs to Kubo was to make it clear that RPC at /api/v0/ is Kubo-specific, and other implementations are free to go their own way, create own APIs.
Until now, Core API in JS-IPFS was forced to conform to whatever Kubo legacy existed.
- Creation of kubo-rpc-client aims at removing this co-dependency, allowing Kubo RPC and JS-IPFS RPC to go their own ways.
- Someone will ask: if these are not distinct creatures, then why did we create kubo-rpc-client in the first place?

At the same time we have years of legacy code that assumes both projects expose the same API.
Issue at hand with js-ipfsd-ctl is only a symptom of a problem that we will see over and over again.
Sadly, this is one of the reasons why Stewards tried to postpone the split and rename for so long – it is a ton of work and bad DX for preexisting users.

Some thoughts on minimizing the pain

Accounting for the legacy we live with, and being pragmatic (Ideally, we don't want to spend the rest of 2022 on this), I see three ways going forward, would appreciate sanity check from @achingbrain: how feasible (and painful) below are?

Make kubo-rpc-client conform to Core API types from JS-IPFS.
- We admit defeat for now, and keep them joined at a hip, but the roles are reversed and it is JS-IPFS wilggling the kubo-rpc-client from now.
- Main win is that nobody has to refactor 999 projects (incl. dragon lairs like ipfs/interop)
Create distinct types for kubo-rpc-client but keep using ipfsd-ctl
- This may be the least painful way to do things, complexity and refactor-wise.
- Migration plan: js-ipfsd-ctl exposes existing JS API as api but when node type is go, kubo-rpc-client is used for talking to Kubo, and api is replaced with a stubs that throw error asking people to use api from kuboRpc field instead.
  - Good: there is no silent breakage caused by api having different behavior when Kubo is used: this ensures GO people are forced to refactor their code to switch to kuboRpc field and Kubo types ( api field's types remain as Core API).
  - Bad: every internal app that uses ipfsd-ctl with Kubo daemon will have to be refactored (incl. dragon lairs like ipfs/interop)
Create kubod-ctl
- Same as 2, but with more work: adds maintenance burden of a separate package.
- We could update ipfsd-ctl to throw error when go type is used, pointing people at this new package

BigLep · 2022-09-23T05:56:15Z

Thanks for the comments here @lidel. A couple of thoughts.

Please huddle to sync synchronously if it would help for getting decisions made. Maybe the js-ipfs triage time could be used (or afterwards since that is early for @SgtPooki ).
This probably goes without saying, but just so it's clear what I care about, I don't care as much about having a JS Kubo RPC client that is named as such (although I do think it's ideal for users to have a name that's clear). My chief concern is allowing the Kubo team to own their RPC client without any considerations for js-ipfs and vice versa. Decoupling is the key. If in the process, it's just as easy to rename, then great - please do it. If renaming anything is causing pain, please feel free to skip. I want to optimize for future developer velocity more so than user clarity.

achingbrain · 2022-09-23T14:40:50Z

In js-ipfs triage we have come up with a 4th option:

The neat trick of the Core API is that it's supported by js-ipfs in-process and both js-ipfs and Kubo over http. We have leveraged this to write the interop suite which is a) massively useful in proving the two implementations can talk to each other and b) about to cause significant type-related pain now that we want to allow the RPC APIs to diverge.

The 4th option is to just type the parts of the API that the interop suite uses, store those types in the interop suite and have it use them instead of the types from the Core API.

The theory is that both kubo-rpc-client and ipfs-http-client will have overlap with these types so tsc should be happy with them, and then both implementations (just Kubo really since js-ipfs is getting No New Features:tm:) are free to innovate in other areas of the API.

When Kubo breaks the API so badly that the interop suite types are no longer valid, hopefully we'll have the shiny new version of IPFS-on-js that will have it's own interop suite that maybe orchestrates kubo via some sort of API external to it a la go-libp2p-daemon's simple protobuf-based API and doesn't depend on the RPC client at all.

SgtPooki · 2022-09-23T17:07:49Z

type the parts of the API that the interop suite uses, store those types in the interop suite and have it use them instead of the types from the Core API.

Excellent. The interop work living in the interop package makes the most sense and should keep us all unblocked for the longest period. Type augmentations should help keep the diverging types manageable.

* feat: add js-kubo-rpc-client see ipfs/kubo#9125 * chore: update kubo-rpc-client package name * fix: use cross-env to set USE_KUBO_JS * fix: Remove USE_KUBO_JS env var * chore(deps): remove cross-env * docs(readme): update npm install cmds for using kubo-rpc-client * chore: log whether kubo-rpc-client or ipfs-http-client is in use * chore(tests): use kubo-rpc-client for commmunication with go controllers * chore(lint): fix lint failure * fix: addCorrectRpcModule returns the additionalOpts object * docs(readme): correct kubo-rpc-client name Co-authored-by: Alex Potsides <alex@achingbrain.net> * fix: use correct logger namespace convention * fix: move kubo-rpc-client to devDeps * fix: support generics for Controller type * chore(deps): use kubo-rpc-client@1.0.1 * feat!: convert to typescript Converts this module to typescript * chore: fix linting * chore: fix tests * chore: apply suggestions from code review * fix: resolve issues with js-kubo-rpc-client in typescript * fix: use kuboRpcModule for go in factory.spec.ts * fix: broken build from bad merge * chore: address PR comments Co-authored-by: Alex Potsides <alex@achingbrain.net>

BigLep · 2022-11-19T00:08:47Z

FYI I made a "CI test that fails when a new command is added to Kubo and is not supported by kubo-rpc-client yet" as followup that isn' required for closing this issue since we didn't have that with the ipfs-http-client.

SgtPooki · 2022-12-05T18:46:47Z

Note that the latest work to close out the last action item in this issue (ipfs/js-kubo-rpc-client#5) is being handled via pull request ipfs/js-kubo-rpc-client#83.

I'll be meeting with @hacdias tomorrow (2022-12-06) to go over some of these tests and confirm next steps and handoff requirements.

Note that the original interface tests were testing against go-ipfs@0.12, and so the done criteria for closing this issue does not include updating interface tests to 0.17.

ninja-edit: There are technically two last items as seen on the milestone tracking the closing of this issue: https://github.com/ipfs/js-kubo-rpc-client/milestone/1, but the "revert patch-only release changes" is trivial so was not mentioned in the original comment.

SgtPooki · 2022-12-07T05:24:41Z

Latest actions required prior to handoff: ipfs/js-kubo-rpc-client#83 (comment)

SgtPooki · 2022-12-15T22:40:05Z

Note that ipfs/js-kubo-rpc-client#83 has been merged!

SgtPooki · 2022-12-16T00:37:50Z

We should be good to close this. will defer to @BigLep and @lidel

BigLep · 2022-12-17T07:04:06Z

Thanks @SgtPooki for bringing this to this point. The done criteria are satisfied, thus I think it's reasonable to close this out. I know there are additional tasks to do to further harden js-kubo-rpc-client and those are in ipfs/js-kubo-rpc-client#56

Closing - thanks again!

2color · 2023-01-13T12:26:33Z

With this issue closed, should we point all users of https://github.com/ipfs/js-ipfs/tree/master/packages/ipfs-http-client to this?

The NPM page could also use a notice: https://www.npmjs.com/package/ipfs-http-client

achingbrain · 2023-01-13T12:53:43Z

ipfs-http-client is still used to connect to js-ipfs, so not all users, but it could use a message saying to use the kubo rpc client with kubo.

lidel added the kind/maintenance Work required to avoid breaking changes or harm to project's status quo label Jul 19, 2022

This was referenced Jul 19, 2022

Point users at canonical kubo/client/rpc library #9124

Closed

Minimal rename of go-ipfs in 2022Q2 #8959

Closed

List other Kubo RPC API clients ipfs/ipfs-docs#1194

Closed

BigLep mentioned this issue Jul 24, 2022

[Tracking Issue] Cleanup Kubo RPC Client Story #9142

Open

3 tasks

SgtPooki self-assigned this Aug 18, 2022

SgtPooki added a commit to SgtPooki/js-ipfsd-ctl that referenced this issue Aug 19, 2022

feat: add js-kubo-rpc-client

20ea2f9

see ipfs/kubo#9125

SgtPooki mentioned this issue Aug 19, 2022

feat: add js-kubo-rpc-client ipfs/js-ipfsd-ctl#764

Merged

This was referenced Sep 2, 2022

Looking for a maintainer familiar with Python! 🙏 ipfs-shipyard/py-ipfs-http-client#316

Closed

Looking for a maintainer familiar with Java! 🙏 ipfs-shipyard/java-ipfs-http-client#194

Closed

This was referenced Oct 5, 2022

docs: ipfs-http-client -> kubo-rpc-client #9331

Merged

docs: mention kubo-rpc-client ipfs/ipfs-docs#1299

Merged

feat: use kubo-rpc-client ipfs-examples/js-ipfs-examples#517

Merged

tinytb added kind/enhancement A net-new feature or improvement to an existing feature effort/weeks Estimated to take multiple weeks labels Oct 21, 2022

SgtPooki mentioned this issue Nov 3, 2022

webui and ipfs-desktop do depend on js-ipfs/js-ipfs-http-api ipfs/js-ipfs#2542

Closed

SgtPooki mentioned this issue Nov 28, 2022

IPFS Desktop and Webui are decoupled from js-ipfs ipfs/ipfs-gui#107

Open

SgtPooki added the starmaps label Nov 28, 2022

BigLep assigned hacdias Dec 1, 2022

This was referenced Dec 16, 2022

feat: kubo 0.18 with /quic-v1 and /webtransport ipfs/ipfs-desktop#2363

Merged

ipfs-http-client migration: Additional tasks ipfs/js-kubo-rpc-client#56

Open

BigLep closed this as completed Dec 17, 2022

lidel mentioned this issue Dec 18, 2022

Breaks with Kubo 0.18 when ipfs.swarm.peers returns /quic-v1 ipfs/ipfs-companion#1118

Closed

SgtPooki mentioned this issue Jan 12, 2023

feat!: replace ipfs-http-client with kubo-rpc-client ipfs/ipfs-webui#2079

Closed

2color mentioned this issue Jan 13, 2023

Interplanetary Apps: Disco Chat - Blog Post ipfs/ipfs-blog#517

Merged

3 tasks

2color mentioned this issue Jan 13, 2023

docs: update-ipfs-http-client ipfs/js-ipfs#4293

Merged

rmlearney-digicatapult mentioned this issue Feb 25, 2024

README.md out of keeping with link to WIP milestone? ipfs/js-kubo-rpc-client#216

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create js-kubo-rpc-client library #9125

Create js-kubo-rpc-client library #9125

lidel commented Jul 19, 2022 •

edited by SgtPooki

Loading

SgtPooki commented Aug 18, 2022 •

edited

Loading

SgtPooki commented Aug 18, 2022 •

edited

Loading

SgtPooki commented Aug 19, 2022

SgtPooki commented Aug 19, 2022 •

edited

Loading

SgtPooki commented Aug 19, 2022

SgtPooki commented Aug 19, 2022

SgtPooki commented Aug 19, 2022

SgtPooki commented Aug 19, 2022

SgtPooki commented Aug 23, 2022

SgtPooki commented Aug 23, 2022

BigLep commented Aug 23, 2022

SgtPooki commented Aug 23, 2022

BigLep commented Sep 7, 2022 •

edited

Loading

lidel commented Sep 7, 2022 •

edited

Loading

lidel commented Sep 22, 2022 •

edited

Loading

BigLep commented Sep 23, 2022

achingbrain commented Sep 23, 2022

SgtPooki commented Sep 23, 2022

BigLep commented Nov 19, 2022

SgtPooki commented Dec 5, 2022 •

edited

Loading

SgtPooki commented Dec 7, 2022

SgtPooki commented Dec 15, 2022

SgtPooki commented Dec 16, 2022

BigLep commented Dec 17, 2022

2color commented Jan 13, 2023

achingbrain commented Jan 13, 2023

Create js-kubo-rpc-client library #9125

Create js-kubo-rpc-client library #9125

Comments

lidel commented Jul 19, 2022 • edited by SgtPooki Loading

Current state

Desired state

TODO

Followups

SgtPooki commented Aug 18, 2022 • edited Loading

SgtPooki commented Aug 18, 2022 • edited Loading

SgtPooki commented Aug 19, 2022

SgtPooki commented Aug 19, 2022 • edited Loading

SgtPooki commented Aug 19, 2022

SgtPooki commented Aug 19, 2022

SgtPooki commented Aug 19, 2022

SgtPooki commented Aug 19, 2022

SgtPooki commented Aug 23, 2022

SgtPooki commented Aug 23, 2022

BigLep commented Aug 23, 2022

SgtPooki commented Aug 23, 2022

BigLep commented Sep 7, 2022 • edited Loading

lidel commented Sep 7, 2022 • edited Loading

lidel commented Sep 22, 2022 • edited Loading

Some thoughts on minimizing the pain

BigLep commented Sep 23, 2022

achingbrain commented Sep 23, 2022

SgtPooki commented Sep 23, 2022

BigLep commented Nov 19, 2022

SgtPooki commented Dec 5, 2022 • edited Loading

SgtPooki commented Dec 7, 2022

SgtPooki commented Dec 15, 2022

SgtPooki commented Dec 16, 2022

BigLep commented Dec 17, 2022

2color commented Jan 13, 2023

achingbrain commented Jan 13, 2023

lidel commented Jul 19, 2022 •

edited by SgtPooki

Loading

SgtPooki commented Aug 18, 2022 •

edited

Loading

SgtPooki commented Aug 18, 2022 •

edited

Loading

SgtPooki commented Aug 19, 2022 •

edited

Loading

BigLep commented Sep 7, 2022 •

edited

Loading

lidel commented Sep 7, 2022 •

edited

Loading

lidel commented Sep 22, 2022 •

edited

Loading

SgtPooki commented Dec 5, 2022 •

edited

Loading