Prototype smoldot browser extension

[raoulmillais]

Introduction

Create an end-to-end prototype to get more insight into this:

@gavofyork:matrix.parity.io
so whether it's substrate-lite or substrate --light, i don't really care, but what it should be doing is running either:
in-browser (in a site's instance, this should allow completely new users to use the site without needing to go to (RPC) servers)
in-browser-extension (better, since it's stateful between restarts and across sites)

Theory

We can make a polkadot-js API provider that makes the use of an extension transparent to the application developer.

Proposed API :

import { ApiPromise } from '@polkadot/api';
import { SubtrateConnectProvider } from './';  // name up to debate :)

const provider = new SubstrateConnectProvider('westend');
await provider.connect();
const api = await ApiPromise.create({ provider });

Proposed flow:

Extension installed: Web page -> SmoldotConnectProvider -> extension -> SmoldotClientManager -> smoldot
Extension not installed: Webpage -> SmoldotConnectProvider -> SmoldotProvider -> smoldot

Aim

• Understand better the integation points
• Validate that this is possible.
• Get a feel for the proposed API. Is it too magic? Should we be more explicit about the detection?

Approach

Create a browser extension that is not configurable and has minimal UI that is used transparently by webpages instead of embedding a node when the extension is installed.

• Create an extension that runs a single Smoldot browser node (westend) - using the existing SmoldotProvider and PolkadotJS API.
• Create limited UI to show basic node status.
• The limited UI should follow the design guidelines in Figma
• The extension should expose an API for webpages to interact with the node.
• The API should give the impression that their is more than one possible node to interact with. I.e. JavaScript running on a page must specifiy the name of the node they intend to interact with.
• Create a wrapper provider that detects the extension and uses its API if it is installed otherwise falls back to using the existing SmoldotProvider which embeds a smoldot client in the current page.

What should the SmoldotClientManager do

The SmoldotClientManager inside the extension must broker RPC requests and responses from multiple clients residing on browser pages (aka content pages) to multiple smoldot blockchain clients in the extension (for now we have one but there will be many). This entails:

• The SmoldotProvider in each webpage will issue RPC requests with ids in its own namespace (starting from 1). The SmoldotClientManager must maintain a translation table of client request ids to the actual request ids and send the correct responses to the correct clients. The translation table will be a map of clients to ID mappings where an ID mapping is the originating request id and the actual request id sent to smoldot. The SmoldotClientManager will change the request and response IDs in the JSON on the fly and maintain the list of live mappings.
  • Remove the ID mapping when the response has been received for non-subscriptions
  • Remove the ID mapping when an unsubscribe request / response has happened
  • Remove all mappings when a page disconnects from the extension
• The SmoldotClientManager must detect when clients disconnect and send unsubscribe messages for all that clients' subscriptions.
• The SmoldotClientManager could also intelligently cache queries in the future (out of scope for now).

[tomaka]

For what it's worth, smoldot should be capable in the future of connecting to multiple chains by running only one client.

Parachains aren't launched yet, so this isn't implemented, but for example, in order to talk to parachain A, smoldot will need to be connected both to chain A and to Polkadot.
I'm not very familiar with parachains messaging, but in the future it might also be necessary for smoldot to connect to chains A, B, C, and Polkadot, just to be able to exploit A.

It would make sense to me to add the possibility of specifying multiple chains, or even the possibility to add or remove chains while it's running.

[raoulmillais]

If we have a PolkadotJS API instance on both sides; in the page and in the browser extension what happens for version mismatches? An extension has a long lifetime. It has to exist in a user's browser for many months and it has to have its own upgrade path. This presents challenges / quesions:

• How do we deal with upgrading the API package in the extension?
• How do we manage the fact that different pages might have different versions of the API.?
• What are the strategies for dealing with a version mismatch when a page detects a different API version in the extension from the one loaded on the page?
• Does the extension need to expose to pages what API version it is running?
• When upgrading the extension with a new API version should it have a record of which pages have used it and warn the user?

The reasons for us to want to upgrade an extension will naturally be different from those for the PolkadotJS API and the benefits of having the PolkadotJS API in the extension are limited. (strawman - what are the benefits? - let me know if you can think of some).

Case for having a different abstraction inside the extension

• I initially thought the PolkadotJS API did more caching than it does - it actually only caches derive queries.
• How and what to cache are going to be different to the Polkadot JS API which was only ever supposed to live in page. We can make independent decsions about query caching for example.
• Our reasons for caching will be to optimise for several different clients on potentially different pages making the same or similar queries.
• We only upgrade if we want to upgrade smoldot or change / improve the extension not for the PJS API changing

Change in scope for the prototype

The same but no SmoldotProvider inside the extension - a new SmoldotClientManager

BEFORE:

Proposed flow:

Extension installed: Web page -> SmoldotConnectProvider -> extension -> SmoldotProvider -> smoldot
Extension not installed: Webpage -> SmoldotConnectProvider -> SmoldotProvider -> smoldot

AFTER:

Proposed flow:

Extension installed: Web page -> SmoldotConnectProvider -> extension -> SmoldotClientManager -> smoldot
Extension not installed: Webpage -> SmoldotConnectProvider -> SmoldotProvider -> smoldot

[raoulmillais]

This proposed SmoldotClientManager thingy that will live in the extension is a departure in the way smoldot has been used before, so next steps in my investigation is to read through the smoldot json_rpc_service code to understand what it does and what assumptions it makes.

[tomaka]

It might be easier to just ask me, in particular because the json_rpc_service isn't finished.

To give a summary:

• You can send transactions.
• You can read storage values and watch changes in storage values.
• You can detect changes in the runtime/metadata (this might be automatically handled by PolkadotJS?).
• You should always be able to access the best block, the finalized block, plus recent best and finalized blocks.

[raoulmillais]

Thanks @tomaka I think it's good for me to understand what's going on too and it can't hurt if I get to know smoldot internals better. What I want to know is are there implications of having something wrapping smoldot that is multiplexing RPC requests? I.e. there is one smoldot client in the extension but there are potentially many pages making those requests - a map of PolkadotJS API client request ids -> actual request ids in the extension. I don't think this should be a problem but maybe there are assumptions about there only being one actual client.

[tomaka]

Indeed, feel free to read the code as well.

Everything should work even in the presence of multiple API clients. Note however that if an API client disconnects and has active subscriptions, you'll need to send unsubscribe requests to clean them up.

[raoulmillais]

I have turned this into an umbrella issue to track all the tasks and to split up the work.