Fetch Interface based on URLs/ HTTP Methods

[RangerMauve]

What's the problem you want solved?

I'd like to integrate protocol handlers for EarthStar in Electron apps or other browser-like apps, specifically in my p2p web browser project.

Ideally it should provide read and write access to stuff and have key management handled by the protocol handler.

We talked about this a bit on SSB here: %MbsFkcmbZU/pnx5JS5PpcVDtueMM0xslrzAXpqXfDwE=.sha256

Is there a solution you'd like to recommend?

I'd like to propose something that looks like the following psudo code:

GET earthstar://:workspace/path/to/document => JSON blob or whatever

GET earthstar://:workspace/path/prefix?author=example => Array of docs, or maybe a multipart response if it's raw files?

// If this is the first time you're using `name` as an identity
// The browser can prompt the user to initialize it
// If `name` exists but hasn't been used here before, ask permission from user
PUT earthstar://:name@:workspace/path/to/document {content in body}

// You can use whatever strings for methods, so why not invent SYNC? :P
SYNC earthstar://workspace/ {list of peers or pubs in body}

SYNC earthstar://workspace/path/to/document {list of peers or pubs in body}

I think that this could also be adapted for an HTTP API for some sort of daemon where earthstar:// could be repalced with http://somgateway.com/someapi/path/ to point at an HTTP server running either locally or remotely.

I'm interested in making a JS library called earthstar-fetch which will encapsulate this functionality in a similar way to dat-fetch and play around with it before drafting a full standard. A side effect would be that we'd get a fetch-like interface into regular web browsers along side protocol handlers in browsers like Agregore.

[cinnamon-bun]

Nice! ⭐ Sorry for the wall of text here, I'm just enthusiastic about this :)

Big picture thoughts

I had assumed people would build Earthstar apps by using the Storage API and Layers APIs provided by this Javascript implementation, and we'd achieve app portability around those APIs by making a variety of swappable classes that work in different settings (browser, node, etc).

Now there are other access patterns emerging, like this fetch API and a graphQL interface. This is a good thing:

• These are more widely understood, less specific to Earthstar. One principle of Earthstar is "use boring technologies and paradigms that are widely known"
• The network-related part of Earthstar isn't solid and standardized yet, so experimentation is good
• These allow access across languages so we don't need Earthstar implementations in every language

Standard methods across access patterns

To avoid fragmentation let's try to use the same the method names across REST, fetch, graphQL, etc.

I've just added a bunch of documentation about these methods to types.ts, especially

• Storage methods
• Query options
• Sync options. Note the "sync filters" aren't implemented yet but will be important to support.

URL details

The workspace and author address formats are designed to work well in URLs with two caveats:

• Sometimes the sigils need to be removed (+ from workspace, @ from author)
• base58 is case sensitive, but URL locations are supposed to be lowercase (at least for HTTP)

These rules make them URL-safe:

• Paths are already required to use percent-encoding for weird characters, so they work in URLs without change
• Workspaces and authors only contain alphanumeric chars, ., +, and @
• Workspace and authors have a dot in the middle so they're recognized as domains by browsers
• Workspace and author parts can't start with a number (including the base58 keys)
• Paths can't start with /@. This avoids ambiguity between WORKSPACE/PATH and WORKSPACE/AUTHOR

So except for the base58 case issue, these are all guaranteed to be valid URLs.

earthstar://WORKSPACE_NO_PLUS
earthstar://WORKSPACE_NO_PLUS?QUERY
earthstar://WORKSPACE_NO_PLUS/PATH
earthstar://WORKSPACE_NO_PLUS/AUTHOR

earthstar://AUTHOR_NO_AT@WORKSPACE_NO_PLUS/PATH
earthstar://AUTHOR_NO_AT:AUTHOR_SECRET@WORKSPACE_NO_PLUS/PATH

http://mypub.com/WORKSPACE
http://mypub.com/WORKSPACE?QUERY
http://mypub.com/WORKSPACE/PATH
http://mypub.com/WORKSPACE/AUTHOR

examples, using xxxxx to stand in for long base58 strings

earthstar://gardening.xxxxx
earthstar://gardening.xxxxx?pathPrefix=/wiki/&limit=5
earthstar://gardening.xxxxx/wiki/Flowers
earthstar://gardening.xxxxx/@suzy.xxxxx

earthstar://suzy.xxxxx:xxsecretxx@gardening.xxxxx/wiki/Flowers

http://mypub.com/+gardening.xxxxx/wiki/Flowers

The author:password@workspace format puts the @ in a confusing place compared with normal Earthstar sigils. It could be OK just for doing writes in the fetch API if end users never see it in the browser UI?

HTTP verbs

Let's add PUSH and PULL in addition to SYNC, for one-way syncs.

Pubs & swarms

The earthstar:// URLs don't include pub or swarm information. Not sure where to put it. There could be multiple pubs and multiple swarms.

There will be 2 kinds of swarms:

• the usual main swarm: swarm key = hash(workspaceAddress), like DAT
• secret swarms: swarmKey = hash(workspaceAddress + swarmPassword)

Swarm passwords are just secret strings you share with friends. They let you connect only to trusted peers when you're in a very large or public workspace to protect your IP address privacy.

(Swarms aren't implemented yet)

[cinnamon-bun]

Parsing earthstar:// URLs -- case sensitivity problems

I tried shoehorning this into the javascript URL parser. It almost worked but it lowercased the workspace, which will break the base58 workspace secret. :(

let orig = 'earthstar://+gardening.xXxXx/@suzy.xxx';

// parse it.
// URL() expects protocol to be http, so let's hack it
// also have to remove the '+'
let hacked = orig.replace('earthstar://+', 'http://');
let url = new URL(hacked)
console.log(url);

// extract info from the parsed URL
let workspace = '+' + url.host;
let author = null;
let path = null;
if (url.pathname.startsWith('/@')) {
    author = url.pathname.slice(1);  // remove leading slash
} else {
    path = url.pathname;
}

console.log(workspace, author, path);
// --> +gardening.xxxxx    @suzy.xxx    null

Switch to base32?

Maybe we need to switch Earthstar keys from base58 to base32 so it can be case insensitive. It would make them 8 characters longer.

secret length:
256 bits
32 raw bytes
44 base58 characters
52 base32 characters

They would still fit in the limit of 63 characters for domain name segments (between periods).

Added issue #29 for this idea