v8.0.0

This is a breaking change for users of the NPM package.

Previously all exports were available via earthstar:

import { Replica, ReplicaDriverIndexedDB } from "earthstar"; 

Platform-specific exports have been moved into their own subpaths:

import { Replica } from "earthstar";
import { ReplicaDriverIndexedDB } from "earthstar/browser";

There are also Node specific exports available at earthstar/node.

This was done for better compatibility with bundlers unable to treeshake unused exports, which would cause errors like fs not being available in the browser.

Earthstar v7.0.0 - Stone Soup

This is a ground-up rebuild of Earthstar which has been in the making since last April. It brings a lot of new features, the foundations for more to come, and is about as breaking as breaking changes get.

There are new features, APIs, and concepts. It's a lot to take in, so in addition to this new version of Earthstar, we've got a shiny website with explainers, API tours, technical documentation, and more!

So what's new? Lots.

Peer

There is a new Peer class which acts as the point of contact between your application and Earthstar.

• It's used to store, manage, and access replicas, with methods like peer.getReplica.
• And can be used to synchronise with other peers locally or over the network using Peer.sync. This method accepts lots of 'syncable' things like another Peer or a URL to a replica server.

Syncing

Syncing is now protocol-agnostic. Whereas previously you could only sync locally or over HTTP, it's now possible to sync over theoretically any transport. We've included transports for HTTP, Websockets and BroadcastChannel in our new library earthstar-streaming-rpc. We're to add lots more so that Earthstar can synchronise in many different contexts, like local networks.

We've also made it so that two Peers can determine which shares they have in common without revealing what those shares actually are. This is thanks to something we call a 'salted handshake', and makes it so that you can freely associate with other Peers without worrying that your share addresses are leaking.

There is also a low-level Syncer class you can use to handle scenarios and transports not catered to by the Peer.sync convenience method, e.g. replica servers or web workers (I mentioned we added a transport for BroadcastChannel, right?)

Replica

Previously data could be stored in two kinds of 'Storage', one with a synchronous API and one with an asynchronous API. We've replaced both with a single Replica class with an asynchronous API.

The functionality of this new Replica class is much like the previous storage, but it is now far easier to write persistence drivers for using the IReplicaDriver interface. Thanks to this we've been able to add a new IndexedDBdriver and Sqlite drivers for both Node and Deno.

ReplicaCache

There are many contexts where an asynchronous API can be a royal pain. ReplicaCache sticks a synchronous API in front of a backing Replica by keeping a cache of results for queries. The best part is you can subscribe to events where the cache has been updated, and use this to re-query the cache. We suspect this will be a very useful tool in contexts like a React app!

QueryFollower

There are many applications where you will want to subscribe to new events on a Replica, such as new documents coming in. Using a QueryFollower you can subscribe to events filtered by a Query. Callbacks triggered by this abstraction have special qualities like being called blockingly so that they apply backpressure (this is something that makes certain situations much easier to deal with).

New default ed25519 driver

Earthstar uses ed25519 for author verification. Previous versions of Earthstar could use Node's native crypto, or chloride for Node or the browser. Chloride is fast, but adds a lot to a browser bundle. TweetNacl (also exposed via Chloride) is much smaller, but very slow in comparison.

We've added a new default driver for ed25519 operations using noble/ed25519. This driver is smaller than libsodium-wrappers and much faster than TweetNacl, and works in all JS runtimes

Deno support

Earthstar can now be used in Deno projects:

import * as Earthstar from "https://deno.land/x/earthstar@v7.0.0/mod.ts";

Deno is a really great way to write and run Typescript, and we think it will make it far easier to write personal scripts and utilities using Earthstar.

It's also worth mentioning that Deno is now our primary development environment, and that we were able to make that switch without sacrificing Node support. Deno's support for web platform APIs and all-in-one tooling is something we hope will make contributing to Earthstar that much easier.

Web bundle

Not interested in Node or Deno? We also have a new web bundle, which you can use with a <script> tag:

<script type="module">
  import * as Earthstar from "https://cdn.earthstar-project.org/js/earthstar.bundle.v7.0.0.js";
</script>

Thanks

• Thank you to @cinnamon-bun, who led this reworking!
• Thank you to @tlowrimore, who contributed a nice template querying feature!
• Thank you to NLnet, who funded my own work on this new release!

v6.8.0: Support extra properties in docs during syncing

We now support this new section of the specification: https://earthstar-docs.netlify.app/docs/reference/earthstar-specification/#extra-fields-for-syncing

When sending documents over the network, peers may add additional properties as synchronization metadata. These extra fields MUST all have names beginning with an underscore to separate them from the core fields, and the the receiving peer MUST remove these fields before storing the document.

Likewise, when Earthstar libraries store documents internally, and send them over the network, they MAY add their own extra fields which MUST have names beginning with an underscore.

This should let us sync with stone-soup which adds an extra property _localIndex to the documents it sends. For now we discard this extra property when we receive it.

We're not currently checking the underscore rule; we just know which fields are extra fields because they're not the expected core fields.

v6.7.0: set max content length to 4mb

Documents now have a limit on their content length: 4 million bytes (measured as utf-8 enoded bytes). Now every field in a Document has a length limit.

Note that documents encoded as JSON can be about 500 bytes larger than this limit because of the other fields; this limit is specifically for the content field.

This closes #89 .

The specification has been updated; see https://earthstar-docs.netlify.app/docs/reference/earthstar-specification#content

Stone-soup is also updated to match, as of v2.1.1.

v6.6.2: Bugfix: Replace `multibase` package with `rfc4648`

This fixes issue #118

It also just seems to be a more solid, tightly focused package anyway.

This is purely a bugfix for compilation/bundling troubles, it should be completely backwards compatible at runtime.

Small bugfixes

Fixes

Some defenses against malformed input, discovered by earthstar-fuzz

Sort-of breaking change

We have a "standalone browser build" in dist/ that can be just dropped into an HTML file without using a bundler.

It used to be named earthstar-v6.6.1.js and now it's just named earthstar.js because it was too much work to rename it by hand every time. PRs accepted to automate that process.

If you access the file via a CDN like jsdelivr (see docs below) you can still specify the version.

Wrote some docs about using that file:
Using the standalone browser build

v6.6.0: Added a function: insertVariablesIntoTemplate

Commit: d20e725

This completes the set of template functionality. It lets you insert specific variable values into a template, e.g.

let vars = { category: 'gardening', okToHaveExtra: 'notUsedVariables' },
let template = '/posts/{category}/{postId}.json';
insertVariablesIntoTemplate(vars, template) === '/posts/gardening/{postId}.json'

(It's basically just doing a string replace on the variable names in brackets.)

v6.5.0: Template strings

Template strings for querying paths and extracting variables from them

You can make template strings like

    /posts/{category}/{postId}.json

You can use these to query for documents, and to extract the variables out of a path (giving you { category: 'gardening', postId: 'abc' })

Read the PR #94 for details, and the long comments in queryHelpers.ts which holds the code for glob and template searching.

As before, glob and template searching are not part of the official Query type that all Earthstar storages must support. Instead they are internally converted down into Queries plus a second step of regex filtering the results, since Queries are not powerful enough to capture all this functionality. We're trying to keep Queries very simple so we can add more varied kinds of storage backends.

Plans

We're planning to use these template strings to let users specify schemas for their data, and then auto-generate Layer classes to load and save that data to Earthstar. More soon!

Breaking change to glob search

Now,

• * matches any characters except / -- it's limited to stay within a path segment.
• ** matches any sequence of characters including /

Previously, there was only * and it worked like ** does now.

Consider 6.4.0 and 6.4.1 deprecated because it had the old glob behavior.

v6.4.1: Bugfixes for glob queries from v6.4.0

Bugfixes for glob queries:

• Forgot to export the new functions from index.ts, oops.
• By default we were adding the query option { contentLengthGt: 0} which skips docs with empty content (e.g. "deleted" docs). Removed that; now empty docs are also returned. if you want to skip empty docs you can put that option back into the moreQueryOptions argument.

See the release notes for v6.4.0 for details on the new glob query functionality.

v6.4.0: Glob queries

Added some helper functions which let you query documents using a glob-style syntax with asterisks.

• An asterisk is allowed to match any sequence of characters in a path, including slashes.
• There are no other special characters like ** or ?, just plain old *.
• There can be multiple asterisks in your query string.

Usage

Example glob queries:

/posts/*
/posts/*.txt
/posts/owner:*/category:*/*.txt
*.txt
*

Use it like this:

let docs = queryByGlobSync(myStorage, '/posts/*.txt')

Note! This is a special helper function -- it's not part of the normal Query functionality. You cannot use it directly in a query like this:

// NO, you can't do this
let docs = storage.documents({ author: '@suzy.xxxxxx', glob: '/posts*.txt' });
// NO

...read the Efficiency section below to find out why.

Documentation

The new functions are:

// functions for querying docs using globs
queryByGlobSync(
    storage: IStorage,
    glob: string,
    moreQueryOptions: Query = {}
): Document[]

async queryByGlobAsync(
    storage: IStorage | IStorageAsync,
    glob: string,
    moreQueryOptions: Query = {}
): Promise<Document[]> 

// lower-level functions used by the above
globToEarthstarQueryAndPathRegex(glob: string): { query: Query, pathRegex: string | null }
escapeStringForRegex = (s: string): string

More detailed documentation is in the comments here and here.

Efficiency and lack thereof

IStorages don't natively support this kind of query, so we have to transform it into a pathStartsWith and pathEndsWith query. If there's only one asterisk and it's at the beginning or end of the string, that's all we need.

In most cases though, the glob query is more specific than we can get with pathStartsWith and pathEndWith. In those cases we have to do a second step of filtering on the resulting documents using a regex.

So this is most efficient when your glob starts or ends with a long sequence of actual characters, so that we can get some help from pathStartsWith or pathEndsWith and not have to do much additional filtering.

Notes

This code was originally from earthstar-graph-db and it's now been moved into the core earthstar package.

commit 4eeafec