Standard paths and data formats used by apps
Since many apps can read and write documents into the same Earthstar workspace, like it was a shared folder, we need some agreement on common data formats if we want our apps to interoperate.
Here we've documented the data formats used by known apps.
Document paths should end in a file extension such as .json or .txt to guide apps in how to read them.
Unless otherwise specified, users should be able to edit and delete their documents after posting them.
To "delete" a document, just edit it to contain an empty string. Documents containing just an empty string should be considered deleted and should be hidden from the UI.
Document content is always in UTF-8 and is allowed to contain emojis and other special characters.
When designing new formats:
- Use a separate document for each part of your data that should be independently changeable by different people at the same time. For example a Todo text label and the Todo checkbox would each be a separate Earthstar document.
- Specify if you will use milliseconds (Javascript style) or microseconds (Earthstar style) for your timestamp. (Microseconds are
Date.now()*1000.) It will be a common compatibility problem between apps to use different timestamp styles. - Try to make paths that are alphabetically sorted in the order you want to display them in, to save a sorting step on display. When you query data in Earthstar it comes back already sorted by path.
- If you need to filter your documents more deeply, for example to only show things by one author, the author name needs to come first so you can do a
pathStartsWithquery for efficiency. - Remember to include the
~in front of usernames if you want to limit write permissions.
For deeper reading, see the Tutorial and the Rules of Earthstar.
So far only data in /about/ is considered standardized; the rest are app-specific.
| /about/ | /lobby/ | /todo/ | /twodays-v1.0/ | /wikiblocks-v1/ | /buntimer-v1/ | |
|---|---|---|---|---|---|---|
| twodays-crossing | yes | yes | ||||
| earthstar-foyer | yes | yes | yes | |||
| cozy-lobby | yes | yes | ? | |||
| earthstar-lobby | yes | yes | ||||
| react-earthstar | yes | |||||
| wikiblocks | yes | yes | ||||
| buntimer | yes | |||||
| earthstar-status | yes including status messages |
Note also that most apps are built with react-earthstar which provides a UI for setting display names in /about/.
A human-friendly name to show instead of the shortname (@cinn).
/about/~@cinn.bmxtgzizm62a5emwygayfomzs4z3wwe2y6wvhybkz6dgjm7smxfqa/displayName.txt --> Cinnamon
A single line of UTF-8 text with no formal length limit. Input forms should not allow newlines (use <input type=text>). Test your app with very long display names to make sure it doesn't break the layout. This works well:
.displayName {
/* show text on one line, with ellipsis if too long */
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}Most apps read this data format to show meaningful usernames. React-earthstar provides UI for setting your display name.
It's like the chat status from AIM or other old instant messengers. What are you doing right now?
Expect a single line of plain text. Emojis ok. No markdown or anything. UIs will probably ignore newlines when rendering this.
No formal length limit; UIs will truncate if it's longer than a sentence or two.
UIs can auto-linkify URLs they find in the status text, to make them clickable.
/about/~@cinn.bmxtgzizm62a5emwygayfomzs4z3wwe2y6wvhybkz6dgjm7smxfqa/status.txt --> going for a walk
Currently only earthstar-status shows this data format.
Lobby posts form a basic single-threaded chatroom or guestbook, with new messages typically appearing at the top. There's only one chat channel for the whole workspace.
/lobby/~@cinn.bmxtgzizm62a5emwygayfomzs4z3wwe2y6wvhybkz6dgjm7smxfqa/1597335926372.txt
Display them in sorted order by creation timestamp (from the path). The path's timestamp is made using Javascript milliseconds (Date.now()), not Earthstar style microseconds.
Posts can contain newlines \n. Input forms should allow the insertion of newlines (use <textarea>). Make sure to show newlines when rendering posts. You can use CSS like this to show multiple lines with nice word-breaks:
.multiLineLobbyPost {
/* make sure text wraps inside its container */
white-space: break-spaces;
overflow-wrap: break-word;
overflow: auto;
}There's no length limit; if one is very long consider having a "click to read more" button to show the entire thing.
The content is plain text with no markup. You may auto-link URLs, workspace names, @author names, etc. If someone enters HTML it should not be rendered as live HTML.
Users can edit their posts. This doesn't change the timestamp in the path because the path can't change, but the Earthstar document's timestamp field will change when edited. The user interface can optionally show "edited at 12:30". (For display, sort documents by creation time e.g. the time in the path.)
An IRC-style chatroom. There is only one chat channel for the whole workspace. This is a variation on the Lobby format.
/twodays-v1.0/~@cinn.bmxtgzizm62a5emwygayfomzs4z3wwe2y6wvhybkz6dgjm7smxfqa/characterName.txt
Because Twodays Crossing is a sort of roleplaying space, it lets you use a different Display Name than in other apps. Besides the path difference, this follows all the same rules and assumptions as regular Display Names (e.g. /about/~AUTHOR/displayName.txt
/twodays-v1.0/~@cinn.bmxtgzizm62a5emwygayfomzs4z3wwe2y6wvhybkz6dgjm7smxfqa/1607201748711.txt!
The timestamp is Javascript-style milliseconds (Date.now()).
Messages should be sorted by the timestamp from their path, not the timestamp they were edited (from document.timestamp).
All messages should be ephemeral messages with deleteAfter set to 2 days after the posting time. Because these are ephemeral messages, they must contain a ! in the path, and we've put it at the end in this case.
Earthstar will delete ephemeral documents for you when they expire -- you don't have to do anything. Unlike regular deletion when you just write an empty string, ephemeral documents are physically deleted and you won't be able to tell they ever existed.
If apps find non-ephemeral messages, TODO: what to do, ignore them? ignore them after 2 days? let them persist permanently?
Each message is a single line of plain text. It may begin with these IRC-style codes:
-
/me does something--> Cinnamon does something -
/describe The sun sets--> The sun sets -
/nick My New Name--> Changes your nickname to My New Name (see Character Name above)
This format only allows for a single todo list in the entire workspace. TODO: fix this!
Each todo item is made of two documents: one for the text label, and one for the checkbox state.
The label is a single line of UTF-8 text with no markup. Don't allow newlines when the user is entering it (use <input type=text>).
The checkbox state is the string "true" or "false" without quotes, which happens to be valid JSON.
/todo/1606780176966000-8898272/text.txt --> get apples
/todo/1606780176966000-8898272/isDone.json --> true or false
If the checkbox state document is missing, it defaults to false.
If the label document is missing, don't render the Todo at all, even if the label document exists.
Todos can be edited by anyone.
To make the "id" of a Todo, do this:
let sevenDigitString = ('' + Math.floor(Math.random()*9999999)).padStart(7,'0');
let id = `${Date.now() * 1000}-${sevenDigitString}`;Note that this timestamp is Earthstar-style microseconds, not javascript-style milliseconds, that's why it's multiplied by 1000. Ideally all timestamps in Earthstar paths should be like this, but it's too late to change some of the existing formats like Lobby.
The Earthstar tutorial is about handling this Todo format and has lots of example code.
(This is likely to change)
A Page is made of Blocks, which are small chunks of Markdown text.
There are also Comments which are attached to either Pages or Blocks (not sure).
A Page exists if it has at least one Block text.md document.
Pages don't have any Earthstar documents themselves; their existence is implied by Blocks.
Blocks can be reordered by changing their sort.json document. If there's no sort.json, a Block's sort value is the timestamp portion of its block ID (just the integer, without the entropy).
paths:
OWNER can be "common" | "~@suzy.bxxx" | "@suzy.bxxx" in the case of comments
COMMENT_AUTHOR is "~@zzzz.bxxx"
TITLE is a percent-encoded string, use encodeURIComponent
BLOCKID and COMMENTID include microsecond timestamps and entropy like "1607997091921015-Gc0r8"
// blocks
/wikiblocks-v1/OWNER/TITLE/block:BLOCKID/text.md -- markdown text of block
/sort.json -- a single float, defaults to current microsecond timestamp, oldest sorts first
// comment on a block (?)
/wikiblocks-v1/OWNER_NO_TILDE/TITLE/block:BLOCKID/comments/comment:COMMENTID/COMMENT_AUTHOR/text.md
// comment on the page (?)
/wikiblocks-v1/OWNER_NO_TILDE/TITLE/comment:COMMENTID/COMMENT_AUTHOR/text.md
Buntimer is visual countdown timer for daily uses like cooking, chores, and appointments.
Each timer is stored as a single document holding JSON. They are ephemeral documents which expire in 7 days, since we expect to only care about today's events anyway and we don't want to accumulate clutter.
Timer ids are randomly generated strings.
`/buntimer-v1/timers/common/${timer.id}!.json`
All timers are "common" (editable by anyone in the workspace).
interface Timer {
id: string; // a random unique string
endTime: number; // in javascript-style ms, not earthstar-style microseconds
name: string;
isDone: boolean;
}Example
{
"id":"564143514787",
"name":"lunchtime",
"endTime":1614362416976,
"isDone":false
}UIs should support these actions on a timer:
- Delete -- remove it completely, by overwriting the Earthstar document with an empty string
- Complete -- mark it as done. Completed timers may be hidden by default but users may want to look back and see a history of them.