Machine header envelopes for Subtext

[gordonbrander]

This issue exists to document background and decisions w/r/t machine headers in Subtext. Related to #19. No action is necessary.

Background

Key-value metadata is a generally useful primitive (see If headers did not exist it would be necessary to invent them). In our design discussions, we broadly identified two kinds of key-value metadata:

• Machine metadata (tracked here in Machine header envelopes for Subtext #41). this includes things like cache-control headers, content-type flags, user agent strings, etc. This kind of metadata is often invisible to the user and authored by the program.
• User metadata (tracked in RFC for user metadata in Subtext #19). Anything authored by the user for their own purposes.

Both of these use-cases can technically be supported by the same mechanism. However, machine headers are often visually noisy and potentially confusing. We believe it is valuable to have separate mechanisms for these two features so that the mess of machine headers can be hidden from the user.

Design goals

• Should be expressive enough that an app can reify state as metadata, such that any internal caches, indexes, or databases can be regenerated from files.
• Should provide a content-type mechanism
• Ideally, plain text body content should be readable as plain text

Prior art and notes

HTTP-like

Packets, HTTP, and email all use the memo pattern for key-value metadata.

A document begins with a series of key: value lines, followed by an empty line. Lines after the first empty line are considered content.

Title: Floop
Date: 2022-01-15
exotic-header: {"msg": "you can put anything in header body"}

Content.

This design has been very successful in the wild, and our sense is that our design solution should be heavily influenced by this pattern.

Note: earlier version of HTTP allowed for multi-line headers through line folding, then later obsoleted this feature. We wish to avoid line folding in Subtext. See #40.

Observations:

• Pro: It's what you would write anyway
• Pro: Header keys are case-insensitive, so you can write them nice style
• Pro: It works for email and the web
• Pro: Header bodies are specified separately from header syntax. This makes headers a completely open-ended extension mechanism.
• Pro: different applications can write to the same file without conflicts provided they use their own header fields (e.g. subconscious-meta, obsidian-meta)
• Pro: Happens to fit with the way SQL thinks about JSON. Blobs of JSON belong to a specific column.
• Con: Trends toward bespoke DSLs for each header.
  • Pro: This is also a pro, because it allows for open-ended evolution.
  • Synthesis: we could strongly encourage all headers to be one of a few types: String, Number, Bool, JSON. At the same time, it would be technically possible to do things beyond these types.
• Con: Requires at least one header, or a line break at the beginning of the file

Sidecar files

Another option is to make metadata a separate file that references the content file. In this case, the obvious choice for metadata format would be JSON. Like the envelope format approach, no specification would be necessary for Subtext to support this.

Tradeoffs:

• Pro: Multiple apps can generate sidecar files without conflicts
• Con: they don't travel together with the file (not the biggest deal)

Prior art:

• iTunes used to (still?) generates its plist data store by crawling your music library.
• You can delete your plist, and the application will regenerate it.

Reference:

• https://en.wikipedia.org/wiki/Sidecar_file

[gordonbrander]

Design decision

• Instead of specifying a machine header syntax in Subtext, we specified an envelope format in Noosphere, called a memo. Memos support arbitrary HTTP-style String: String headers for any content type, not just Subtext. See writeup on memos.
  • Memos could be seen as a kind of sidecar file, since they reference the body part by content ID.
• Ideas for a Subtext user metadata mechanism are tracked in RFC for user metadata in Subtext #19