Storage architecture

[osmaczko]

I'm starting on #112 (expose and implement MLS storage). Before coding I want to settle how storage works across core, client, and app, and record it as an ADR in docs/adr/. Three candidate shapes below; I want input before writing the decision down.

Goals:

1. Great developer experience at every level: app integrator, custom-store author, conversation-type author.
2. A new conversation type with its own storage requirements ships with minimal (ideally zero) changes at client and app level.
3. A new conversation type ideally does not change or break the existing db schema (a single shared SQLite file may be all an app can offer).

In the diagrams: amber = touched when a hypothetical next type ("V3") ships, red = broken by it, teal = the injected boundary, green = untouched.

A: one typed store, grown per domain

ChatStore stays a union of typed sub-traits and every backend implements each domain natively. Shipping V3: a new sub-trait, tables plus a migration, an implementation per backend, and a breaking change for every store implemented outside the repo.

Strongest typing and real SQL for everything. But the boundary breaks on every new type, goals 2 and 3 fail for custom stores, and OpenMLS's 57-method StorageProvider makes external backends effectively unimplementable (a SQLite implementation of that surface alone is ~925 lines).

B: a namespaced kv substrate; types own their schemas above it

The injected contract shrinks to six verbs over (namespace, key, value) bytes: get, put, delete, scan_prefix, delete_prefix, and an atomic write scope, all taking &self because OpenMLS requires it. The stock backend becomes one table: CREATE TABLE kv (ns TEXT, key BLOB, value BLOB, PRIMARY KEY (ns, key)). Everything typed lives above the seam inside libchat, written once: the StorageProvider adapter, the PeerScoreStorage adapter, the client's state layer, each type's own namespace. Shipping V3: code in libchat only; zero DDL, zero trait change, zero backend change.

What MLS looks like over it

OpenMLS never exposes group internals as a query. The only way to get a group's details is MlsGroup::load(storage, group_id), which fans out to twelve typed getters on the provider, so the adapter's whole job is composing a key per entity:

ns "mls"   group/<gid>/tree
           group/<gid>/context
           group/<gid>/join_config
           group/<gid>/proposal/<proposal_ref>
           group/<gid>/epoch/<epoch>/<leaf_index>/enc_keys
           key_package/<hash_ref>        // installation-scoped, outlives every group
           signature_key/<public_key>

// bounds elided; every entity is a generic parameter in the real trait
impl StorageProvider<CURRENT_VERSION> for MlsAdapter<'_, B> {
    fn write_tree(&self, gid: &GroupId, tree: &TreeSync) -> Result<(), Self::Error> {
        self.backend.put(MLS, &group_key(gid, "tree"), &serde_json::to_vec(tree)?)
    }

    fn tree(&self, gid: &GroupId) -> Result<Option<TreeSync>, Self::Error> {
        match self.backend.get(MLS, &group_key(gid, "tree"))? {
            Some(bytes) => Ok(Some(serde_json::from_slice(&bytes)?)),
            None => Ok(None),
        }
    }

    fn queued_proposals(&self, gid: &GroupId) -> Result<Vec<(ProposalRef, QueuedProposal)>, Self::Error> {
        self.backend
            .scan_prefix(MLS, &group_prefix(gid, "proposal/"))?
            .iter()
            .map(decode_ref_and_proposal) // the ref is the key's tail
            .collect()
    }

    // the other 53 methods, same two lines each
}

Group ids are opaque bytes, so group_key hex-encodes them and the separators stay unambiguous. Reading a group back, and dropping one:

let group = MlsGroup::load(&mls_adapter, &group_id)?;       // eleven gets and one scan
backend.delete_prefix(MLS, &group_prefix(&group_id, ""))?;  // purge, no cascade needed

That is also how OpenMLS's own reference store works: a flat map keyed by label || serde_json(key) || version. Two deliberate departures: the group id leads, so one group is one prefix instead of fourteen group-scoped labels to visit on removal, and lists come from scan_prefix rather than a separately maintained array of refs.

Goals 2 and 3 met fully and backends become trivial (a SQLite table, a HashMap), so storage stays genuinely pluggable. The cost: nothing is SQL-visible anymore; listing is scan-and-decode, inspection sees blobs, schema discipline moves from DDL to serialization conventions, and removal is a delete-by-prefix sweep instead of a cascade.

C: hybrid, a typed ClientStore plus the substrate

Same substrate as B for everything conversation-type-owned. Client-level state goes below the seam instead: ChatStore keeps a typed ClientStore for the stack's own slow-moving domains (the conversation list the client hydrates at open, identity, whatever the client grows next). One trait for simplicity, possibly composed of smaller stores behind it. Shipping V3: same as B, code in libchat only, because a new kind is a row value, not DDL.

trait ClientStore {
    type Error: std::error::Error;

    fn upsert_conversation(&self, convo: &ConversationRow) -> Result<(), Self::Error>;
    fn conversations(&self) -> Result<Vec<ConversationRow>, Self::Error>; // ORDER BY created_at
    fn remove_conversation(&self, id: &ConvoId) -> Result<(), Self::Error>;

    fn identity(&self) -> Result<Option<IdentityRow>, Self::Error>;
    fn set_identity(&self, identity: &IdentityRow) -> Result<(), Self::Error>;
}

Why this is worth a second idiom: client state is the part a human or a tool actually looks at. As rows with columns, sqlite3 and any DB browser answer "which conversations does this install have, of what kind, since when" with libchat out of the loop, which is what support and bug reports need; ordering and filtering come from the engine instead of a decode-and-sort in Rust; and constraints (primary key on the convo id, NOT NULL on the kind, foreign keys once the client grows a second table) are enforced by the database rather than by convention inside encoders. Identity secrets stay typed columns a backend can treat deliberately, rather than one more opaque blob.

The cost is that a ClientStore change is a breaking migration for backends, and that is the bet worth arguing about: conversation types keep arriving, potentially every few weeks, while the client's own model (a list of conversations, one identity) is close to complete and should stabilise. All the churn sits on the substrate side, where it costs nobody a migration; the typed side changes rarely, in coordinated steps. If the bet proves wrong, folding client state into namespaces converges C back to B, and nothing in C blocks that.

Scorecard

	A (typed union)	B (substrate)	C (hybrid)
New type: app on stock store	dep bump, auto-DDL on open	dep bump, no DDL	dep bump, no DDL
New type: custom store author	breaking, must implement new traits	untouched	untouched
New type: schema impact	new tables	new rows in the kv table	new rows in the kv table
Custom backend effort	very high (MLS surface)	trivial	small
Typed/SQL visibility	full	none (blobs)	client state (conversations, identity)
Client state (conversations, identity)	typed traits, growing union	namespaces over the substrate	typed ClientStore tables
Change to today's stock store	keeps growing per domain	typed tables become kv namespaces	keep identity + conversations, add the kv table

Where I lean

C. It delivers B's property where it matters (the boundary a new conversation type would move), keeps the queryable state queryable, and is the smallest change from today's store.

[jazzz]

I think this is a good write up of the problem, and potential paths.

1. I like the "meta-storage" approach: Allowing ConvoTypes to associate properties to their data, means that storage backends are coupled from the actual data. Resulting in easier rollouts. This is exactly the type of solution we are looking for.

2. Re: Where I lean:C. I'd separate the problem domains from a architecture POV, and unify during implementation if needed.

eg. ConvoStorage and ClientStorage are treated as separate problem/solutions. If generic client wants to enforce that the storage provider implements both traits, that is a implementation detail. But perhaps I've missing something that requires these be strictly coupled.

3. Our model adds an extra degree of freedom which we may need to account.

(namespace, key, value)

This works well for a singular protocol version. We may want to consider using (proto, instance, key, value) => (GroupV2, e4a2..12b3, "enabled", "true"). This ensures that data is scoped to the protocol that generated it so maintenance, and sandboxing becomes easier.

[jazzz]

Agreed, and nothing forces them together.

I think we are in agreement here. Hidden in my suggestion was that we perhaps keeps this document focused on ConversationType storage - Rather than addressing all storage in one doc. Account, Signer, ConvoState, ClientState, Messages all require storage solutions, and those need to be defined. If there are independent storage approaches we should approach them independently to keep docs focused.

I'm happy to leave this as is, if we put a small note specifying that this document spans both the Core layer and client layer but does not include local Identity management.

[jazzz]

Right, and I would rather pay for it with a restriction than with a fourth field. The namespace stops being a free-form string: it is the protocol name, drawn from the closed set of protocols libchat implements (GroupV1, DirectV1, GroupV2, InboxV2). A protocol name already carries its version.

Perhaps I am confused. As the doc is written today, what is an example of a namespace value? Does it correspond to the protocol name or the group_id?

I would rather pay for it with a restriction than with a fourth field

I'm more concerned that the data is available, and less concerned with how it is stored (Enum vs string). (Though there are impacts on scalability that ought to be discussed).

[osmaczko]

Perhaps I am confused. As the doc is written today, what is an example of a namespace value? Does it correspond to the protocol name or the group_id?

Quote from the post:

What MLS looks like over it

OpenMLS never exposes group internals as a query. The only way to get a group's details is MlsGroup::load(storage, group_id), which fans out to twelve typed getters on the provider, so the adapter's whole job is composing a key per entity:

ns "mls"   group/<gid>/tree
           group/<gid>/context
           group/<gid>/join_config
           group/<gid>/proposal/<proposal_ref>
           group/<gid>/epoch/<epoch>/<leaf_index>/enc_keys
           key_package/<hash_ref>        // installation-scoped, outlives every group
           signature_key/<public_key>

Namespace is mls, key is group/<gid>/tree, group/<gid>/context, etc. With your valid suggestion to namespace per protocol, it would become:

ns "GroupV1"   mls/<gid>/tree, mls/<gid>/context, mls/<gid>/epoch/<epoch>/<leaf>/enc_keys
ns "GroupV2"   mls/<gid>/...                 same shapes, same code, different scope
               convo/<convo_id>/config, convo/<convo_id>/peer_scores/<ident>
ns "InboxV2"   key_package/<hash_ref>

I'm more concerned that the data is available, and less concerned with how it is stored (Enum vs string). (Though there are impacts on scalability that ought to be discussed).

Not sure I am following this part, but if you are concerned some protocols might reach others' data, we can come up with a bound scope of some sort, example:

let convo: Box<dyn Convo<S>> = match record.kind {
    ConversationKind::GroupV1 => Box::new(GroupV1Convo::load(
        cx, self.storage.scope(Namespace::GroupV1), record.local_convo_id)?),
    ConversationKind::GroupV2 => Box::new(GroupV2Convo::load(
        cx, self.storage.scope(Namespace::GroupV2), record.local_convo_id)?),
    ConversationKind::Unknown(kind) => return Err(ChatError::UnsupportedConvoType(kind)),
};

The above gives protocol isolation but not instance isolation, and I would keep it there. The type shapes its own keys.

[jazzz]

Namespace is mls, key is group//tree

Ah that is much clearer. [Sand] I'd turn that "ns mls..." block into a markdown table, to cut down on confusion.

Not sure I am following this part,

Ignore for now.

[jazzz]

Option A

NS	Key	Value
GroupV3	group/<convo_id>/epoch	4

Option B

NS	ConvoId	Key	Value
GroupV3	a421..23ef	mls/epoch	4

I don't have strong opinions here and could be convinced with a compelling argument either way.

But heres my rationale for B:

• In A: If a convo type does not properly include a convo_id in the key, the state from one convo could bleed into another, causing bugs. In B adding the ConvoID is required, so it will never be an issue.
• Storage Management. When deleting a conversation, there is no unified guarantee to retrieve & remove all records belonging to a specific instance using option A.
• Schema flexibility: Option B provides storage implementors better information to store data how ever they would like - even if the interface uses a KeyValue model. e.g. select mls_epoch from convostate_GroupV3 where convo_id like 'a421..23ef'. Without isolating the instance key, storage options are more limited.