Proposal: Document Views

[b5]

I've had time to play with iroh a bunch now that prefix-based deletion is in, and I must say, it's been very fun learning how iroh works 😜. With that said, I think prefix deletion and author ID's as key prefixes are too much for a user to get their head around on first go. As a user when I hear "key value store", I expect all keys to be unique based solely on the key I provide.

We have some new UI that at least illustrates the problem:

In this example there are two foo keys in the same document, because they have different authors. What's more, because I am neither of these authors, I cannot edit these keys, which is super confusing. As someone who works at n0, I fully understand why I can't edit stuff in someone else's subspace, so I'm not advocating we change anything about the way iroh works, instead I want a new "views" feature that gives me a "merged" view of the document, which is a union of all known replica keys, using timestamps to break ties.

Iroh is missing a true "shared" view of a document. I need this ASAP because it's too hard to introduce users to both the author-replica thing and the prefix thing at the same time. I think the right answer is to start with teaching prefix deletion, but have the default display be this merged view that will feel like a true key-value store. That shared view will naturally invite the "but wait, just anyone can edit the db!?" question, which will open us up to talk about how there are ways to change the view to match individual authors, or the full-blast "all" view.

Views

The term view here means "view" in the classical database sense:

a view is a virtual table representing the result of a database query.

We already have the concept of filters which perform largely the same function, but co-mingle querying with views:

iroh/iroh-sync/src/store.rs

Lines 141 to 155 in 8169053

	/// Filter a get query onto a namespace
	#[derive(Debug, Serialize, Deserialize)]
	pub enum GetFilter {
	/// No filter, list all entries
	All,
	/// Filter for exact key match
	Key(Vec<u8>),
	/// Filter for key prefix
	Prefix(Vec<u8>),
	/// Filter by author
	Author(AuthorId),
	/// Filter by key prefix and author
	AuthorAndPrefix(AuthorId, Vec<u8>),
	}

I think the All and Author are views, with Key being an implicit query on the All view, AuthorAndPrefix is an implict query on the Author view.

Merged View

A merged view takes the following steps:

1. construct the union of all entries across all authors, sorting by key field
2. construct the union of entries based on the key field. last-writer wins for conflicts on the key field, lexographic sort of author_id wins in the event of equal key & timestamp fields.
3. apply prefix deletion to the remaining keys, again using timestamps to break ties.

I'm sure the algorithm could be better. This is just how my dumb brain expresses "I want unique key fields, last-writer-wins, regardless of author"

Proposal:

• remove the concept of filters, split them into queries and views
• There are three views: all (all keys from all authors), author (only keys from one author), and merged (all keys in one space)
• "merged" becomes the new default view
• APIs are updated with a --view flag

[dignifiedquire]

My first pass at how I think this can look for the first round: https://gist.github.com/rust-play/0cda7cc7eee6787ce9904742d154d89e

[dignifiedquire]

Implemented and refined in #1766