Private, non-shared data in repo?

[raisch]

I have a use-case for keeping private data in a repo, i.e. data that does not leave the repo via the standard sharing mechanisms, and before I leap into modifying the code to support it, I'd like to get some feedback (or at least reasons why this would not be a "good idea.") While I know I can run a side-car service to support this, I'd very much like to leverage existing atproto tools/tech.

My strategy is that, for data with any NSID in the form local.* (per RFC6762) or perhaps localhost.* (per RFC2606), disable update event generation and contribution to CBOR packaging.

A possible enhancement would be to enable storage of such "private" data to other data-stores via a startup feature flag and adding support for CRUD operations via custom interfaces to external storage.

Ok all, rip me to shreds, please. ;)

Thanks.

[DavidBuchanan314]

I think this could work. You'd still want to CBOR-ify it (since CIDs are integral to various APIs, and the data model in general), but you could skip MST updates and the firehose broadcast. You'd need to make sure the records can only be fetched by the currently-authed user (from their own repo).

[DavidBuchanan314]

I've been thinking about this some more, and I have a similar but modified idea.

Rather than using special NSIDs, I propose using a special "private DID" prefix. This could be something like internal:, so if the main repo is identified by did:web:example.com, the "private repo" would be identified by internal:did:web:example.com.

(internal: is not a reserved uri scheme, maybe some more thought could be put into choosing a prefix that won't collide with anything else.)

This addresses two issues:

1. With the "local.*" NSID prefix approach, if you tried writing one of these records to an oblivious PDS that didn't implement this feature, it would "fail dangerous" and end up publishing your private record on the firehose. On the other hand, a regular PDS would give an error if you tried writing to an internal:* repo.

2. It simplifies PDS implementation logic, and you can still retain a full MST for it (separate from the main MST), meaning APIs like getRepo would also still work, and features like swapCommit would also still work.

The only downside I can think of is that you couldn't use applyWrites to write public and private records in the same transaction.

[bnewbold]

To hint at the direction we are thinking for folding this in to the protocol proper:

• a concept similar to repos called something like "namespace" would be introduced. namespaces are sets of records identified by collection and record key, stored in the user's PDS
• the public repo is basically a special namespace which is always public, and has data stored in an MST, via "commits", and with changes going out publicly on the firehose
• the other namespaces don't have MSTs, don't have overall commits, and don't go out publicly on a firehose. they are just stored in a boring database, and request authentication+authorization to access. they would probably have a different set of APIs to get/create/update/delete. they might involve CBOR serialization (so they can have CID versions), or might not. records would use the same Lexicon schema system.
• access is gated to the user's client (via auth scopes); and to other users (by DID); and to other services (also by DID). there would be at least collection-level gating, and possibly record-level. there are also meta-controls over the ability to list which namespaces of which kind exist for a given user
• there would be a special "personal" namespace for each user, which is for things like preferences and mutes. these might give access to services, but not to other users
• other namespaces probably have an overall "type", and then an "instance". these are kind of like another layer of collection (NSID) and record-key, but for the overall namespace. access to create and enumerate entire namespaces would be granted by "type" to clients. as an example, a type could be "private-travel-photos", and an instance could be "my family trip to the beach". that namespace could contain multiple records of multiple types inside it. blobs are probably associated with a namespace.
• you don't move data between public repos and namespaces. content is either public or not. you don't toggle an account between "public" and "private". it is technically possible to have the same data types in both public and private storage, but you don't necessarily mix public and private content together in API views or client UI
• this should scale to ACLs with 10s of millions of accounts. PDS resource consumption scales with group sizes; this is different from how public repos scale
• in this vision of private data, nothing is encrypted at rest or E2EE, only in transit (eg, TLS). PDS has full visibility in to the data and ACLs. E2EE group DMs would be an entirely separate mechanism
• complexity comes in around how to make it efficient to synchronize updates between many users and services; especially allowing multiple client/user-agent implementations.
• TBD if this uses a variant of the at:// URI scheme, or some new URI scheme

We aren't fully committed to that overall architecture, but that is the general idea of what we are thinking.

[surfdude29]

I don't have any special insight into what the Bluesky team are planning in terms of drafts, but I imagine they might be only stored on your device at first, as this would be the simplest option and would still cover a lot of use cases.

If there is an option to store them on your PDS – which would have a lot of benefits – hopefully they could make use of the "personal" namespace mentioned?

there would be a special "personal" namespace for each user, which is for things like preferences and mutes. these might give access to services, but not to other users

I'm hoping that bookmarks and private lists might be able to be stored there too 🤞

[verdverm]

@bnewbold Is there a related path to multi-account ownership or control of collections?

Perhaps it could be as simple as recommending a best practice is to create a separate account and grant permissions as needed in this scheme?

[surfdude29]

• you don't move data between public repos and namespaces. content is either public or not. you don't toggle an account between "public" and "private".

Thinking about this again after reading comments in another discussion, e.g. this one:

[…] Going private is one of the known best defenses when there's coordinated harassment against you. […]

If private data is implemented along the lines of something similar to what was hinted at in February (of course, I realise the thinking might have changed since then), might some kind of partial account deactivation be possible, where a user's public repo becomes temporarily hidden and inaccessible, but they're still able to communicate and share content on their account's non-public namespaces?

I think this might be quite a good ATProto version of "going private". A user's public content would disappear from view, but the user would still be able to use their ATProto identity to communicate with and share content with users who they trust. Ideally, they would still be able to use the forthcoming E2EE group DMs as well.

[verdverm]

A way to "go private" on a switch could be

1. Use a private namespace as your primary record store
2. Set permissions to anonymous, logged in, bsky user, or whatever (i.e. it's nearly as permissive as public records)
3. Change permissions on that namespace when adversarial attacks happen

[npetrangelo]

I'm developing an app with public data like what AT Proto already supports, group chats, and potentially sensitive private data, each at a different stage in the user flow. (Users finding each other's posts, leading into group chats, that produce private artifacts)
I think the characteristics I would like for private data hosting are:

1. e2ee (data not transparent to PDS host)
2. configurable access scopes (i.e. how private it is)
3. database agnostic (PDS implementations can use whatever they want)
4. credible exit (user can take there data somewhere else with continuity)

[bnewbold]

@CanRau drafts are a good use-case and example! I think they would work fine in this setup, as long as the draft isn't "the same entity" as the final post. For example, if you shared a draft with a couple friends, and they commented, those comments would not come along to the final post. The final post would have a different URI/URL from the draft, as another example. It is your last example, going from public to private that would not really work. You could of course copy the content to a new private version of the post, but it wouldn't be the same as the public one.

@MikeBeas migration would work by pulling a copy and uploading to the new PDS. David covers the case which you mention: best practice would be to keep independent backups. Optimistically, that sort of backup will get automated, either by client apps or folks configuring a backup service that holds on to everything for them. In the long run, we see PDS hosting as fairly commodity/utility. Something you might pay a small amount for, or get as part of a package of digital hosting. In that case interests would be aligned and the provider is less likely to pull the rug. If the provider isn't an app/network/brand, they don't have as much of a network effect to exploit. Note that today Bluesky PBC, specifically, is both an app/brand and a PDS hosting provider.

Bookmarks would definitely fit in as personal data.

[verdverm]

AppViews could handle drafts -> published via permissions and an extra field indicating state (mainly for UI affordances).

1. draft mode has restrictive permissions
2. published mode has permissive permissions (but not include editor comments, assuming RBAC here)
   Essentially publishing could be the process of adding permissions to the record for a wider audience

Copying records or permission management forms of publish both seem legitimate. Probably up to the developer of the appview which they prefer to use. Both are very in line with how one might accomplish the same thing on Google Docs.

[npetrangelo]

I think for that first use case, it should be possible to basically publish a private post into public scope such that comments carry over, but that's a one way operation and you can't put the genie back in the bottle. Maybe in addition to the public namespace with an MST, there can also be a special staging namespace that also has an MST. In principle it could be similar to the git workflow of committing to your local repo before pushing, or working in a fork before submitting a PR. Or, perhaps every namespace you make in addition to the public one should have the option of being an MST namespace when you create it, and you could also fork one of your public repositories again similar to git. Honestly taking a lot of inspiration from git workflows would be a good idea here.

[kotx]

For now, could a workaround be implemented with preferences (app.bsky.actor.getPreferences)? I'm not familiar but it appears to be private synced data (or is that stored in the appview and not the pds?)

[LukaJCB]

in this vision of private data, nothing is encrypted at rest or E2EE, only in transit (eg, TLS). PDS has full visibility in to the data and ACLs. E2EE group DMs would be an entirely separate mechanism

In the case of a rogue PDS, this would be devastating. This feels a little disappointing to read, in essence giving the providers full access to your private data.

[bumblefudge]

this is one of the major beefs people have with mastodon, which i think is valid. in DIF, the PDS WG originally had two work items, one more oblivious and solid-like (all data encrypted to at least one key, and all sharing done by deriving new keys for new groups/RBAC categories), and the other more like a BS PDS. the former ended up being abandoned when the companies driving the design put the project on backburner, but similar design space persists in the W3C Linked Web Storage WG, which is also a lot of people with SoLiD experience trying to reduce the complexity to various hypotheses of what the minimum might be for different use-cases.

[ianopolous]

If someone has the resources (we don't yet), we would love to formally standardise the peergos pds protocol which implements the former case of everything being E2EE. Or at the very least use it as a base for a standard. This is a working solution that is already widely deployed. It could also easily be merged with atproto for public data, maintaining portable data and accounts.

[ianopolous]

@whyrusleeping wants me to make a proposal based on Peergos. So here goes.

You could use the Peergos design as is by just switching Bluesky dids for our dids. I'll try and give a high level overview of this here.

You PDS would have a private blockstore which is for access controlled blocks. The blocks in this store form distinct writing spaces. Each writing space is a champ (compressed hash array mapped prefix trie - the unordered sister structure of MSTs) with values that are dag-cbor encoded cryptree nodes (the encrypted metadata of the file/dir). The cryptree nodes have merkle links to the raw ciphertext of the file (raw nodes in dag-cbor as they are leaf nodes). The key pair for each writing space signs the root for that space (actually the root, prev root and a counter so the server can enforce CAS updates).

To read a file you need a read capability which is (owner did, writer public key, champ label, block access token or bat, symmetric read key). So to get a file you ask the server for the encrypted metadata - server.get(writer, champ label, bat). Then decrypt the result with the read key. The champ labels, bats and read keys are all 32 random bytes. The server checks the bat (which is stored in the block itself) before returning a block or a champ lookup. So bats are basically bearer tokens for blocks.

This gives you an E2EE filesystem, with fine grained access control. You can grant read or write access to individual files or entire subtrees with a single capability. Metadata is protected from the PDS, including filenames, file sizes, the directory topology, whether or how many people have access to something. These blocks obviously aren't published to any relays, but are retrieved directly from the owner's PDS.

To share access you need to share a capability, which can be done using any secure channel like the proposed MLS group chats. Revoking access means rotating the keys and re-sharing a capability with those you want to keep access.

Files are also chunked and padded so each chunk ends up a different champ mapping. You can choose the chunk size, but 5 MiB works well for streaming videos in the browser.

To see more details of cryptree see: https://book.peergos.org/security/cryptree.html

[bumblefudge]

sorry, I got a little confused on why this requires a DID method change? Couldn't an additional key just be added to the DID doc under a new verificationMethod that serves as "parent key" to the derived keys for each champ/space? Are the capability objects [DAG-CBOR] UCANs or something with different properties/DSL?

[ianopolous]

I meant switching from peergos dids that we use to atproto dids.

The capabilities are described here https://book.peergos.org/security/capabilities.html
They scale to a p2p setting where authorised retrievers of blocks (like relays for example) can continue to enforce the same auth on block retrieval without any extra state outside the block itself. That part of the access control is described here: https://peergos.org/posts/bats

More concretely, a capability is (owner DID, writer public key, champ label (32 bytes), bat (32 bytes), symmetric read key (32 bytes)) and an additional symmetric write key if it is a write capability. The did is used to find the PDS to route the requests to. The writer is used to verify the signed champ root. The champ label and bat are used to lookup and authorise retrieval of the cryptree node, and the read key to decrypt it.

[verdverm]

This gives you an E2EE filesystem, with fine grained access control.

Records are stored in CAR -> sqlite table, would this work with that or would all records (i.e. likes and follows) need to be saved to a filesystem like setup?

Revoking access means rotating the keys and re-sharing a capability with those you want to keep access.

This seems like the big UX complaint around E2EE records vs private / permissioned content that works more like Google Docs / Dropbox permissioning

[ianopolous]

Cars are just groups of dag-cbor blocks, nothing special there. Everything we do is dag-cbor. The underlying storage doesn't matter. You can choose how to store the higher level data like likes in the filesystem.

The resharing is automatic. So it works exactly like Google docs/dropbox permissions as far as the users are concerned.

[LukaJCB]

Hey all,
I took some time and implemented a tiny little POC based on the statusphere example app: https://github.com/LukaJCB/statusphere-example-app.
The gist of it is that have a separate collection for private content and when you upload a new post you first generate a unique data encryption key (DEK) for that post, encrypt the post with AES, then upload it to the private collection. Next you take the AES key and use MLS to encrypt it with the established group key.
The client keeps a list of all the DEKs and can send them to a newly added group member so that they can see all the posts prior to them joining.
This of course means you forgo forward secrecy, so you could also implement skip this step and don't allow new group members to see prior posts.

I made the choice to use the PDS as a message store for the MLS messages, but that's likely not ideal, these messages aren't meant to be around forever, so likely a different delivery mechanism would be much preferable. Potentially bluesky could just use the existing messaging system to send and receive MLS messages. The important part is that your content remains on your PDS and in the case of an "adversarial exit" you could easily retain all of your posts and the group state would remain on your client.

In this implementation most of the metadata for your posts is public in the PDS, but there is no reason it would have to be. The whole post and all it's metadata could be fully encrypted. The firehose could just use the follower graph to distribute the encrypted posts to their followers.