Upcoming Disruptive Protocol and Infra Changes (2023 / early 2024)

[bnewbold]

This document tracks currently-planned changes to atproto (the protocol) and Bluesky (the company) infrastructure services which may impact the larger ecosystem of developers and services.

These are all changes which may happen in coming months, as a heads up, though this document is mostly directional and not a prioritization or commitment to any specific features.

For changes that would have a direct impact on third parties, we will continue to blast via mailing list (sign up here!), announcements here on Github, and via the @atproto.com account in-app.

AT Protocol Changes

• did:plc hardening: the plc.directory instance enforces per-DID operation rate-limits, and some new constraints on length of strings and arrays
• DID document key syntax: switching to "Multikey" syntax
  • plc.directory scheduled to switch over week of 2023-10-09
  • Planned Changes to DID Documents (August 2023) #1510
• Admin Lexicon refactor: the com.atproto.admin.* Lexicons are likely to change in backwards-incompatible ways, to better support moderation in a federated network
  • the current moderation reporting Lexicons, used by clients, should not break
• Auth refactor: both third-party auth flows (eg, OAuth2), and to support verifiable inter-service requests (eg, with UCANs)
  • UPDATE: see OAuth proposal
• Explicit account lifecycle and state: clarifying and possibly changing the repo event stream "tombstone" and "migrate" events. Possibly some mechanism to broadcast PDS account takedowns. Possibly a mechanism for account temporary self-deactivate without deletion.
  • UPDATE: this will likely happen on the same timeline as full account migration support.
• Client request proxy header: Documented ability to control which appview instance requests will be proxied to from the PDS, controlled using an HTTP header
  • UPDATE: have made some progress on this, but no firm deadline as a specified protocol feature
• Firehose event stream: likely to iterate on the seq number semantics for backfill. Possibly sharding, "epoch", and "light" (no MST nodes) query parameter options
  • repo v3 semantics, including rev, commit, and "diffs", are more stable and less likely to change
  • UPDATE: looking less likely to make changes on this in early 2024
• More HTTP status codes: XRPC endpoints currently a limited set of status codes, using error types in the JSON response to indicate the error. Additional codes like 410 Gone, 404 Not Found may be adopted.
  • UPDATE: this is a bit amorphous of a task. documenting recommended status codes could happen any time; including status code as part of error response spec is a bit further off
• Third-party Labeling: protocol-level ability to distribute and subscribe to labels from many sources, including cryptographic signatures for verification
  • labels from multiple sources (configurable) may end up in hydrated "view" label arrays
• Record Key validation: when creating new repo records, the record key (rkey) will be validated against the rules in the Lexicon, along with the record schema itself.
  • will be possible to override (same as schema validation) with a query parameter
• Timestamp semantics: mostly a specification and documentation task, the validation and interpretation of various timestamps in the system will be clarified and enforced
  • UPDATE: see https://docs.bsky.app/docs/advanced-guides/timestamps
• Account migration: new protocol features to enable both "happy path" and antagonistic/fallback migration of repos (and blobs, and identity) between PDS instances
• Identity control: more accessible control over did:plc updates and rotation keys, from PDS endpoints and end clients. Already possible today by talking directly to PLC directory, but only to very technical folks
  • UPDATE: changes on this will likely be made on the same timeline as account migration

bsky Lexicon Changes

• Remove app.bsky.unspecced.getPopular: an "off-spec" endpoint, has been deprecated for some time
  • deprecating `app.bsky.unspecced.getPopular` #1648
• Search updates: the current "search posts" endpoint is unspecified and lives on search.bsky.social. A new "post search" endpoint will be part of the app.bsky.* Lexicons, served by the AppView (api.bsky.app), and the old search service removed
  • actor search already moved to Lexicon version
  • post search Lexicon added, but not yet implemented in AppView
  • search quality and features also improved (i18n analysis, from: account filtering, quoted phrases)
  • there will be a transition period when search.bsky.social remains online until clients migrate

Infrastructure and Operational Changes

These are not changes to the protocol per say, but they are changes to network topology, or could otherwise impact third party services.

• Image CDN update: the CDN for images and thumbnails will change. This should be mostly seamless (no URL change) but could change latency or routing
• BGS firehose on main network: firehose consumers should switch to the bsky.network BGS instance
  • recommend switching subscriptions soon, new PDS instances expected by mid-October 2023
  • sequence numbers are differend from bsky.social PDS
• Multiple PDS instances: the monolithic Bluesky PDS (bsky.social) will have accounts split out across multiple instances
  • likely to continue allowing auth and most endpoints on bsky.social with virtual proxying, so clients won't break
  • but PDS repo streams will be per-PDS, with distinct endpoints
  • DID documents will be updated with specific PDS hostnames
  • Migrating bsky.social to Multiple PDS Instances (Nov 2023) #1832
• Increased did:plc operation churn: DID documents have been fairly stable for the majority of accounts. With PDS migrations, both the PDS endpoint and signing/recovery key will be updated for millions of accounts
  • likely to start mid-October and be ongoing
  • services doing data and identity verification will need to refresh any cached DID documents more frequently
  • reminder that PLC directory has an "export" feed endpoint to scrape all updates
  • Migrating bsky.social to Multiple PDS Instances (Nov 2023) #1832

Policy / Feature Changes

These are less likely to developers on a technical level, but could impact product expectations and dynamics.

• Public logged-out web interface: will likely be deployed, making it clearer that all atproto/bsky content is indeed world-public
• Email address verification: tracking verification state, and allowing self-serve email updates
• Automated spam and abuse detection: complementing human review and API rate-limits, likely to have an automated rules engine to scan the full firehose and detect patterns of spam and network abuse
  • automated CSAM detection has been in place for some time
• Network growth control: when federation opens up on the main network, there will likely be some time period where some technical measures are taken to limit the number of overall accounts in the network, to prevent infrastructure and third-party services (like feed generators) from all being overwhelmed
  • UPDATE: we have done a fair amount of planning and preparation on this. Will share more details soon, on the same timeline as opening up federation. See discussion below about what this will look like technically.

[snarfed]

ooh more HTTP status codes! /me crosses fingers for 302, eg for getBlob (#1228)

[bnewbold]

I'm leaning towards it!

Any thoughts on HTTP redirects for WebSocket (subscribe) endpoints, specifically?

[snarfed]

Ohhh no not at all, I don't know websockets well enough to opine. Personally right now I'm fine without them there though.

[snarfed]

Back here to add that 401 for bad/expired tokens etc would be nice too!

[bnewbold]

Some other ones on the stack are:

• clarifying 5xx and other upstream errors in the context of proxied requests (now that we have a bunch of proxying going on)
• 422 for content that matches Lexicon schema but is invalid for another semantic reason
• 410 for some deleted content (eg, tombstoned accounts)

[sneakers-the-rat]

@bnewbold I wonder if you can say any more about this part:

Network growth control: when federation opens up on the main network, there will likely be some time period where some technical measures are taken to limit the number of overall accounts in the network, to prevent infrastructure and third-party services (like feed generators) from all being overwhelmed

what kind of technical measures are we talking about here?

[sneakers-the-rat]

Got it, thanks for the clarification. So that would look like, from the POV of a PDS, just a failed API call to the relay, standard HTTP error code stuff? I imagine the actual quotas aren't worked out, so i'll just look for those later when federation opens up.

The rate limiting makes sense, no questions re: creating new posts/actions, but i'm a little foggy on how that works w.r.t. throttling account creation - does the relay handle identity like that? or do you mean the relay would only handle traffic from the first n repos from a given PDS? Curious because of how repos are portable, how one would distinguish between a million people moving from one PDS to another vs. a million new accounts, and whether the relay is set up to handle repo-level moderation like that or if it's agnostic.

Similar question from a safety POV and how this interacts with labeling systems - will the BGS have any sort of content moderation or is it strictly a technical rate limiting kind of filter? eg. if a given PDS has x quantity of {y} labels can/would the relay block them? I'm again curious mostly to get a sense of the protocol's federation architecture here and the division of labor between the relay, app views, and labels. So since an identity is a repo and a repo can be moved between PDS's, there isnt' a 1:1 mapping between a DID and a network identity like an IP or a domain name. And, i could be wrong, but a PDS receiving objects and interactions from the relay wouldn't know which PDS it comes from, correct? or am i looking at the wrong part of the lexicon. So it seems like there might need to be some kind of content moderation at the level of a relay (or do the labeling systems get additional information like source IP/domain/etc?), and i'm curious if that's something y'all had planned or if these limits for the early days of federation are strictly technical rate limits?

thanks for answering my question :)

[bnewbold]

The relay subscribes to the PDS stream (a websocket). The PDS may send a notification to the Relay via HTTP endpoint when it has new info to subscribe to (eg, if the PDS is new, or no accounts have posted in some time so the Relay dropped the connection). But the PDS does not "push" to the Relay, the Relay "pulls", and might decide not to for whatever reason. That means the PDS (and client) don't get an explicit error condition when this happens, though the PDS can notice (and monitor/alert the operator, speculatively) that any expected relays are not subscribed.

For accounts/clients, there is a mechanism to notice and alert if repo content is not getting subscribed and indexed by the appview (which is downstream of the relay), via an HTTP header. We haven't documented this well, but should get to it soonish.

the relay would only handle traffic from the first n repos from a given PDS

this one. each PDS would have a quota of how many currently active accounts can be indexed coming from that PDS. It doesn't matter if those accounts are new (created on that PDS) or migrated; in both case they count against the quota.

Moderation systems are expected to have the option to track things like the total number of bad accounts per PDS, both as absolute numbers and ratios. But relying on context like "what PDS is hosting the account" or "what IP address" or "what email domain" are themselves problematic, and it is individual accounts (DIDs) which are the primary subject of moderation.

[sneakers-the-rat]

Alright! thanks again. clear on my questions now. the relay 'pulling' rly changes the dynamics i was thinking about, so thx for your patience explaining that. makes it much clearer how the relay throttling would work.

yes yes, some of the friction points that make identity costly are v problematic to moderate with, but also super useful for handling abuse at scale, so I'll be very curious to see how the moderation systems hold when the only point of moderation is the effectively free DID. hoping y'all got your redteam galoshes on mucking around in that sandbox, glhf on opening federation <3

[singpolyma]

it is worth pointing out that there is nothing stopping other orgs from running their own Relay

Is this available somewhere to play with? Or you just mean one could read the spec and write a relay in theory?
(I did try to look for it, but maybe I'm just looking in the wrong places)

[snarfed]

@singpolyma yes! code is in https://github.com/bluesky-social/indigo , afaik mostly in bgs. (That repo also has many of the other official underlying services and tools.) Some minimal docs on running their relay in https://github.com/bluesky-social/indigo/blob/main/cmd/bigsky/README.md

[surfdude29]

FYI the link in Timestamp semantics doesn't seem to be working any longer, can now be found at:

https://docs.bsky.app/docs/advanced-guides/timestamps

[bnewbold]

updated, thanks!