Migrating bsky.social to Multiple PDS Instances (Nov 2023) #1832
Replies: 6 comments 45 replies
-
|
Pinging some specific folks about this change. As with previous pings, not trying to promote/exclude anybody; and sorry if this is spammy. Feed generators: there are two parts of this: fetching feeds from a migrated account, and posts from a migrated account getting indexed. I have not seen any issues with fetching third-party feeds, but don't have enough data to verify that indexing is working for many (though no evidence that it is not). Tested "For You" (skygaze.io), "Blacksky" (@rudyfraser), "Catch Up" (skyfeed.xyz, @redsolver), "Home+", "Discover", "Likes" (@jcsalterego) Clients: there is a lot of surface area to cover here, but tested briefly with GreySky (@mozzius) and seems functional. Libraries: it looks like the great python library that @MarshalX maintains had an issue with additional fields included in JWTs. This is partially on us for making that particular change without thinking that it could impact downstream folks. Looks like there was a fast fix and upgrading fixes the problem. |
Beta Was this translation helpful? Give feedback.
-
|
Quick question, when will the bsky.social PDS firehose stop including all events? One of my servers is still using that endpoint, so I would like to know if I need to do the switch today or tomorrow is enough, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Just checked |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Yup - I suppose this is the kick I need to finally move FurryList over to BGS 😓 |
Beta Was this translation helpful? Give feedback.
-
|
So to confirm: bsky.network will bring all traffic in one place? Is a firehose even possible in a fully distributed future? |
Beta Was this translation helpful? Give feedback.
-
|
It is! https://atproto.com/guides/overview#achieving-scale , https://blueskyweb.xyz/blog/5-5-2023-federation-architecture |
Beta Was this translation helpful? Give feedback.
-
|
deck.blue's dev here One thing that I need to test is the trick I'm using to show uploaded GIFs, we're just using the direct getBlob call and if that's broken between PDS instances, the trick won't work if any of the test accounts upload a .gif file I'll keep an eye for any other issues and let you guys know! Also pinging @myConsciousness so he knows about this |
Beta Was this translation helpful? Give feedback.
-
|
Perhaps related to @rudyfraser's thread |
Beta Was this translation helpful? Give feedback.
-
|
Can you elaborate on the getBlob redirect? I use that for image classification right now. Do I need to do anything differently besides using the bsky.social endpoint? |
Beta Was this translation helpful? Give feedback.
-
|
It's the same token issued by the user's home server, right? The client treats it as opaque and doesn't modify in any way. So a rogue PDS can use it to impersonate the user to the PDS that hosts their account. Or am I missing something? |
Beta Was this translation helpful? Give feedback.
-
|
Ohhh I think I see what you mean. I don't think a trusted PDS should ever redirect to a non-trusted PDS. But, if there is a 302 redirect, I don't think you should ever send auth tokens to the redirect target in any case (the |
Beta Was this translation helpful? Give feedback.
-
|
It's always OK to send your auth tokens to Jerry's PDS Emporium |
Beta Was this translation helpful? Give feedback.
-
|
Yup, I also left a comment to that end here golang/oauth2#500 (comment) For "normal" web services it's a bit less of an issue, since use cases for redirecting to a third-party host are carved out well enough. But in a federated environment it can bite your ass pretty hard. I didn't find any specific requirements for removing headers on redirect in RFC7231, so client behaviour might vary :-\ |
Beta Was this translation helpful? Give feedback.
-
|
@redsolver @videah @gildaswise My understanding is that Dart's http package handles redirects automatically. So since my packages also use the http package, we will not have to be aware of this redirect. If you have any problems with my part of the package, please let me know. |
Beta Was this translation helpful? Give feedback.
-
|
Hey - just wanted to clarify something as we've noticed an unexpected change in behaviour. As part of our logic, when we onboard a new actor to the FurryList feeds, we fetch the latest commit cid for the user using Since actors have started to be migrated to the new PDS, calling My understanding from reading this document was that we should expect most calls to bsky.social to continue working correctly. If this starting to fail is expected behaviour, should we instead be calling these RPCs on the BGS? The AppView? Or should we be determining the address of the PDS for that individual user and making the request directly there instead? nb: I'm aware that we need to move off Many thanks and apologies if I'm missing something here! |
Beta Was this translation helpful? Give feedback.
-
|
That was my first guess, but unfortunately, if you change your profile back to something it was previously, the CID is then the same. It feels "incorrect" to me to not be able to distinguish between those two points in time. I may just have to bite the bullet on it and do that. |
Beta Was this translation helpful? Give feedback.
-
|
If you want a point-in-time thing you can use the |
Beta Was this translation helpful? Give feedback.
-
|
Hm - I think I actually figured out how to get the Commit CID from the results of package main
import (
"bytes"
"context"
"fmt"
"github.com/bluesky-social/indigo/api/atproto"
"github.com/bluesky-social/indigo/repo"
"github.com/bluesky-social/indigo/xrpc"
"github.com/ipfs/go-datastore"
blockstore "github.com/ipfs/go-ipfs-blockstore"
)
func main() {
ctx := context.Background()
x := &xrpc.Client{
Host: "https://bsky.social",
}
myDID := "did:plc:dllwm3fafh66ktjofzxhylwk"
latestCommitOut, err := atproto.SyncGetLatestCommit(ctx, x, myDID)
if err != nil {
panic(err)
}
buf := new(bytes.Buffer)
params := map[string]interface{}{
"collection": "app.bsky.actor.profile",
"did": myDID,
"rkey": "self",
}
if err := x.Do(ctx, xrpc.Query, "", "com.atproto.sync.getRecord", params, nil, buf); err != nil {
panic(err)
}
data := buf.Bytes()
bs := blockstore.NewBlockstore(datastore.NewMapDatastore())
root, err := repo.IngestRepo(ctx, bs, bytes.NewReader(data))
if err != nil {
panic(err)
}
fmt.Println("Latest commit : " + latestCommitOut.Cid)
fmt.Println("Root from SyncGetRecord: " + root.String())
// These two are the same
}This works fine until I point it at |
Beta Was this translation helpful? Give feedback.
-
|
Calling the
The data returned by the BGS is not enough for |
Beta Was this translation helpful? Give feedback.
-
|
Looking at the implementation of |
Beta Was this translation helpful? Give feedback.
-
|
We have an academic project that we think could be pretty exciting and I'l love to talk with someone at BlueSky about what federation would mean for it. The project would require running, and generating feeds for, thousands of accounts as a botnet. But it would be benevolent botnet. We were originally thinking of the bird app, but then Elon blew that up and took away "Twitter for Good", which would have been a natural partner. We've been considering our own mastadon server, which we know could work in principle, but Bluesky federation might give us another option. Is there any way to set up a discussion on this? I don't want to sidetrack the discussion board for a niche conversation. |
Beta Was this translation helpful? Give feedback.
-
|
Sure, i'll send you an email. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. Just added my email address to my profile. |
Beta Was this translation helpful? Give feedback.
-
As part of our operational work in preparation for federation, we are starting to migrate PDS accounts away from the existing monolithic
bsky.socialinstance to several*.*.host.bsky.networkPDS instances. We have implemented a new "account entryway" service as a "virtual PDS" which will continue to operate atbsky.social.These changes are expected to roll out on the main network the week of 2023-11-06 (aka, this week, starting today). Every Bluesky-hosted account will be migrated to a new PDS, which will involved updating DID PLC documents with new PDS service locations and new signing keys. The roll-out may take longer than a week.
What will not change
bsky.socialas an API endpoint, and requests will be proxied to the appropriate endpoint. They do not need to connect directly to the account's PDS for now, especially for authentication (login and session refresh)api.bsky.app) is unaffectedWhat will change:
bsky.social; subscribe to the BGS firehose (bsky.network) to get all events from all accounts.bsky.socialbsky.socialendpoints may return 3xx redirects to the specific PDS endpoint, instead of proxying. Specifically,com.atproto.sync.getBlob(for downloading images and other media). Most clients and applications use the AppView-provided CDN URLs, not blobs directly.bsky.sociallogin endpoint, which allows clients to connect directly to the back-end PDS if they want (for potentially better network latency)Will my stuff be impacted?
Regular accounts using Bluesky App: should be seamless. We expect that folks will not even need to re-log in unless something goes wrong.
Client Applications: same as the Bluesky App.
Bots: should be mostly unaffected; same as the Bluesky App.
Feed Generators: if verifying firehose repo signatures, will need to detect identity signing key changes (eg, via "cache busting"). Likewise, if verifying JWT auth token signatures, will need to detect identity changes. If subscribing to legacy PDS firehose, will need to switch to BGS firehose.
Accounts (repos) with
did:webDIDs: we will probably need to coordinate with you to update DID documents, stay tuned. Updating to newer DID document formatting at the same time would be great: #1510Sandbox PDS operators: we haven't pushed this round of code changes to the sandbox version of our PDS implementation. These changes will be disruptive when they do land, and will probably require a full wipe for most instances. The PDS reference implementation will only support sqlite (not postgresql), will use a distinct signing key for each account, etc.
Independent PDS implementations: none of these changes will be required by protocol. PDS operators will be free to keep the "account entryway" and "repo hosting" functionality combined in a single service, and can continue to use a single signing key for multiple accounts if they like. This is just a deployment pattern we have chosen to scale out our service for millions of accounts.
A follow-up comments to this post will tag some specific libraries and services which we have tested against.
What is "identity cache busting"?
Federated services in atproto need to keep track of account identity metadata, including handle and DID document resolution. Clients can make requests to PDS instances for this information, but even they are likely to cache the results. Independent services will usually do direct resolution with persisted caching, similar to how DNS caching works.
Our current recommendation is that services cache identity metadata for between 8 and 24 hours. They may also simply rely on native DNS TTL and HTTP caching header information to determine an appropriate cache lifetime. Large services may want to "subscribe" to the DID PLC event log instead of re-fetching information for individual accounts.
In addition to this time-limited caching policy, we recommend automatically "busting" the cache when there is an indication that the identity may be stale. For example, if a repo signature or JWT signature fails to validate. A best practice is to have a separate shorter caching window for "bust" operations, so that failures do not result in a busy-loop of many rapid requests when an identity is in a broken state.
Tell me more about this "virtual PDS" thing
More details coming soon, but this is mostly a mechanism to take advantage of the federated nature of the protocol to help us grow the Bluesky-hosted network beyond a single big database server. The "virtual PDS" handles some of the organization-wide "glue" tasks, like maintaining the
*.bsky.socialhandle namespace, private account email addresses (which should be unique across all of our hosted accounts), and routing new account signups to a particular PDS instance.We'll soon communicate more details of this architecture, and how clients can connect directly to backing PDS instances (for better network performance).
Beta Was this translation helpful? Give feedback.
All reactions