Bluesky = fediverse: can't yet handle load from influx of new Brazilian users

[eusousu]

Instructed my friends migration to bluesky to follow https://bsky.app/profile/ap.brid.gy so I could follow them on mastodon. But they never received the DM informing their mastodon handle and trying to retrive them directly from https://fed.brid.gy/[user] returns 404

Is it overloaded? Is it broken? Are us doing something wrong?

I followed from mastodon and it created my bsky handle in less than a minute and my posts are being bridged perfectly

[Huio-op]

+1
Also having the same issue, my friends and I all followed the @ap.brid.gy account and received no messages.

[snarfed]

Hey all, sorry for the trouble! Bridgy Fed's Bluesky => fediverse direction is indeed overloaded and backed up right now due to the huge wave of new users and usage (10x normal!) from Brazil. It's working through the backlog, but it's definitely behind.

I've been optimizing a few things, and I have at least one or two more things I can do that won't take a bigger redesign, but I won't have much time this weekend to work on it. We'll see. Fingers crossed!

https://bsky.app/profile/pfrazee.com/post/3l2xupwbsfy2f

https://bsky.app/profile/bnewbold.net/post/3l2yppyjo6p2k

[eusousu]

ohhh that's really a lot! Would it be possible to have a mean time for process completion on the main site somewhere?

people on bluesky are usually my least tech savy friends and asking them to follow an account for me to be able to follow them is already a difficulty to explain. When the process breaks the expectation it can lose the opportunity window and they give up the idea altogether as a broken thing (reason I could not convince them to create mastodon accounts again after 2022 exodus fail)

[snarfed]

Totally understood and agreed! And monitoring for our backlog is a good idea, at least internal if not public too. I've filed #1296 for that.

[qazmlp]

people on bluesky are usually my least tech savy friends and asking them to follow an account for me to be able to follow them is already a difficulty to explain. When the process breaks the expectation it can lose the opportunity window and they give up the idea altogether as a broken thing (reason I could not convince them to create mastodon accounts again after 2022 exodus fail)

On that note, it would be great if there was a DM to inform users who get caught in the spam filter of the requirements, too.
Right now it just looks like it fails silently if the account isn't old enough, if I'm not mistaken.

(I'd file this as proper feature request, but I'm a bit 🫠 today. Maybe tomorrow.)

[snarfed]

So, overall firehose event rate is up something like 10x, from 100-200/s to 1-1.5k/s. I've gotten our event handling rate up a bit, but not yet enough to keep up, much less work through the backlog. So far I've tried:

• moving all of our remaining logic (there wasn't a lot) off the main firehose client thread
• switching CBOR parsing lib from hashberg dag-cbor to libipld (helps, but has a bug that makes us miss commits for our bridged users, which I haven't figured out yet)
• even parallelized parsing itself

Still haven't managed to get higher than mid 100s of events/s, nor have I gotten CPU utilization above 1.5-2 cores, which implies that the websocket client itself is the bottleneck.

A bit more background in snarfed/arroba#39, but only somewhat relevant.

https://bskycharts.edavis.dev/static/dynazoom.html?plugin_name=edavis.dev%2Fbskycharts.edavis.dev%2Fbsky&start_iso8601=2024-08-25T12%3A00%3A03-0700&stop_iso8601=2024-09-01T16%3A15%3A39-0700&start_epoch=1724612403&stop_epoch=1725232539&lower_limit=&upper_limit=&size_x=800&size_y=400&cgiurl_graph=%2Fmunin-cgi%2Fmunin-cgi-graph

(doesn't include a number of event types, eg #account, #identity, #tombstone, maybe deletes, etc)

[therealNAAN]

I'm glad someone noticed and made this issue, because I wondered if it was my internet or if something had happened to this service. I don't envy the tough work needed but I am hoping for no working too hard! After all, breaks are important D: Especially with this massive scaling, jeez...Thank you for all your hard work, snarfed!

[snarfed]

Finally managed to optimize our firehose processing from ~200 events/s up to 3-4k/s. Bluesky => fediverse bridging backlog is down from its high of 3d to just 18h now, should be caught up and back to realtime in another 5h or so.

[snarfed]

Done! Back to realtime.