Add follower synchronization mechanism

[ClearlyClaire]

Fixes #14480

Everything in here is an ActivityPub extension that is not exactly in ActivityPub's spirit (since it's domain-based, and introduces synchronization mechanisms that are not a thing in AP to my knowledge) but consistent with the assumptions implementations already make (remote actors being able to create objects with any identifier as long as it is on the same domain name as the actor itself, for instance).

While the PR should be in good shape, everything, especially the vocabulary, may change from feedback from other implementors, as the issue is not strictly a Mastodon issue, but a more general one.

Related discussion: https://socialhub.activitypub.rocks/t/proposal-mechanism-for-synchronizing-followers-lists-across-servers/852

Context

There are many ways—such as software bugs (e.g. the one fixed by #14479, Pleroma forgetting about followers a while ago, and probably other bugs we did not catch or may happen in the future), incompatibilities (while initiating and accepting a follow is pretty well-defined in AP, forcefully removing followers is not), prolonged downtimes, instances being brought back from the dead or restored from backups—in which two instances A and B, acting in good faith, may not agree with who from A follows someone on B.

This may cause confusion and frustration as one may think they are followed/follow by someone they are not. Worse: with Mastodon’s follower-only privacy scope, this can get particularly damaging, as receiving remote instances are asked to deliver the message to who they know to follow the poster: if that gets out of sync, then the messages may reach people that the poster expects to not have access to.

Therefore, a mechanism to detect synchronization issues and correct them is needed. Such a mechanism has to take into account that the full list of an account’s followers may not be public information, and should thus only consider followers residing on a specific domain.

Proposal

I propose the addition of an optional Collection-Synchronization HTTP header following the same syntax as the Signature header, with the following attributes:

• collectionId: must be the sender's followers collection
• url: an URL to a partial collection containing the identifiers of the sender's followers residing on the receiver's instance. MUST reside on the same domain as the actor itself, and SHOULD be only accessible with a signed query from the receiver's instance
• digest: hexadecimal representation of the XORed SHA256 digests of each of the identifiers in the partial collection

Example

When delivering a private message to https://mastodon.social/users/foo/inbox, the instance running on social.sitedethib.com sets the following HTTP header:
Collection-Synchronization: collectionId="https://social.sitedethib.com/users/Thib/followers", url="https://social.sitedethib.com/users/Thib/followers_synchronization", digest="b08ab6951c7d6cc2b91e17ebd9557da7fae02489728e9332fcb3a97748244d50"

The result of querying https://social.sitedethib.com/users/Thib/followers_synchronization (when signing the request on behalf of a mastodon.social account):

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "https://social.sitedethib.com/users/Thib/followers?domain=mastodon.social",
  "type": "OrderedCollection",
  "orderedItems": [
    "https://mastodon.social/users/Gargron"
  ]
}

FAQ

Why partialCollection, why not re-use the followers collection?

Currently, Mastodon lists either all followers in the followers collection, or none of them (if the user has opted in to “hide their network”). The first behavior is not suitable for synchronization, because it would be potentially much larger than needed. The second would obviously not work.

The behavior we want from partialCollection, that is, listing all known followers residing on the requesting instance, is different from those two options. While we could switch to that behavior when receiving a signed request:

1. Implementations may still want a chance to get the full list of followers when signing a request (e.g., implementations allowing access to the full list of followers, but only to followers)
2. Mastodon currently signs all outgoing requests, and uses the fact that the collection cannot be paginated to respect the “hide your network” option across instances.

Wouldn't that leak followers information?

No, if the partialCollection requires an HTTP Signature, and only returns followers residing on that instance, it will only return information that the requester is supposed to know about (but might not because of bugs or system failures).

Also, the digest is only delivered to the inbox via an HTTP header, unaware applications will simply ignore it, and will not risk relaying it anywhere.

Implementation notes

• caching hashes
  Hashes are cached in Rails' cache, and the cache gets invalidated when followers are created or removed.
  The cache key used is followers_hash:#{account_id}:#{domain} where domain is the remote domain we are interested in if the account is local, or local if it's a remote account (and we thus are interested in the local followers).
• followers synchronization endpoint
  The followers synchronization endpoint provided by this PR is separate from the regular followers collection endpoint, and does not expect a parameter for the requested domain. Indeed, a parameter would be redundant as the endpoint requires a signed fetch, and the used domain is retrieved from that signature.
  In Mastodon's case, having a separate endpoint actually makes the code less complex and easier to reason about.

TODO

• Implement receiving end
• Add tests for receiving end
• Implement sending end
• Add tests for sending end

[cwebber]

This seems to assume that a complete list of who I'm following is public information, whereas it might only be a partial subset? This could be a privacy problem?

Unless it's a hash only for that domain. That could work, though it makes me unhappy because I don't like the assumption of ActivityPub being a domain-oriented system, which I think will create problems long term...

[ClearlyClaire]

The partialCollection would only contain what the remote domain (identified by domain) is supposed to know, and the hash is indeed specific to it.

I understand your concern, but it is precisely in the context of domain-oriented systems where the issue arises.

[cwebber]

In general I think that a "do I really have the correct state" synchronization system is fine and welcome; I wonder if it can be more general though than just related to followers. The hash part at least is elegant and a good idea. For that component, maybe even better would be to use a bloom filter, so that if a mismatching result is found, one could at least query to see if certain elements are part of the set.

I do wonder if there's a more generalized pattern; I've thought for a while that there's something to the Unum pattern that could be followed.

[cwebber]

Yes, I realize I'm pushing more abstract patterns for people who are doing more practical "AP as deployed" approaches (a legitimate criticism).

The domain approach could be generalized though: instead of domain, you could have "node"; conceptually, each Actor could list on its profile the "Node" actor that it's associated with corresponding to its hub in the hub-and-spoke model.

This then could be the actual node that you would query to see whether or not your followers were in that set.

The main problem I guess with that approach is to make it secure you need to have each Node/Hub sign that the actor really does belong to that node/hub.

[ClearlyClaire]

For that component, maybe even better would be to use a bloom filter, so that if a mismatching result is found, one could at least query to see if certain elements are part of the set.

I have thought of that. It might be a good idea. The workflow with this would be, if there's a mismatch, use the provided bloom filter to prune local followers, re-compute the local bloom filter, and if it still doesn't match, perform the HTTP query. This would spare an HTTP query in many (most?) cases, but I am unsure it is worth the computational or code complexity (I'll have a look).

I do wonder if there's a more generalized pattern; I've thought for a while that there's something to the Unum pattern that could be followed.

I will have a look.

The domain approach could be generalized though: instead of domain, you could have "node"; conceptually, each Actor could list on its profile the "Node" actor that it's associated with corresponding to its hub in the hub-and-spoke model.

I'm afraid a lot of security assumptions are already made across most if not all ActivityPub software around the fact that two URLs behind a same domain are managed by the same logical entity. Introducing a concept of managing “Node” might be too late.

[ClearlyClaire]

Alright, I think I have roughly finished a first draft of that feature.

Some notes:

• while a bloom filter could be useful, that's probably overkill, the feature isn't supposed to kick in often anyway. And since the same algorithm with the same parameters would need to be used across implementations, that is probably not worth it.
• I guess we can consider the collection an Unum, but I am not sure what that gives us. I don't believe the article linked earlier really delves into anything useful for the case at hand.
• There is something I'm a bit worried out: different implementations considering different “canonical” domain names for accounts (e.g. when LOCAL_DOMAIN ≠ WEB_DOMAIN in Mastodon). I have a hunch this isn't an issue unless some implementation considers accounts from the same logical instance under different domain names, but I'm too tired right now to think more about this.

[ClearlyClaire]

Re-using existing ActivityStreams vocabulary:

• collection could be subject (in AS, only used for relationships) or describes (in AS, only used for profiles), simply object, or even partOf (used for Collections pages)
• partialCollection could be url
• I can't think anything that would be suitable for domain in AS itself, but https://w3c-ccg.github.io/security-vocab/#domain somewhat fits
• as far as the hash is concerned, it probably would make sense to use https://w3c-ccg.github.io/security-vocab/#Digest even if it is more verbose

[lanodan]

Sounds like a good proposal to me, few small things though:

• domain: Isn't it actually the hostname? (and it's the one from the ActivityPub id not webfinger, right?)
• hash: I'm not sure about putting it there, maybe a last-modified date would be better, also SHA256 is fine right now but might not be in some years.

Also why not use authenticated fetching (HTTP Signatures) so that the followers endpoint just keeps working and is refreshed periodically, like the actor currently is in pleroma and mastodon?

[ClearlyClaire]

Sounds like a good proposal to me, few small things though:

* `domain`: Isn't it actually the hostname? (and it's the one from the ActivityPub id not webfinger, right?)

It's the domain part of the handle, so it's the webfinger thing. I have to think a bit more whether that would cause issues with software that use the AP hostname instead. My hunch is that it wouldn't cause issues as long as:

1. On the receiving end, when checking the domain, they accept both the webfinger thing and the hostname of AP endpoints (that's what my implementation does)
2. On the sending end, when computing the domain-specific hash, all of the remote followers on the remote instance have to be considered with the same domain. Meaning that two accounts on domain.org served by social.domain.org need to be referred both as domain.org or social.domain.org, not a mix of that. Mastodon will always use the webfinger name, and I think Pleroma will always use the AP hostname, but I am not completely sure about it.

* `hash`: I'm not sure about putting it there, maybe a last-modified date would be better, also SHA256 is fine right now but might not be in some years.

I think something like Last-modified would be kinda error-prone, plus you have to store it and can never re-calculate it.

About SHA256, I guess that would be solved by using https://w3c-ccg.github.io/security-vocab/#Digest as I suggested in my last post.

Also why not use authenticated fetching (HTTP Signatures) so that the followers endpoint just keeps working and is refreshed periodically, like the actor currently is in pleroma and mastodon?

Depending on the user's settings, the followers endpoint can currently list:

• all of the followers
• none of the followers

Having a third option (only the followers from the requesting domain) tied to HTTP Signatures does not make a lot of sense (you may want to know all the followers). Plus, currently, Mastodon figures out whether a remote user wants to show their followers by checking if they can access that collection's first page. That assumption would be broken by having that third option.

Furthermore, periodic profile checking in Mastodon is actually not done for remote accounts which are followed: indeed, Mastodon relies on receiving Updates to get the up-to-date info and avoid spurious fetches. One may argue that this poses a similar synchronization issue, though, although one that doesn't put privacy at risk.

In my implementation, I found that using a separate endpoint was simply easier.

[Gargron]

I'd unite this with the line above. Maybe extract to disable_followers_synchronization? method for easier testing.

[ClearlyClaire]

Put it on one line, not sure about a different method.

[Gargron]

Nitpick but I would maybe write "X-AS-Collection-Synchronization" here to be more specific in the error log. Though I also don't remember how we treat errors from "bad input" in other places. Maybe you wouldn't want to fill the warn log level with stuff like this..?

[ClearlyClaire]

I think in general, Mastodon doesn't print enough stuff in logs, but maybe that should be debug loglevel rather than warn. Updated the text itself.

[Gargron]

Could be compacted to one line by doing Account.find and rescuing the error like we do in all other workers.

[ClearlyClaire]

This is a pattern I do not like that much as it could silently hide unexpected ActiveRecord::RecordNotFound exceptions.

[tocharomera]

Hello👋

how could this be triggered in a self-hosted Mastodon server for a specific account?

A mastodon server database I managed got corrupted and I lost some followers on the recovery process on some accounts, now they still follow the accounts on their profiles but they dont show up on the accounts followers section of my server.

Could you give me an advise how to recover them?
Or are they lost forever?