[bug] unable to follow lemmy communities

[terinjokes]

Describe the bug with a clear and concise description of what the bug is.

I’ve tried on a few different occasions to follow various lemmy communities, however my requests always get stuck in “Requested”. I’m able to follow many Mastodon instances without issue, so I think my federation setup is working generally. When I search for the community in my instance, the profile data and icon are fetched without issue.

What's your GoToSocial Version?

V0.13.3

GoToSocial Arch

amd64 binary

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

1. Search for https://lemmy.ml/c/sopranica in GtS instance.
2. Send follow request.
3. See that the request is never fulfilled later.

Anything else we need to know?

No response

[tsmethurst]

Possibly related: #1468

[tsmethurst]

Also possibly related: LemmyNet/lemmy#4451

[daenney]

I'm reasonably certain it's an authorised fetch issue. You should be able to see it if you look at the logs when you do a request. Lemmy should be trying to deliver an "accept" of the follow request (you should see an incoming request with a Lemmy user agent) but it'll lack a signature/fail validation. In which case we drop it on the floor.

[tsmethurst]

Mm, POST requests to an inbox should be authorized. It's just GET requests that Lemmy doesn't (by default) sign.

[terinjokes]

The other issue linked above implies that following should work, just that replies doesn't. I'll try to pull logs in a moment.

[terinjokes]

Mar 03 09:18:28 nixos04 gotosocial[776]: timestamp="03/03/2024 09:18:28.727" func=server.glob..func1.Logger.func13.1 level=INFO latency="221.348µs" userAgent="Lemmy/0.19.3; +https://lemmy.ml" method=GET statusCode=401 path=/users/terinjokes clientIP=54.36.178.XXX errors="Error #01: AuthenticateFederatedRequest: http request wasn't signed or http signature was invalid\n" requestID=ey4sp0we04001qxcdjx0 msg="Unauthorized: wrote 104B"

[tsmethurst]

Mmmhmm, those logs confirm what I was thinking. If Lemmy doesn't sign their GET requests then they won't be able to fetch your Actor to see who's making the Follow request. So they won't be able to send anything back to you either, because they don't know what your Inbox address is, or who you even are. This means that Lemmy also probably doesn't work with mastodon instances that have secure mode switched on.

[Nutomic]

Lemmy has an admin setting for authorized fetch which Ive enabled on our test server. It would be good if you can follow one of the communities and tell me what is wrong with the signature. In principle there should be no problem as its using the same signing logic as for inbox activities.

[tsmethurst]

@Nutomic I tried but got an error that gts.superseriousbusiness.org was not in the allowlist for your test server. Could you add it?

[Nutomic]

Sorry my bad, use ds9.lemmy.ml which has open federation and also has signed fetch enabled.

[tsmethurst]

Alrighty, here's what happens:

timestamp="06/03/2024 15:41:40.308" func=workers.(*Processor).ProcessFromClientAPI level=INFO activityType=Create objectType=Follow fromAccount=dumpsterqueer requestID=s20kc54e04001j7k5msg msg="processing from client API"
timestamp="06/03/2024 15:41:40.309" func=federation.(*federatingActor).Send level=INFO requestID=s20kc54e04001j7k5msg msg="send activity Follow via outbox https://gts.superseriousbusiness.org/users/dumpsterqueer/outbox"
timestamp="06/03/2024 15:41:40.324" func=httpclient.(*Client).DoSigned level=INFO method=POST url=https://ds9.lemmy.ml/inbox requestID=s20kc54e04001j7k5msg pubKeyID=https://gts.superseriousbusiness.org/users/dumpsterqueer/main-key msg="performing request"
timestamp="06/03/2024 15:41:40.400" func=server.glob..func1.Logger.func13.1 level=INFO latency="3.308938ms" userAgent="Lemmy/; +https://ds9.lemmy.ml" method=GET statusCode=401 path=/users/dumpsterqueer clientIP=116.203.31.174 errors="Error #01: AuthenticateFederatedRequest: http request wasn't signed or http signature was invalid\n" requestID=5p13c54e04001pd6rb10 msg="Unauthorized: wrote 104B"
timestamp="06/03/2024 15:41:40.415" func=workers.(*clientAPI).CreateFollowReq level=ERROR requestID=s20kc54e04001j7k5msg msg="error federating follow request: Follow: error sending activity *typefollow.ActivityStreamsFollow via outbox https://gts.superseriousbusiness.org/users/dumpsterqueer/outbox: func1: error delivering to https://ds9.lemmy.ml/inbox: deliver: POST request to https://ds9.lemmy.ml/inbox failed: status=\"400 Bad Request\" body=\"{\"error\":\"unknown\",\"message\":\"Failed to parse object https://gts.superseriousbusiness.org/users/dumpsterqueer with content {\\\"error\\\":\\\"Unauthorized: http request wasn't signed or http signature was invalid: (verifier)\\\"}: data did not match any variant of\""

So GtS makes the POST to Lemmy. Looks like while the POST request is still pending, Lemmy tries to fetch the Actor who did the POST. Signature authentication fails for whatever reason (you sure that GET request is signed?), so GtS returns 401 to Lemmy. And then Lemmy returns Bad Request to GoToSocial, thereby terminating the POST.

[Nutomic]

Yes those steps sound normal. I was hoping you could tell me what exactly is wrong with the signature of the GET request. From what I can see it should definitely be signed in the same was as POST requests to the inbox.

[tsmethurst]

As far as I can tell, it's just not signed. Or not signed in a way that GtS even recognizes as a signature. Could you link me to your HTTP signature logic?

[Nutomic]

Okay I found a problem where the private key wasnt passed in properly. This is fixed now on ds9, can you give it another try?

I did try to fetch from a gotosocial instance myself, but it just returns internal server error. And fetching from a Mastodon instance which requires signed fetch throws Actor https://ds9.lemmy.ml/ has no 'preferredUsername', which is a requirement for Mastodon compatibility which is very odd, why would the instance actor need a username?

[tsmethurst]

GtS has a similar complaint to mastodon:

timestamp="07/03/2024 13:39:46.912" func=server.glob..func1.Logger.func13.1 level=ERROR latency="218.706077ms" userAgent="Lemmy/0.19.3-68-gd043d6dda; +https://ds9.lemmy.ml" method=GET statusCode=500 path=/users/dumpsterqueer clientIP=116.203.31.174 pubKeyID=https://ds9.lemmy.ml/#main-key errors="Error #01: AuthenticateFederatedRequest: error dereferencing account https://ds9.lemmy.ml/: enrichAccount: error converting accountable to gts model for account https://ds9.lemmy.ml/: ASRepresentationToAccount: unusable username for https://ds9.lemmy.ml/\n" requestID=0n2yt64e040014mjkxh0 msg="Internal Server Error: wrote 54B"

It oughtn't return Internal Server Error but nevermind.

why would the instance actor need a username

while preferredUsername is an optional property according to AP afaik most implementations (including GtS and Mastodon) use that username for quite a lot of things internally

the "canonical" way to do instance actor usernames is to just use the hostname of the instance, so in this case something like "preferredUsername": "ds9.lemmy.ml" would probably get things working

[Nutomic]

Thats odd considering you can easily extract the domain from the url. Anyway Ive added that field and deployed it. But now Mastodon is throwing Webfinger error when resolving ds9.lemmy.ml@ds9.lemmy.ml: Request for acct:ds9.lemmy.ml@ds9.lemmy.ml returned HTTP 400. Why does it require webfinger for the instance actor???

[tsmethurst]

Bear in mind that Mastodon etc don't necessarily know it's the instance actor. Also bear in mind that "instance actor" isn't even a concept in ActivityPub full stop, it's just a convention that many implementations have adopted (in my opinion, wrongly, but that's another story...). So Mastodon is treating the instance actor just like any other actor and trying to resolve it through webfinger. I believe GoToSocial will fail on the same hurdle, but I don't have time to check right now.

Generally speaking, webfinger is a good way of checking "does this account really exist on this server, according to the server itself". This avoids issues where a user on an instance just uploads some JSON that looks like an actor, and points people to it. If remote servers can't resolve the supposed actor using webfinger, they know it's not legit (just one example).

[daenney]

Webfinger works for the instance actor on GtS. For example, the GtS instance: https://gts.superseriousbusiness.org/.well-known/webfinger?resource=acct:gts.superseriousbusiness.org@superseriousbusiness.org

As long as its a local account, which the instance actor is, webfinger will work.

But yeah, if GtS sees an account by Lemmy's instance actor, it'll try and resolve it using Webfinger, in order to amongst other things determine where to retrieve the signing key from. Same for Pleroma/Akkoma and the Misskeys I believe.

[Nutomic]

Alright I added those changes and again deployed them to ds9.lemmy.ml. It seems to be working now, I can fetch a Mastodon user which requires authorized fetch.

[NyaaaWhatsUpDoc]

@Nutomic i just attempted a follow of one of your test users on ds9.lemmy.ml and am getting the following error:

timestamp="02/04/2024 11:13:17.087" func=httpclient.(*Client).DoSigned level=INFO method=POST url=https://ds9.lemmy.ml/inbox requestID=1spr77me04001see5t2g pubKeyID=https://k.iim.gay/users/kim/main-key msg="performing request"
timestamp="02/04/2024 11:13:17.183" func=server.init.func1.Logger.15.1 level=INFO latency="4.975793ms" userAgent="Lemmy/0.19.3-76-g255e69563; +https://ds9.lemmy.ml" method=GET statusCode=200 path=/users/kim pubKeyID=https://ds9.lemmy.ml/#main-key requestID=f9pr77me04001fa0k5d0 msg="OK: wrote 1.58kiB"
timestamp="02/04/2024 11:13:17.216" func=server.init.func1.Logger.15.1 level=INFO latency="7.791935ms" userAgent="Lemmy/0.19.3-76-g255e69563; +https://ds9.lemmy.ml" method=GET statusCode=406 path=/ errors="Error #01: no format can be offered for requested Accept header(s) [application/activity+json]; this endpoint offers [text/html]\n" requestID=k5pr77me0400089w2mp0 msg="Not Acceptable: wrote 145B"

any idea why you're hitting the root level path in response to the AP request flow?

[Nutomic]

Lemmy has the instance actor at / which is used to federate the instance description and some other things. However thats optional so no problem if it returns an error.

[daenney]

Can we conclude this is fixed now? It doesn't sound like there's anything left to do.

[tsmethurst]

I guess if someone on GtS can follow a lemmy actor and get posts delivered then it should be OK. @Nutomic can you confirm whether that's the case now?

[Nutomic]

You can try following https://ds9.lemmy.ml/u/nutomic

[terinjokes]

I haven't upgraded GtS to 0.15 yet, but I was able to follow the "nutomic" user account from my instance as well as the "ZZZ Test Community 1" community.

[daenney]

Nothing changed on the GtS side related to this, so no need to upgrade (though do upgrade, it's a good release 😄).

Sounds like the implementation changes in Lemmy have resolved the issue and everything's working correctly now with instances that enforce authorized fetch.

I'll close this for now, but if you notice there's still problems feel free to reopen it.