Question: Mastodon AUTHORIZED_FETCH not working?

[Tealk]

Question

Hello,

can it be that Lemmy cannot talk to Mastodon instances if the instance AUTHORIZED_FETCH is activated?

lemmy_1          |              at src/root_span_builder.rs:16
lemmy_1          | 2024-02-13T17:32:37.659047Z  WARN lemmy_server::root_span_builder: Unknown: Failed to parse object https://rollenspiel.social/users/Tealk with content {"error":"Request not signed"}: data did not match any variant of untagged enum PersonOrGroup
lemmy_1          |    0: lemmy_server::root_span_builder::HTTP request
lemmy_1          |            with http.method=POST http.scheme="http" http.host=rollenspiel.forum http.target=/inbox otel.kind="server" request_id=c0083b1f-5173-4cdb-9b30-86e747d7c1dd

[Nutomic]

You need to enable local_site.federation_signed_fetch for this. Its not implemented in lemmy-ui yet, so either set it directly in the db, or with a manual api request.

[Tealk]

manual api request

Can you tell me how?

[Nutomic]

Firefox has an option to copy the request from dev tools as curl command, then edit it and add the param. Otherwise simply edit directly in the db with psql.

[Tealk]

shouldn't the source code also say true? I was able to send the command successfully, but when I look at the page it remains false.

[Nutomic]

Sounds like you wrote it wrong. Best try it directly via psql.

[shellbot]

Hi, sorry to jump in here but hoping for some clarification from the user side.

1. Is there some way for me as a user to see which instances have this turned on already (ie which ones I can post to from Masto)?

2. Presumably this setting isn't enabled by default because there is some downside to enabling it. What are those downsides? I don't want to go around asking Lemmy admins to turn something on that's going to be overall harmful to their server but I also can't seem to find any info anywhere other than this thread.

Pointers much appreciated, just trying to wrap my head around the issue before trying to action anything. Thanks!

[Tealk]

2. What are those downsides?

this is what mastodon says
https://docs.joinmastodon.org/admin/config/#authorized_fetch

[shellbot]

Yeah federation from my Mastodon server to Lemmy broke when this was turned on on the Mastodon side. Been living without crosspost ability for three months now and it sucks!

But if I'm understanding this thread correctly, in order for federation to start working again it's the Lemmy admins that would need to enable a setting on their side? The local_site.federation_signed_fetch mentioned earlier?

This is the one I'm struggling to find info on, sorry if that was unclear.

[Nutomic]

Is there some way for me as a user to see which instances have this turned on already (ie which ones I can post to from Masto)?

Open /api/v3/site and check if site_view.local_site.federation_signed_fetch is true.

Presumably this setting isn't enabled by default because there is some downside to enabling it. What are those downsides? I don't want to go around asking Lemmy admins to turn something on that's going to be overall harmful to their server but I also can't seem to find any info anywhere other than this thread.

Its disabled mainly because its not required for Lemmy to work. Specific disadvantages are: every federation request gets signed now so it will use slightly more cpu (but I doubt it would be noticable in practice). Also remote servers know which instance is fetching which objects. This could be considered a privacy issue for single-user instances.

[Tealk]

Otherwise simply edit directly in the db with psql.

Does anyone happen to have the commands to hand? I work relatively rarely with docker

[Nutomic]

sudo docker-compose exec postgres psql -U lemmy

[Tealk]

here is a short guide for those who also want to change it:

2. Join Lemmy path:

   cd /opt/lemmy

   Replace /opt/lemmy with the path to your lemmy files

3. Connect to PostgreSQL:

   docker-compose exec postgres psql -U lemmy

   This command uses Docker Compose to execute the PostgreSQL command-line client (psql) with the specified username (-U lemmy). It connects to the PostgreSQL database server.

4. List Databases:

   \l

   After connecting to the PostgreSQL server, use \l to list all available databases.

5. Connect to a Database:

   \c lemmy

   Use \c to connect to a specific database, in this case, the "lemmy" database.

6. List Tables:

   \dt

   After connecting to the "lemmy" database, use \dt to list all tables in the current database.

7. Select Specific Column from a Table:

   SELECT federation_signed_fetch FROM local_site;

   Retrieve values only from the "federation_signed_fetch" column of the "local_site" table. Should currently be set to f

8. Update Table Rows:

   UPDATE local_site SET federation_signed_fetch = 't';

   Update the value of the "federation_signed_fetch" column to 't' for all rows in the "local_site" table. Now you can check it once, either with point 6 or with /api/v3/site

9. Quit PostgreSQL:

   \q

   Use \q to quit the PostgreSQL command-line interface.

[Tealk]

According to /api/v3/site "federation_signed_fetch" is now active but when I activate AUTHORIZED_FETCH on the mastodon instance it still comes up:

lemmy_1          | 2024-03-04T11:49:32.013797Z  WARN lemmy_server::root_span_builder: Unknown: Failed to parse object https://rollenspiel.social/users/Tealk with content {"error":"Request not signed"}: data did not match any variant of untagged enum PersonOrGroup
lemmy_1          |    0: lemmy_server::root_span_builder::HTTP request
lemmy_1          |            with http.method=POST http.scheme="http" http.host=rollenspiel.forum http.target=/inbox otel.kind="server" request_id=caf21bec-8446-400d-82f7-3fdcb6538c38
lemmy_1          |              at src/root_span_builder.rs:16

[Nutomic]

Are there any relevant logs on the Mastodon side?

[Tealk]

In the sidekiq_push log:

HTTP::TimeoutError: Read timed out after 10 seconds on https://hubzilla.eu/inbox

Mar 04 14:45:42 mastodon bundle[1263608]: 2024-03-04T13:45:42.741Z pid=1263608 tid=yd4m4 WARN: {"context":"Job raised exception","job":{"retry":16,"queue":"push","dead":false,"class":"ActivityPub::DeliveryWorker","args":["{\"@context\":[\"https://www.w3.org/ns/activitystreams\",{\"ostatus\":\"http://ostatus.org#\",\"atomUri\":\"ostatus:atomUri\",\"inReplyToAtomUri\":\"ostatus:inReplyToAtomUri\",\"conversation\":\"ostatus:conversation\",\"sensitive\":\"as:sensitive\",\"toot\":\"http://joinmastodon.org/ns#\",\"votersCount\":\"toot:votersCount\"}],\"id\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027/activity\",\"type\":\"Create\",\"actor\":\"https://rollenspiel.social/users/Tealk\",\"published\":\"2024-03-04T13:45:31Z\",\"to\":[\"https://www.w3.org/ns/activitystreams#Public\"],\"cc\":[\"https://rollenspiel.social/users/Tealk/followers\",\"https://rollenspiel.forum/u/Tealk\"],\"object\":{\"id\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027\",\"type\":\"Note\",\"summary\":null,\"inReplyTo\":\"https://rollenspiel.forum/post/114393\",\"published\":\"2024-03-04T13:45:31Z\",\"url\":\"https://rollenspiel.social/@Tealk/112037719685239027\",\"attributedTo\":\"https://rollenspiel.social/users/Tealk\",\"to\":[\"https://www.w3.org/ns/activitystreams#Public\"],\"cc\":[\"https://rollenspiel.social/users/Tealk/followers\",\"https://rollenspiel.forum/u/Tealk\"],\"sensitive\":false,\"atomUri\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027\",\"inReplyToAtomUri\":\"https://rollenspiel.forum/post/114393\",\"conversation\":\"tag:rollenspiel.social,2024-02-13:objectId=26368599:objectType=Conversation\",\"content\":\"<p><span class=\\\"h-card\\\" translate=\\\"no\\\"><a href=\\\"https://rollenspiel.forum/u/Tealk\\\" class=\\\"u-url mention\\\">@<span>Tealk@rollenspiel.forum</span></a></span> federation_signed_fetch=t</p>\",\"contentMap\":{\"de\":\"<p><span class=\\\"h-card\\\" translate=\\\"no\\\"><a href=\\\"https://rollenspiel.forum/u/Tealk\\\" class=\\\"u-url mention\\\">@<span>Tealk@rollenspiel.forum</span></a></span> federation_signed_fetch=t</p>\"},\"attachment\":[],\"tag\":[{\"type\":\"Mention\",\"href\":\"https://rollenspiel.forum/u/Tealk\",\"name\":\"@Tealk@rollenspiel.forum\"}],\"replies\":{\"id\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027/replies\",\"type\":\"Collection\",\"first\":{\"type\":\"CollectionPage\",\"next\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027/replies?only_other_accounts=true&page=true\",\"partOf\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027/replies\",\"items\":[]}}}}",1,"https://hubzilla.eu/inbox",{"synchronize_followers":false}],"jid":"9cfc4a45a414262a95d09d7d","created_at":1709559931.7934685,"enqueued_at":1709559931.7973483}}
Mar 04 14:46:07 mastodon bundle[1263608]: 2024-03-04T13:46:07.284Z pid=1263608 tid=ycsys WARN: {"context":"Job raised exception","job":{"retry":16,"queue":"push","dead":false,"class":"ActivityPub::DeliveryWorker","args":["{\"@context\":[\"https://www.w3.org/ns/activitystreams\",{\"ostatus\":\"http://ostatus.org#\",\"atomUri\":\"ostatus:atomUri\",\"inReplyToAtomUri\":\"ostatus:inReplyToAtomUri\",\"conversation\":\"ostatus:conversation\",\"sensitive\":\"as:sensitive\",\"toot\":\"http://joinmastodon.org/ns#\",\"votersCount\":\"toot:votersCount\"}],\"id\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027/activity\",\"type\":\"Create\",\"actor\":\"https://rollenspiel.social/users/Tealk\",\"published\":\"2024-03-04T13:45:31Z\",\"to\":[\"https://www.w3.org/ns/activitystreams#Public\"],\"cc\":[\"https://rollenspiel.social/users/Tealk/followers\",\"https://rollenspiel.forum/u/Tealk\"],\"object\":{\"id\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027\",\"type\":\"Note\",\"summary\":null,\"inReplyTo\":\"https://rollenspiel.forum/post/114393\",\"published\":\"2024-03-04T13:45:31Z\",\"url\":\"https://rollenspiel.social/@Tealk/112037719685239027\",\"attributedTo\":\"https://rollenspiel.social/users/Tealk\",\"to\":[\"https://www.w3.org/ns/activitystreams#Public\"],\"cc\":[\"https://rollenspiel.social/users/Tealk/followers\",\"https://rollenspiel.forum/u/Tealk\"],\"sensitive\":false,\"atomUri\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027\",\"inReplyToAtomUri\":\"https://rollenspiel.forum/post/114393\",\"conversation\":\"tag:rollenspiel.social,2024-02-13:objectId=26368599:objectType=Conversation\",\"content\":\"<p><span class=\\\"h-card\\\" translate=\\\"no\\\"><a href=\\\"https://rollenspiel.forum/u/Tealk\\\" class=\\\"u-url mention\\\">@<span>Tealk@rollenspiel.forum</span></a></span> federation_signed_fetch=t</p>\",\"contentMap\":{\"de\":\"<p><span class=\\\"h-card\\\" translate=\\\"no\\\"><a href=\\\"https://rollenspiel.forum/u/Tealk\\\" class=\\\"u-url mention\\\">@<span>Tealk@rollenspiel.forum</span></a></span> federation_signed_fetch=t</p>\"},\"attachment\":[],\"tag\":[{\"type\":\"Mention\",\"href\":\"https://rollenspiel.forum/u/Tealk\",\"name\":\"@Tealk@rollenspiel.forum\"}],\"replies\":{\"id\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027/replies\",\"type\":\"Collection\",\"first\":{\"type\":\"CollectionPage\",\"next\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027/replies?only_other_accounts=true&page=true\",\"partOf\":\"https://rollenspiel.social/users/Tealk/statuses/112037719685239027/replies\",\"items\":[]}}}}",1,"https://hubzilla.eu/inbox",{"synchronize_followers":false}],"jid":"9cfc4a45a414262a95d09d7d","created_at":1709559931.7934685,"enqueued_at":1709559967.2775068,"error_message":"Read timed out after 10 seconds on https://hubzilla.eu/inbox","error_class":"HTTP::TimeoutError","failed_at":1709559942.7408955,"retry_count":0}}

[Nutomic]

Those logs dont look relevant, and anyway what does Hubzilla have to do with it?

[Tealk]

I have simply provided all the logs I have found for this case. I don't run a hubzilla instance, so I can't answer that for you.

[corgana]

Just wanted to share that we're getting this error after restart when enabling it:

thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/activitypub_federation-0.5.1-beta.1/src/config.rs:180:14:
actor does not have a private key to sign with

[Tealk]

Ah after a restart I get the same error and the ui no longer connects

[Nutomic]

@corgana Thanks, I just noticed the same problem and fixed it in #4516. Not sure if anything else is needed, there is some additional discussion in superseriousbusiness/gotosocial#2697

[Tealk]

is the branch available as a docker build? then I would test it.

[Nutomic]

You can test with ds9.lemmy.ml where Ive deployed it. Anyway its not working yet because Mastodon has some weird requirements.

Edit: Got it working now.

[corgana]

Tested to ds9.lemmy.ml from a Mastodon instance with authorized_fetch enabled, and is working! 🎉

[Tealk]

Also tested, works: https://ds9.lemmy.ml/post/9603