[Bug]: outbound federation of comments not working on personal instance #3380
Comments
This comment was marked as abuse.
This comment was marked as abuse.
|
Last week, everything worked. I can't tell exactly when it stopped but at least 4 days ago. I've restarted the container a few times but as far as I can remember, I haven't changed anything about the configuration in that timeframe. Edit: thinking back, it might have coincided with the upgrade from 0.17.3 to 0.17.4 but I can't be sure. |
This comment was marked as abuse.
This comment was marked as abuse.
|
I approved your account |
This comment was marked as abuse.
This comment was marked as abuse.
|
So my instance is sending malformed requests, possibly with a single incorrect parameter and we have no logging that can tell us what exactly those requests contain? |
This comment was marked as abuse.
This comment was marked as abuse.
|
I'd rather not reset my instance and instead focus on finding a way to debug this so we can hopefully find a solution for everyone else who might have the same problem in the future. I'm pretty sure I haven't changed anything about my configuration that could have influenced outgoing requests. All networking is the same as before, including hostnames. My best guess is that something broke during a restart and now has to be fixed in the database. Unfortunately, I'm not very familiar with rust so I have no clue how I could debug this. If anyone has a good idea, I could easily spin up a second instance so I have something to test against without spamming real communities. |
This comment was marked as abuse.
This comment was marked as abuse.
|
Well, guess I'll have to learn Rust unless someone else can tell me how to log the full request or at least more details about what causes the http 400. |
|
i was going to try the "mirror" method listed here: https://stackoverflow.com/a/67127911 to try and debug #3354. maybe that will help here too? |
|
Alright, I built a small setup that puts mitmproxy between nginx and the backend. Here's a subscribe request: Request headers: POST /inbox HTTP/1.0
X-Real-IP: 10.0.2.25
Host: lemmy:8536
X-Forwarded-For: 10.0.0.2, 10.0.2.25
Connection: close
Content-Length: 914
User-Agent: Lemmy/0.18.0; +https://lemmy.helios42.de
Accept: */*
Content-Type: application/activity+json
Date: Thu, 29 Jun 2023 13:07:34 GMT
Digest: SHA-256=i7WvKDyIbBTs6R7GoilDS9u1recJrX13oVqQnpKoGgQ=
Signature: keyId="https://lemmy.helios42.de/u/dfyx#main-key",algorithm="hs2019",headers="(request-target) content-type date digest host",signature="SCiYhZLbjw1Y5uwLpt8xIgWCz6LErBWJBU5/c5ubg0KxL9Yr5vwYgxE6kzpT3xNxOFbU53NJLibRP6S2DflweMZFNGGAIE310Vi1B4nhWLB04Qf3S8jLYR8OFgOoTRvmx9MuQC6FBcim8f3Af5VJV3fOVv03BDTZEbUdy8gHFFsalyu1Rj81O0fD2QnPi5NxV1iSVPoAcOjNaJqZI/AJ1Re3m51CSR4n1/MzMzb6zFu81B6HuJtcFAV9Mo+vGCCpNZmQkcxkDJoNgc/oWwDLOXo/yiennokysTDWQg77e5rYm4mJxaYRU4o0tPeUqfr6tmSsQoEUpV9+gktwG8m6Gg=="
X-Forwarded-Host: lemmy-debug.helios42.de
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 7ee5f7197c73
Accept-Encoding: gzipRequest body: {
"@context": [
"https://www.w3.org/ns/activitystreams",
"https://w3id.org/security/v1",
{
"ChatMessage": "litepub:ChatMessage",
"commentsEnabled": "pt:commentsEnabled",
"distinguished": "lemmy:distinguished",
"expires": "as:endTime",
"identifier": "sc:identifier",
"language": "sc:inLanguage",
"lemmy": "https://join-lemmy.org/ns#",
"litepub": "http://litepub.social/ns#",
"matrixUserId": "lemmy:matrixUserId",
"moderators": {
"@id": "lemmy:moderators",
"@type": "@id"
},
"postingRestrictedToMods": "lemmy:postingRestrictedToMods",
"pt": "https://joinpeertube.org/ns#",
"removeData": "lemmy:removeData",
"sc": "http://schema.org/",
"sensitive": "as:sensitive",
"stickied": "lemmy:stickied"
}
],
"actor": "https://lemmy.helios42.de/u/dfyx",
"id": "https://lemmy.helios42.de/activities/follow/b3b76cc3-5394-4265-a667-789123ad2b27",
"object": "https://lemmy-debug.helios42.de/c/testcommunity",
"to": [
"https://lemmy-debug.helios42.de/c/testcommunity"
],
"type": "Follow"
}Response Debug instance logs: (might be from a different identical request, I tried multiple times) Essentially the same for posting a comment, except of course for the different request body. Edit: updated for readability |
This comment was marked as abuse.
This comment was marked as abuse.
|
Okay, I'm a step closer. The problem isn't in the requests my instance sends but that the receiving instance sends its own requests back (probably to get public keys and further info) Seems like my main instance generally replies with HTML when it should reply with JSON.
Edit: a force restart of the nginx container has finally solved the problem. Seems like I've had a broken config nginx.conf at some point and even after I changed it, |
|
Struggling seemingly with the same issue, but am just a user; don't know how to troubleshoot the cause. @dfyx Could you please tell what was wrong with the nginx.conf ? |
|
Definitely not the same issue. Beehaw replies correctly to |
Requirements
Summary
(Second attempt, GitHub suddenly threw a 404 the first time I submitted this)
For the last few days, none of the comments I made on my personal instance have federated to other instances, effectively shadowbanning me from the fediverse. For example one of my latest comments:
This seems to have happened to all my comments from the last few days, so it's probably neither a problem specifically with 0.18 (which I installed yesterday) nor a simple delay.
Steps to Reproduce
Technical Details
Installed as a docker stack on an Ubuntu 20.04 LTS host. Slightly modified the docker-compose.yml to accomodate this setup:
Full logs of restarting the instance and creating one new comment (not the one in the example above):
Version
both BE 0.17.4 and BE 0.18
Lemmy Instance URL
lemmy.helios42.de
The text was updated successfully, but these errors were encountered: