Returning 400: Target post https://mastodon.technology/@snarfed/3194674 has no Atom link

[sparxsys]

Hello,

I am trying to setup this on my Drupal site using Indieweb module.

Source: https://www.ravisagar.in/reply/trying-send-webmention-again-mastadon
Target: https://mastodon.social/@swentel/100776214865184909

Error:

I 2020-09-02 06:45:13.381355 Response source target: https://www.ravisagar.in/reply/trying-send-webmention-again-mastadon https://mastodon.social/@swentel/100776214865184909
I 2020-09-02 06:45:13.383153 Returning 400: Target post https://mastodon.social/@swentel/100776214865184909 has no Atom link
E 2020-09-02 06:45:13.454242 Error 400, response body: '400 Bad Request\n\nThe server could not comply with the request since it is either malformed or otherwise incorrect.\n\n Target post https://mastodon.social/@swentel/100776214865184909 has no Atom link '

I have done the setup as mentioned on this page: https://fed.brid.gy/. I believe my atom feed is fine. Also I created .well-known folder and added these lines in the .htaccess

RewriteEngine on
RewriteBase /
RewriteRule ^.well-known/(host-meta|webfinger).* https://fed.brid.gy/$0 [redirect=302,last]

Not sure what I am missing here so raising an issue. I might have missed some configuration but my site is sending the webmention which I can also see in the logs: https://fed.brid.gy/responses

Thanks for the great application :)

[snarfed]

hmm! sorry for the trouble. not sure what's going on here, but i'll take a look soon.

[snarfed]

to add a bit of context here, based on the log, Bridgy Fed first tried to reply via ActivityPub, which is the protocol that Mastodon supports. it sent a valid AP request, but Mastodon returned an error. Bridgy Fed then tried to fall back to OStatus, which is why it complained that https://mastodon.social/@swentel/100776214865184909 (not your own post) "has no Atom link."

the real question is, why did mastodon.social fail a (seemingly) perfectly good AP request? i'll keep looking.

[sparxsys]

Thanks for looking into this :)

[nekr0z]

Seems to have happened to me, too:
https://fed.brid.gy/log?key=https%3A%2F%2Fevgenykuznetsov.org%2Fen%2Freactions%2F2020%2Fface-masks%2F+https%3A%2F%2Ffed.fab.industries%2F%40fabsh%2F105069505860853845&start_time=1603300816

[snarfed]

argh, sorry for the trouble, and thanks for the nudge! i'll look at this again.

[nekr0z]

Thing is, I have successfully used Bridgy Fed for quite a while to interact with that very Mastodon instance. Either they pushed some Mastodon update that broke things, or it's some corner case I've just hit.

[snarfed]

@nekr0z interesting. when bridgy fed sends fed.fab.industries this AP Create request for your reply:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Create",
  "object": {
    "published": "2020-10-21T20:13:17+0300",
    "content": "<p>While I do totally agree with you...",
    "url": "https://fed.brid.gy/r/https://evgenykuznetsov.org/en/reactions/2020/face-masks/",
    "inReplyTo": "https://fed.fab.industries/users/fabsh/statuses/105069505860853845",
    "@context": "https://www.w3.org/ns/activitystreams",
    "type": "Note",
    "attributedTo": [
      {
        "url": "https://fed.brid.gy/r/https://evgenykuznetsov.org",
        "type": "Person",
        "name": "Evgeny Kuznetsov",
        "image": [
          {
            "url": "https://evgenykuznetsov.org/img/avatar.jpg",
            "type": "Image",
            "name": "Evgeny Kuznetsov"
          }
        ],
        "icon": [
          {
            "url": "https://evgenykuznetsov.org/img/avatar.jpg",
            "type": "Image",
            "name": "Evgeny Kuznetsov"
          }
        ],
        "preferredUsername": "evgenykuznetsov.org",
        "id": "https://fed.brid.gy/evgenykuznetsov.org"
      }
    ],
    "tag": [
      {
        "type": "Mention",
        "href": "https://fed.fab.industries/users/fabsh"
      }
    ],
    "id": "https://fed.brid.gy/r/https://evgenykuznetsov.org/en/reactions/2020/face-masks/",
    "cc": [
      "https://www.w3.org/ns/activitystreams#Public",
      "https://fed.fab.industries/users/fabsh",
      "https://fed.fab.industries/users/fabsh/followers"
    ]
  }
}

fed.fab.industries responds with:

HTTP 401
Client Error: Unauthorized for url: https://fed.fab.industries/users/fabsh/inbox

not sure why yet.

[snarfed]

maybe fed.fab.industries was recently configured to block replies somehow? cc @fabsh...? or some other update, like you mentioned?

[snarfed]

401 implies that the HTTP Signature header is bad or isn't getting accepted. i haven't changed the way bridgy fed generates that recently, but evidently mastodon 3.2.1 - which fed.fab.industries runs - evidently changed their handling of it. from https://github.com/tootsuite/mastodon/releases/tag/v3.2.1 :

Add support for latest HTTP Signatures spec draft
...
Fix dereferencing remote statuses not using the correct account for signature when receiving a targeted inbox delivery

guess i need to dig in and understand what they changed.

[snarfed]

looks like this is the culprit: mastodon/mastodon#15016

[snarfed]

ok, i'm now including and signing the Host and Digest HTTP headers, as required by mastodon 3.2.1 (see mastodon/mastodon#14556 (comment)), but i'm still getting a generic 401 Client Error: Unauthorized for url: ... error back. i don't think it's due to the new req'ts, since while i was adding them and testing, i got specific error messages for them, eg missing required header "digest". the generic error message makes me think something else is wrong. 🤷

[nekr0z]

As I said before, Bridgy Fed did successfully pass my replies to that very Mastodon instance (this one, for example, just 4 days before things broke). And @fabsh does seem to be running 3.2.1 now (which was released in the meantime), so whatever the issue is, it is very much likely related to that release. Not sure whether it's the same thing @sparxsys originally reported — there was no 3.2.1 back then.

[ClearlyClaire]

@snarfed is there a way I can help with debugging that? The code in Mastodon 3.2.1 is indeed stricter than it used to be, but as you said, it should return specific error messages when issues occur.

[snarfed]

@ThibG thanks for offering! you're right, the specific error messages were definitely helpful. i worked through those, but i'm now getting the generic 401 Client Error: Unauthorized for url: ... error, for a Create request posted to a user's inbox that succeeded before 3.2.1. full JSON activity example here.

i expect this means Mastodon couldn't verify the HTTP signature. i'm using the httpsig Python package to generate it, which was working before. any tips on debugging generic 401 errors from Mastodon like this? (apart from installing and testing Mastodon locally, which I'm not quite ready to commit to. 😁)

[ClearlyClaire]

401 Client Error: Unauthorized for url: ... doesn't sound like something Mastodon itself would return. I'm surprised you wouldn't have a body with some more details.

You could try sending to https://social.sitedethib.com/users/Thib/inbox, I added some instrumentation on my end.

[snarfed]

thanks for the nudge @ThibG! you're absolutely right, there was indeed a body with more detail. i thought i was logging it, but evidently not.

Verification failed for evgenykuznetsov.org@evgenykuznetsov.org https://fed.brid.gy/evgenykuznetsov.org using rsa-sha256 (RSASSA-PKCS1-v1_5 with SHA-256)

that confirms my hunch that this is a signature verification failure. still a bit confusing, since these signatures were succeeding with Mastodon before 3.2.1, but 🤷. i'll look at a few more things, and i may take you up on your offer of testing against social.sitedethib.com with its instrumentation. thanks again for helping!

[ClearlyClaire]

Nothing should have changed regarding how the signatures are verified/computed. My hunch is that maybe, in signing more headers to please Mastodon's new requirements, you inadvertently added some header twice (e.g., Host), signing only one copy, but having it twice in the request?

[snarfed]

Good hunch, thanks! Didn't pan out, but gave me some other ideas. I'll keep looking.

[swentel]

Hah, I got hit by this one as well (mastodon/mastodon#14556 (comment) I mean). I haven't got to the digest signing part yet, but I'll report back the changes in my Drupal AP module if I can get it working.

[swentel]

@snarfed I got it working again for mastodon 3.2.1 adding the digest header, this is the commit in case it you're interested: https://git.drupalcode.org/project/activitypub/commit/0bdd562

[snarfed]

thanks @swentel! yeah i did the same thing a bit earlier here, and finally committed and pushed it just now, d22903c, but still no luck yet.

[ClearlyClaire]

@snarfed I guess the body as serialized by requests might be different from what you actually sign? That's just a hunch, I have not tested your code. The offer to try on social.sitedethib.com/inbox is still up, and I've added a bit specific to the Digest header for debugging

[snarfed]

@ThibG thanks again for the help! i definitely appreciate the instrumenting you set up on social.sitedethib.com, i may try it soon. i'll let you know.

[snarfed]

ok @ThibG! i sent an AP activity to https://social.sitedethib.com/users/Thib/inbox a couple times just now, at 20:37:42 and 20:47:41 UTC. it had Date, Digest, and Host headers, and a Signature header that included them. full activity is below. let me know if you see anything obvious in your instrumentation. thanks again!

{
  "published": "2020-10-31T13:37:22-07:00",
  "content": "likes <a class=\"u-like u-like-of\" href=\"https://social.sitedethib.com/@Thib/101878161113758535\">Claire: \u201cMy girlfriends are the most wonderful people ever\u2026\u201d \u2013 Mastodon (instance perso)</a>\n<cite class=\"via\"><a href=\"https://fed.brid.gy/\">via Bridgy Fed</a></cite>",
  "url": "https://fed.brid.gy/r/https://snarfed.org/2020-10-31_claire-my-girlfriends-are-the-most-wonderful-people-ever-mastodon-instance-perso",
  "actor": {
    "url": "https://fed.brid.gy/r/https://snarfed.org/",
    "type": "Person",
    "name": "Ryan Barrett",
    "image": [
      {
        "url": "https://secure.gravatar.com/avatar/947b5f3f323da0ef785b6f02d9c265d6?s=96&d=blank&r=g",
        "type": "Image"
      }
    ],
    "icon": [
      {
        "url": "https://secure.gravatar.com/avatar/947b5f3f323da0ef785b6f02d9c265d6?s=96&d=blank&r=g",
        "type": "Image"
      }
    ],
    "preferredUsername": "snarfed.org",
    "id": "https://fed.brid.gy/snarfed.org"
  },
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Like",
  "object": "https://social.sitedethib.com/users/Thib/statuses/101878161113758535",
  "id": "https://fed.brid.gy/r/https://snarfed.org/2020-10-31_claire-my-girlfriends-are-the-most-wonderful-people-ever-mastodon-instance-perso",
  "cc": [
    "https://www.w3.org/ns/activitystreams#Public",
    "https://social.sitedethib.com/users/Thib",
    "https://social.sitedethib.com/users/Thib/followers"
  ]
}

[ClearlyClaire]

Hmm, my instrumentation was lacking some info to debug that more easily, but it's definitely an issue with the Digest not matching, and I'm pretty sure it's because json_dumps and requests' json parameter do not actually result in exactly the same body. I'd suggest passing the result of json_dumps to requests.

[snarfed]

thanks! you did mention that before, i should have tried it earlier. good idea, but no luck. tried just now, 22:24:12 UTC, same result. code in 199d4d6.

[ClearlyClaire]

@snarfed ok found it, in your Digest header, you use “SHA256” while the expected algorithm name is “SHA-256”

[snarfed]

hah, wow, great catch! single character typo bugs like that are the worst. thank you so much for your help! got a 202 from social.sitedethib.com this time, looks like that may have done the trick. 🎉

[nekr0z]

I can confirm it's working for me now. Thank you!

[fabsh]

Hi! Just saw this now. Sorry I wasn't able to help, but glad to see you got it fixed. 👍