New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

interpret Mastodon ActivityPub HTTP 202 response as failure #16

Closed
snarfed opened this Issue Oct 17, 2017 · 23 comments

Comments

3 participants
@snarfed
Owner

snarfed commented Oct 17, 2017

mastodon's AP code returns HTTP 202 to inbox POSTs that fail signature verification:

TODO:

@snarfed

This comment has been minimized.

Owner

snarfed commented Oct 17, 2017

@snarfed

This comment has been minimized.

Owner

snarfed commented Oct 18, 2017

aha, looks like mastodon changed this 202 to a 401 in head (2.0.0): tootsuite/mastodon@dfaa219#diff-b71581bf00ae84fbac757c5373533ebb

@snarfed snarfed closed this Oct 23, 2017

@snarfed

This comment has been minimized.

Owner

snarfed commented Jan 26, 2018

tentatively reopening. i'm seeing 202 from mastodon again on Create but the reply isn't showing up, e.g. https://snarfed.org/2018-01-26_chris-aldrich-%f0%9f%8d%8d-anarchivistchaos-social-edsudigipres-club-sn-mastodon to https://mastodon.social/@chrisaldrich/99418039176997784 .

https://github.com/tootsuite/mastodon/blob/master/app/controllers/activitypub/inboxes_controller.rb at master shows that mastodon evidently returns 202 for success, so probably something else is going on that prevents it from appearing.

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "object": {
    "inReplyTo": "https://mastodon.social/users/chrisaldrich/statuses/99418039176997784",
    "cc": [
      "https://www.w3.org/ns/activitystreams#Public",
      "https://mastodon.social/users/chrisaldrich",
      "https://mastodon.social/users/chrisaldrich/followers",
      "https://mastodon.technology/users/snarfed",
      "https://digipres.club/users/edsu",
      "https://chaos.social/users/anarchivist"
    ],
    "attributedTo": [
      {
        "preferredUsername": "snarfed.org",
        "name": "Ryan Barrett",
        "url": "https://snarfed.org/",
        "image": [
          {
            "url": "https://secure.gravatar.com/avatar/947b5f3f323da0ef785b6f02d9c265d6?s=96&d=blank&r=g",
            "type": "Image"
          }
        ],
        "type": "Person",
        "id": "https://fed.brid.gy/snarfed.org",
        "icon": [
          {
            "url": "https://secure.gravatar.com/avatar/947b5f3f323da0ef785b6f02d9c265d6?s=96&d=blank&r=g",
            "type": "Image"
          }
        ]
      }
    ],
    "tag": [
      {
        "name": "reply"
      }
    ],
    "id": "https://snarfed.org/2018-01-26_chris-aldrich-%f0%9f%8d%8d-anarchivistchaos-social-edsudigipres-club-sn-mastodon",
    "name": "reply \n\nJanuary 26, 2018 Ryan Barrett\nLeave a comment\n\n\nthanks chris! bridgy fed is definitely open and ready for (more) users. it\u2019s not perfect yet, but it is usable. i\u2019m using it to post this reply right now! \n\nStandard",
    "url": "https://snarfed.org/2018-01-26_chris-aldrich-%f0%9f%8d%8d-anarchivistchaos-social-edsudigipres-club-sn-mastodon",
    "content": " <p>thanks chris! bridgy fed is definitely open and ready for (more) users. it\u2019s not perfect yet, but it is usable. i\u2019m using it to post this reply right now! <a class=\"u-in-reply-to\" href=\"https://mastodon.social/@chrisaldrich/99418039176997784\"></a> <cite class=\"via\"><a href=\"https://fed.brid.gy/\"></a></cite></p> ",
    "published": "2018-01-26T12:50:25-08:00",
    "@context": "https://www.w3.org/ns/activitystreams",
    "type": "Note"
  },
  "type": "Create"
}

@snarfed snarfed reopened this Jan 26, 2018

@snarfed

This comment has been minimized.

Owner

snarfed commented Jan 27, 2018

i suspect this is due to settings on mastodon.social specifically (currently 2.2.0rc2), since replies and likes still work on lgbt.io (2.1.2), chaos.social (2.1.2), and others.

@swentel

This comment has been minimized.

Contributor

swentel commented May 25, 2018

Some feedback re: 202 responses.

Been playing around with integrating Drupal. I've done 3 tests, of which 2 work fine. Test url is https://mastodon.technology/@snarfed/3194674

What works is like (https://realize.be/like/1468) and repost (https://realize.be/repost/1469)
Interestingly enough, I got "202 response! If this is Mastodon 1.x, their signature verification probably failed. :(" back for both responses.

When trying to send a reply, I get back the same response, but nothing shows up on the post on mastodon itself.

(It might be that I'm missing something, currently, I have the .htaccess and atom feed, but nothing done yet with WebSub. But AFAICT, that shouldn't matter for sending a webmention.)

@myfreeweb

This comment has been minimized.

myfreeweb commented Jul 6, 2018

mastodon 2.4.2, replies still don't show up (log), and likes do still seem to work

@swentel

This comment has been minimized.

Contributor

swentel commented Sep 8, 2018

So out of curiosity, I tried a reply again today :) (to https://jawns.club/@timplunkett/100606109220664799)

The latest reply I tested now gives something back like 'Public key not found for key acct:realize.be@realize.be'. I haven't checked yet what that means and/or if it's relevant, but just wanted to leave it here.

@snarfed

This comment has been minimized.

Owner

snarfed commented Sep 8, 2018

@swentel thanks for the update! that looks like #29, but the conclusion there was that they needed a representative h-card (#29 (comment)), which isn't your problem. https://fed.brid.gy/.well-known/webfinger?resource=realize.be@realize.be works fine.

...ah, but your site isn't actually set up for bridgy fed. https://realize.be/.well-known/host-meta and https://realize.be/.well-known/webfinger currently 404. they should redirect to fed.brid.gy instead. https://fed.brid.gy/#setup

@swentel

This comment has been minimized.

Contributor

swentel commented Sep 8, 2018

Right - my bad, I commented out the lines in my .htaccess .. still the same outcome now.

@snarfed

This comment has been minimized.

Owner

snarfed commented Sep 8, 2018

@swentel

This comment has been minimized.

Contributor

swentel commented Sep 24, 2018

So, I checked mastodon, in app/controllers/activitypub/inboxes_controller.rb it will return 202 when the request is signed valid. But process_payload can still well basically do nothing. Kind of weird behavior if you tell me ..

@swentel

This comment has been minimized.

Contributor

swentel commented Sep 26, 2018

So I finally found out why replies don't show up. The problem is that mastodon compares the id on the object with the actor url.

See https://github.com/tootsuite/mastodon/blob/master/app/lib/activitypub/activity/create.rb#L8
See https://github.com/tootsuite/mastodon/blob/master/app/lib/activitypub/activity/create.rb#L269

Since fed.brid.gy sends the url of the post on the website as id, and the actor url is something like https://fed.brid.gy/{sitename} this will now fail. Also, the 'url' property in an object gets the same check, but is ignored when it fails, the reply will still show up if the id check is fine.

When sending an id which looks like 'https://fed.brid.gy/realize.be?unique_id_here', the reply will show up. This has an impact on the canonical URL of the post being fed.brid.gy and not the website url. As a consequence, I guess round trips can't work anymore. One workaround could be to create the id in the form of https://fed.brid.gy/realize.be?url={original_url} - that works (see the 'Welcome to the club! AGAIN' canonical url). Of course, if at some point mastodon decides to start ignore get params, then it's back to the drawing table.

(I guess this also has impact on discovery of posts as they basically will have the same problem)

I often wonder whether it would be possible to set the id of https://fed.brid.gy/realize.be' to realize.be, but I'm afraid that ain't possible at all ?

Working replies:
https://mastodon.social/users/swentel/statuses/100776214865184909
https://mastodon.technology/@snarfed/3194674

@swentel swentel referenced this issue Sep 26, 2018

Closed

My toot doesn't show up :( #8328

2 of 2 tasks complete
@swentel

This comment has been minimized.

Contributor

swentel commented Sep 26, 2018

this is the minimal object that I send:

stdClass Object
(
    [@context] => https://www.w3.org/ns/activitystreams
    [type] => Create
    [actor] => https://fed.brid.gy/realize.be
    [to] => Array
        (
            [0] => https://www.w3.org/ns/activitystreams#Public
            [1] => https://mastodon.social/users/swentel
        )

    [object] => stdClass Object
        (
            [id] => https://fed.brid.gy/realize.be?unique-id-needed-7
            [url] => https://realize.be/reply/content/1567
            [type] => Note
            [published] => 2018-09-26T17:27:21Z
            [attributedTo] => https://fed.brid.gy/realize.be
            [inReplyTo] => https://mastodon.social/users/swentel/statuses/100776214865184909
            [content] => <a href="https://mastodon.social/users/swentel">@swentel</a> A minimal object
            [to] => Array
                (
                    [0] => https://www.w3.org/ns/activitystreams#Public
                    [1] => https://mastodon.social/users/swentel
                )

        )

)

This is another one which also includes the 'mention' tag. This makes it so that a notification is also send to the user. The reply above did show up on the post, so it isn't that much of a deal.

So, I guess the when posting, bridgy should prepend @{username}.
Or, documentation could also say that we we find @{username} urls in the post, we add the mention tag. (this could be another issue of course, but wanted to document this too)

stdClass Object
(
    [@context] => https://www.w3.org/ns/activitystreams
    [type] => Create
    [actor] => https://fed.brid.gy/realize.be
    [to] => Array
        (
            [0] => https://www.w3.org/ns/activitystreams#Public
            [1] => https://mastodon.social/users/swentel
        )

    [object] => stdClass Object
        (
            [id] => https://fed.brid.gy/realize.be?unique-id-needed-6
            [url] => https://realize.be/reply/content/1567
            [type] => Note
            [published] => 2018-09-26T17:28:38Z
            [attributedTo] => https://fed.brid.gy/realize.be
            [inReplyTo] => https://mastodon.social/users/swentel/statuses/100776214865184909
            [content] => <a href="https://mastodon.social/users/swentel">@swentel</a> with url
            [to] => Array
                (
                    [0] => https://www.w3.org/ns/activitystreams#Public
                    [1] => https://mastodon.social/users/swentel
                )

            [tag] => Array
                (
                    [0] => stdClass Object
                        (
                            [type] => Mention
                            [href] => https://mastodon.social/users/swentel
                            [name] => @swentel
                        )

                )

        )

)
@snarfed

This comment has been minimized.

Owner

snarfed commented Sep 26, 2018

this is really really great progress. thank you so much @swentel!

@swentel

This comment has been minimized.

Contributor

swentel commented Sep 26, 2018

So I've been reading the comparison code. It compares the id of the object with @account.uri which is fetched here:
https://github.com/tootsuite/mastodon/blob/master/app/services/resolve_account_service.rb#L123
and then goes to
https://github.com/tootsuite/mastodon/blob/master/app/services/resolve_account_service.rb#L161

There's a lot going on here, it goes to fetch the atom feed, and then tries to find the author uri there, if I'm reading that right. Now, my atom feed uses granary, but it seems fine afaics.
When nothing is found, it either fills in nothing or falls back to author_uri.content - which I have no idea atm what this does. Haven't done much ruby, so clueless.

Anyway, I feel like there might be a way out here, but I might be naive. It's also late now, so might not be fresh anymore :)

@swentel

This comment has been minimized.

Contributor

swentel commented Sep 26, 2018

So when running that code on the body of my atom feed, it returns http://realize.be - out of ideas again :)

@snarfed

This comment has been minimized.

Owner

snarfed commented Oct 11, 2018

@swentel i think you're right that the best fix here is probably to wrap our translated post object ids under https://fed.brid.gy/... somewhere, so that they have the same prefix as the already wrapped actor ids.

(i doubt they'd drop query parameters when checking uniqueness, but if they do, we can always put the original post URL in the path instead of query, eg https://fed.brid.gy/realize.be/reply/content/1567.)

one key question: i only see one reply of yours on the original mastodon post you were testing against, https://mastodon.social/users/swentel/statuses/100776214865184909 . mastodon shows its URL as https://fed.brid.gy/realize.be#create . do you know if you put that in both id and url? or just id? i'm wondering which one mastodon uses. if url, then maybe we wouldn't have to actual redirect or serve from our wrapped id URLs at all.

@swentel

This comment has been minimized.

Contributor

swentel commented Oct 11, 2018

Right, I've deleted some replies. I added a new one. So mastodon either gets it from id or url, see https://github.com/tootsuite/mastodon/blob/master/app/lib/activitypub/activity/create.rb#L52

I've added another reply on https://mastodon.social/@swentel/100776214865184909 with "This is pretty cool!"

This is the payload from the request

stdClass Object
(
    [type] => Create
    [@context] => https://www.w3.org/ns/activitystreams
    [actor] => https://fed.brid.gy/realize.be
    [object] => Array
        (
            [type] => Note
            [to] => Array
                (
                    [0] => https://www.w3.org/ns/activitystreams#Public
                    [1] => https://mastodon.social/users/swentel/inbox
                )

            [id] => https://fed.brid.gy/realize.be?post=https://realize.be/notes/1573
            [inReplyTo] => https://mastodon.social/@swentel/100776214865184909
            [content] => This is pretty cool!
            [published] => 2018-10-11T07:08:01Z
        )

)
@swentel

This comment has been minimized.

Contributor

swentel commented Oct 11, 2018

So I've looked where the rewrite could happen, but I'm not 100% sure exactly what I'm doing :/
I've attached a diff I have locally, but not sure if the place I'm changing it makes sense .. :)

rewrite.txt

@snarfed

This comment has been minimized.

Owner

snarfed commented Oct 11, 2018

thanks!

i'm still curious, if we point id to fed.brid.gy but keep url pointing to the user's own site, whether mastodon will render links from url or id. i noticed you omitted url in #16 (comment) . mind trying again with url pointing somewhere other than id to see which one mastodon renders?

and your patch looks good! i'd probably put it in common.postprocess_as2() instead, and i'm inclined to put the full URL in the path like the existing/wm/... URL route, eg https://fed.brid.gy/redir/https://realize.be/ and https://fed.brid.gy/redir/https://realize.be/reply/content/1567. but you're definitely on the right path!

@swentel

This comment has been minimized.

Contributor

swentel commented Oct 11, 2018

Url pointing to the post will be discarded, see https://github.com/tootsuite/mastodon/blob/master/app/lib/activitypub/activity/create.rb#L243 - it uses the same validation function (which still annoys me, but oh well)

Just to be sure that I'm not fooling myself, send this payload:

stdClass Object
(
    [type] => Create
    [@context] => https://www.w3.org/ns/activitystreams
    [actor] => https://fed.brid.gy/realize.be
    [object] => Array
        (
            [type] => Note
            [to] => Array
                (
                    [0] => https://www.w3.org/ns/activitystreams#Public
                    [1] => https://mastodon.social/users/swentel/inbox
                )

            [id] => https://fed.brid.gy/realize.be?post=https://realize.be/notes/1550
            [url] => https://realize.be/notes/1550
            [inReplyTo] => https://mastodon.social/@swentel/100776214865184909
            [content] => Sending different host in url. Will be discarded normally.
            [published] => 2018-10-11T21:55:10Z
        )

)

The url has been ignored here. Payload underneath has bridgy host in the url, and then it takes that one

stdClass Object
(
    [type] => Create
    [@context] => https://www.w3.org/ns/activitystreams
    [actor] => https://fed.brid.gy/realize.be
    [object] => Array
        (
            [type] => Note
            [to] => Array
                (
                    [0] => https://www.w3.org/ns/activitystreams#Public
                    [1] => https://mastodon.social/users/swentel/inbox
                )

            [id] => https://fed.brid.gy/realize.be?post=https://realize.be/notes/1530
            [url] => https://fed.brid.gy/realize.be?post=https://realize.be/notes/1545
            [inReplyTo] => https://mastodon.social/@swentel/100776214865184909
            [content] => Sending url with host same as actor, should take one now
            [published] => 2018-10-11T21:57:03Z
        )

)

You can see them both on https://mastodon.social/@swentel/100776214865184909

snarfed added a commit that referenced this issue Oct 14, 2018

add /r/ redirect endpoint
for #16, #32. cc @swentel. more to come.

snarfed added a commit that referenced this issue Oct 14, 2018

@swentel

This comment has been minimized.

Contributor

swentel commented Oct 14, 2018

Tested, awesome, awesome :)

I guess we can close this one!

Or maybe remove the log message in bridgy, we know that 202 is ok it terms of signature.

snarfed added a commit that referenced this issue Oct 17, 2018

@snarfed

This comment has been minimized.

Owner

snarfed commented Oct 17, 2018

agreed! i've removed the log message. thanks for the nudge.

@snarfed snarfed closed this Oct 17, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment