Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send out ActivityPub mentions (T627) #195

Merged
merged 23 commits into from Feb 8, 2020
Merged

Send out ActivityPub mentions (T627) #195

merged 23 commits into from Feb 8, 2020

Conversation

@qwazix
Copy link
Collaborator

@qwazix qwazix commented Oct 9, 2019

This pull request adds code that scans the text of each post to be published for handles of the type @username@domain.tld, uses webfinger to ask ActivityPub endpoints about them, gets their inbox and actor IRI's and adds mention tags to the post. Finally it adds the users to the CC list of the recipients of the activity.

This works in pleroma but not in mastodon unless we set the object type to "Note" instead of "Article"

T627

be Note to work but I will open an issue for them and see what their
reaction will be.
@thebaer

This comment has been minimized.

Copy link
Member

@thebaer thebaer commented on posts.go in dccfae7 Oct 8, 2019

Note that this doesn't fully match domains like writing.exchange. I think we can use word boundary character (I believe it's \b?) instead of doing the {2,6} match at the end.

This comment has been minimized.

Copy link
Collaborator

@qwazix qwazix replied Oct 9, 2019

Done

@thebaer

This comment has been minimized.

Copy link
Member

@thebaer thebaer commented on webfinger.go in dccfae7 Oct 8, 2019

We should return here, otherwise the application crashes on line 100 with a nil resp when the request fails.

This comment has been minimized.

Copy link
Collaborator

@qwazix qwazix replied Oct 9, 2019

Done

@thebaer

This comment has been minimized.

Copy link
Member

@thebaer thebaer commented on posts.go in dccfae7 Oct 8, 2019

Looks like the cc field was wrong when posting to the inbox of an instance where I didn't have any followers -- it retained the list of users from the previous instance:

Oct 08 15:33:03 mangrove matt[26396]: {"@context":["https://www.w3.org/ns/activitystreams"],"type":"Create","id":"https://matt.writefreely.dev/api/posts/dluynq3dsb","actor":"https://matt.writefreely.dev/api/collections/devlog","published":"2019-10-08T15:32:57Z","to":["https://www.w3.org/ns/activitystreams#Public"],"cc":["https://pawoo.net/users/AriaLAnna8","https://pawoo.net/users/Naomii55","https://pawoo.net/users/Haanakko88"],"object":{"type":"Article","id":"https://matt.writefreely.dev/api/posts/dluynq3dsb","published":"2019-10-08T15:32:57Z","inReplyTo":null,"url":"https://matt.writefreely.dev/activitypub-mentions-part-deux","attributedTo":"https://matt.writefreely.dev/api/collections/devlog","to":["https://www.w3.org/ns/activitystreams#Public"],"cc":["https://pawoo.net/users/AriaLAnna8","https://pawoo.net/users/Naomii55","https://pawoo.net/users/Haanakko88"],"name":"ActivityPub mentions, part deux","content":"\u003cp\u003eWe\u0026#39;re working on supporting \u003ca href=\"https://matt.writefreely.dev/tag:ActivityPub\" class=\"hashtag\" rel=\"nofollow\"\u003e\u003cspan\u003e#\u003c/span\u003e\u003cspan class=\"p-category\"\u003eActivityPub\u003c/span\u003e\u003c/a\u003e mentions in \u003ca href=\"https://matt.writefreely.dev/tag:WriteFreely\" class=\"hashtag\" rel=\"nofollow\"\u003e\u003cspan\u003e#\u003c/span\u003e\u003cspan class=\"p-category\"\u003eWriteFreely\u003c/span\u003e\u003c/a\u003e now (\u003ca href=\"https://writefreely.org/tasks/627\" rel=\"nofollow\"\u003eT627\u003c/a\u003e). The work is in progress, and last post didn\u0026#39;t work.\u003c/p\u003e\n\n\u003cp\u003eTesting to see if @matt@pleroma.site gets this.\u003c/p\u003e\n\n\u003cp\u003eLet\u0026#39;s see.\u003c/p\u003e\n","contentMap":{"en":"\u003cp\u003eWe\u0026#39;re working on supporting \u003ca href=\"https://matt.writefreely.dev/tag:ActivityPub\" class=\"hashtag\" rel=\"nofollow\"\u003e\u003cspan\u003e#\u003c/span\u003e\u003cspan class=\"p-category\"\u003eActivityPub\u003c/span\u003e\u003c/a\u003e mentions in \u003ca href=\"https://matt.writefreely.dev/tag:WriteFreely\" class=\"hashtag\" rel=\"nofollow\"\u003e\u003cspan\u003e#\u003c/span\u003e\u003cspan class=\"p-category\"\u003eWriteFreely\u003c/span\u003e\u003c/a\u003e now (\u003ca href=\"https://writefreely.org/tasks/627\" rel=\"nofollow\"\u003eT627\u003c/a\u003e). The work is in progress, and last post didn\u0026#39;t work.\u003c/p\u003e\n\n\u003cp\u003eTesting to see if @matt@pleroma.site gets this.\u003c/p\u003e\n\n\u003cp\u003eLet\u0026#39;s see.\u003c/p\u003e\n"},"tag":[{"type":"Hashtag","href":"https://matt.writefreely.dev/tag:ActivityPub","name":"#ActivityPub"},{"type":"Hashtag","href":"https://matt.writefreely.dev/tag:WriteFreely","name":"#WriteFreely"},{"type":"Mention","href":"https://pleroma.site/users/matt","name":"@matt@pleroma.site"}]}}
Oct 08 15:33:04 mangrove matt[26396]: 2019/10/08 15:33:04 "GET /api/collections/devlog" 200 2.274673ms "Pleroma 1.0.0-1168-ge18c7866-pleroma-dot-site; https://pleroma.site <info@pleroma.site>"
Oct 08 15:33:04 mangrove matt[26396]: 2019/10/08 15:33:04 Status  : 500 Internal Server Error
Oct 08 15:33:04 mangrove matt[26396]: 2019/10/08 15:33:04 Response: "error"
@thebaer
Copy link
Member

@thebaer thebaer commented Oct 9, 2019

Latest changes look good 👍 Confirmed that mentions work with Pleroma.

Michael Demetriou added 4 commits Oct 10, 2019
to the last because mastodon doesn't like

https://my.instance/@me but https://my.instance/users/me
intermediate page (WIP) that shows the user profile page url
there it updates the table to include the handle.

migrations WIP
Copy link
Member

@thebaer thebaer left a comment

Just a few changes to make. Also getting this error during compilation:

./database.go:2476:61: remoteActor.GetSharedInbox undefined (type activityserve.RemoteActor has no field or method GetSharedInbox)
./database.go:2477:157: remoteActor.GetSharedInbox undefined (type activityserve.RemoteActor has no field or method GetSharedInbox)
routes.go Outdated Show resolved Hide resolved
sqlite.sql Outdated Show resolved Hide resolved
schema.sql Outdated Show resolved Hide resolved
collections.go Outdated Show resolved Hide resolved
activitypub.go Outdated Show resolved Hide resolved
@qwazix
Copy link
Collaborator Author

@qwazix qwazix commented Oct 10, 2019

Ack to all, will do.

@thebaer
Copy link
Member

@thebaer thebaer commented Oct 10, 2019

Also, just saw the comments on the Mastodon issue. Is that fixed on our end?

@qwazix
Copy link
Collaborator Author

@qwazix qwazix commented Oct 10, 2019

Yes it works as expected, Gargron was right, I was thrown off because the first alias that webfinger returned was the @ version.

@thebaer thebaer added this to the 0.11 milestone Oct 10, 2019
@thebaer
Copy link
Member

@thebaer thebaer commented Oct 10, 2019

Awesome -- works great with Mastodon 👍


Did some more testing and crashed the app with this post. Logs:

panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 7763 [running]:
github.com/writeas/activityserve.NewRemoteActor(0xc0009a1860, 0x22, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/mattbaer/projects/go/src/github.com/writeas/activityserve/remoteActor.go:34 +0x41e
github.com/writeas/writefreely.(*datastore).getProfilePageFromHandle(0xc000687f00, 0xc000021ef0, 0xc000e20dcf, 0x15, 0xc000e6ee60, 0x2, 0xa, 0xc0009a0570)
	/home/mattbaer/projects/go/src/github.com/writeas/writefreely/database.go:2472 +0x11a
github.com/writeas/writefreely.(*PublicPost).ActivityObject(0xc00059e540, 0xc000021ef0, 0x1)
	/home/mattbaer/projects/go/src/github.com/writeas/writefreely/posts.go:1120 +0xc67
github.com/writeas/writefreely.federatePost(0xc000021ef0, 0xc00059e540, 0x65, 0x2cd9a00, 0x0, 0x0)
	/home/mattbaer/projects/go/src/github.com/writeas/writefreely/activitypub.go:574 +0xf6
created by github.com/writeas/writefreely.newPost
	/home/mattbaer/projects/go/src/github.com/writeas/writefreely/posts.go:593 +0x792
@qwazix
Copy link
Collaborator Author

@qwazix qwazix commented Oct 11, 2019

The crash was due to prismo and pixelfed not having sharedInboxes. I made GetSharedInbox() to return the inbox of the user so implementation won't need to care whether there is a sharedInbox or not (this is in activityserve)

webfinger.go Outdated Show resolved Hide resolved
@thebaer thebaer modified the milestones: 0.11, 0.12 Nov 8, 2019
thebaer added 2 commits Nov 26, 2019
This fixes the `undefined: strings.ReplaceAll` build error.
@thebaer
Copy link
Member

@thebaer thebaer commented Nov 26, 2019

Got everything up to date with develop; this is almost ready to merge! Last pieces I'm thinking about:

  • Maybe using a different path for remote users. Would just like something short and clean.
  • Maybe changing the style of mention links. Initial thoughts are to make the domain part of a user's handle faded out a little (but not completely hidden like on Mastodon). I'm also open to any ideas anyone has here
@thebaer
Copy link
Member

@thebaer thebaer commented Nov 26, 2019

Ah, one more crash caused by this post:

panic: interface conversion: interface {} is nil, not string
goroutine 1153 [running]:
github.com/writeas/activityserve.NewRemoteActor(0xc000d83e00, 0x33, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/matt/go/pkg/mod/github.com/writeas/activityserve@v0.0.0-20191011072627-3a81f7784d5b/remoteActor.go:37 +0x444
github.com/writeas/writefreely.(*datastore).getProfilePageFromHandle(0xc0006acb40, 0xc000085f40, 0xc00139a44f, 0x1c, 0xc000aaec80, 0x2, 0xa, 0xc000d82bc0)
        /home/matt/go/src/github.com/writeas/writefreely/database.go:2506 +0x11a
github.com/writeas/writefreely.(*PublicPost).ActivityObject(0xc000512240, 0xc000085f40, 0x1)
        /home/matt/go/src/github.com/writeas/writefreely/posts.go:1177 +0xc6c
github.com/writeas/writefreely.federatePost(0xc000085f40, 0xc000512240, 0x1, 0x29fbd00, 0x0, 0x0)
        /home/matt/go/src/github.com/writeas/writefreely/activitypub.go:614 +0xf6
created by github.com/writeas/writefreely.newPost
        /home/matt/go/src/github.com/writeas/writefreely/posts.go:612 +0x81f

And the mention links in that post cause a 500 and panic:

ERROR: 2019/11/26 19:58:17 handle.go:469: User: devlog
interface conversion: interface {} is nil, not string: goroutine 204 [running]:
runtime/debug.Stack(0xc000085f40, 0xc000579100, 0xc0004dbd80)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/writeas/writefreely.(*Handler).Web.func1.1.1(0xc0006a6d20, 0xc000579100, 0x1936e40, 0xc0000e01c0, 0xc000515960, 0xbf6f7ed657a0817f, 0x18d536fd42, 0x29fbda0)
        /home/matt/go/src/github.com/writeas/writefreely/handle.go:469 +0x16f
panic(0x161a700, 0xc000628ba0)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/writeas/activityserve.NewRemoteActor(0xc0005a2db0, 0x25, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/matt/go/pkg/mod/github.com/writeas/activityserve@v0.0.0-20191011072627-3a81f7784d5b/remoteActor.go:37 +0x444
github.com/writeas/writefreely.(*datastore).getProfilePageFromHandle(0xc0006a6b40, 0xc000085f40, 0xc00048006d, 0xe, 0xc000443048, 0x403faa, 0x15d7020, 0x191bbb0)
        /home/matt/go/src/github.com/writeas/writefreely/database.go:2506 +0x11a
github.com/writeas/writefreely.handleViewMention(0xc000085f40, 0x1936e40, 0xc0000e01c0, 0xc000579100, 0xc000085180, 0x0)
        /home/matt/go/src/github.com/writeas/writefreely/collections.go:847 +0x9f
github.com/writeas/writefreely.(*Handler).Web.func1.1(0xc0006a6d20, 0xc000579100, 0x1936e40, 0xc0000e01c0, 0x17f67d8, 0x17f6930, 0x0, 0x0)
        /home/matt/go/src/github.com/writeas/writefreely/handle.go:501 +0x166
github.com/writeas/writefreely.(*Handler).Web.func1(0x1936e40, 0xc0000e01c0, 0xc000579100)
        /home/matt/go/src/github.com/writeas/writefreely/handle.go:515 +0x63
net/http.HandlerFunc.ServeHTTP(0xc0006a7180, 0x1936e40, 0xc0000e01c0, 0xc000579100)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0000da240, 0x1936e40, 0xc0000e01c0, 0xc000395e00)
        /home/matt/go/pkg/mod/github.com/gorilla/mux@v1.7.0/mux.go:212 +0xe3
net/http.serverHandler.ServeHTTP(0xc0006c2a90, 0x1936e40, 0xc0000e01c0, 0xc000395e00)
        /usr/local/go/src/net/http/server.go:2774 +0xa8
net/http.(*conn).serve(0xc000706a00, 0x1938a00, 0xc000625240)
        /usr/local/go/src/net/http/server.go:1878 +0x851
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2884 +0x2f4
2019/11/26 19:58:17 Web deferred internal error render
2019/11/26 19:58:17 "GET /mention:@matt@write.as" 500 324.540684ms "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
@qwazix
Copy link
Collaborator Author

@qwazix qwazix commented Nov 27, 2019

This is already fixed in upstream activityserve but that one uses upstream httpsig and not writeas/httpsig (I made those changes as a result of the review of pherephone) so I can't update it. Should I make the required changes to writefreely to support upstream httpsig?

@qwazix
Copy link
Collaborator Author

@qwazix qwazix commented Nov 27, 2019

It turns out this is not necessary. If you do

go get github.com/go-fed/httpsig@master
go get -u github.com/writeas/activityserve

everything should work.

However the new httpsig removes the burden of setting the digest header manually and signs that header too so it might be a good idea to use that throughout.

@thebaer
Copy link
Member

@thebaer thebaer commented Nov 27, 2019

Awesome, just verified that that fixes the issue. I'll update go.mod, and we can work on moving to httpsig in the future.

Did more testing with this post. Last issue I'll address:

  • Invalid mentions link takes user to a blank page. I'll make it so user gets better feedback ("Profile not found").

For the future, a few notes:

  • Funkwhale's response:
2019/11/27 14:08:05 Status  : 403 Forbidden
2019/11/27 14:08:05 Response: {"detail":"{'summary': [ErrorDetail(string='This field may not be blank.', code='blank')]}"
  • Funkwhale mentions link to raw Actor data. Not sure if this is something we need to eventually fix on our end, though (e.g. in our case, when web visitors go to the raw actor URL, WF redirects them to the web-based profile).
  • PeerTube's response: 403 Forbidden
  • Pherephone mention link works correctly 👍
  • When POSTing to the Pherephone inbox, maybe it should also return a 403 Forbidden like PeerTube and Funkwhale? Not sure.
  • microblog.pub seems to work correctly (haven't tested on receiving end), returning a 201 Created
@dariusk
Copy link
Contributor

@dariusk dariusk commented Nov 27, 2019

Funkwhale mentions link to raw Actor data. Not sure if this is something we need to eventually fix on our end, though

My two cents: this is an issue on their end, they should redirect based on the request headers if it's a request from a web browser. Especially since they don't provide a link to the human-rendering URL in the raw JSON data.

@thebaer
Copy link
Member

@thebaer thebaer commented Nov 27, 2019

@dariusk Good point -- just opened an issue for this: funkwhale#979.

thebaer added 4 commits Nov 27, 2019
This fixes activityserve crashes caused by mentioning WriteFreely
instances.
Previously, links would go to /user/mention:@me@this.tld instead of
/mention:@me@this.tld
@gytisrepecka
Copy link
Contributor

@gytisrepecka gytisrepecka commented Dec 12, 2019

  • Maybe changing the style of mention links. Initial thoughts are to make the domain part of a user's handle faded out a little (but not completely hidden like on Mastodon). I'm also open to any ideas anyone has here

Wouldn't suggest hiding domain part - it would be the same as hiding domain part in e-mail address. Perhaps only makes sense to show only username when it is on the same instance (domain), but that wouldn't be the case - I believe in most scenarios users will be mentioning Mastodon account while actually posting from WriteFreely blog.

thebaer added 8 commits Feb 8, 2020
- Export the func
- Remove commented-out code
- Use log, not fmt for debug messages
- Remove named return parameters
- Use standard var naming schemes
- Fix spacing in queries and remove unnecessary chars
by making them absolute, not relative.
This notifies the user that the remote user doesn't exist, instead of
showing a blank page.

Ref T627
from /mention:

Ref T627
@thebaer
Copy link
Member

@thebaer thebaer commented Feb 8, 2020

I merged our latest changes, did some cleanup, moved the remote user route to /@/ instead of /mention:, and changed it so the user gets visual feedback if they mention a remote user that doesn't exist.

I think there's still some work to be done around the datastore.GetProfilePageFromHandle() func, particularly when it comes to users that don't exist. In short, we need to be able to communicate that information back up through the stack, and really we shouldn't have all that network logic inside a database function. So it still needs a lot of refactoring and logic clean up. But we can address that later. For now, this gets the job done. Merging now!

@thebaer thebaer merged commit 9be05ef into develop Feb 8, 2020
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@thebaer thebaer deleted the activitypub-mentions branch Feb 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.