New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ActivityPub support to connect with Mastodon #4688

Open
MrPetovan opened this Issue Mar 27, 2018 · 56 comments

Comments

@MrPetovan
Collaborator

MrPetovan commented Mar 27, 2018

Mastodon has announced that they will drop OStatus in the future to foxus solely on ActivityPub that they were among the first projects to implement.

This makes the Mastodon implementation the de facto reference ActivityPub implementation for microblogging.

While this task is related to #4592, there's a significant chance that implementation to connect with both services will heavily diverge, hence this new separate issue.

Thanks to @lionirdeadman for the reminder.

@tobiasd

This comment has been minimized.

Collaborator

tobiasd commented Mar 27, 2018

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Mar 27, 2018

As far as I know the specs are mostly up to the implementation, so we will have to follow what they are doing in order to be able to fully inter-operate with them.

@hoergen

This comment has been minimized.

hoergen commented Mar 27, 2018

There are some more projects like PeerTube, NextCloud with ActivityPub, that makes it very usefull to support it.
Found this maybe it helps in a way?
https://fed.brid.gy/ (stumpled here upon it https://activitypub.rocks/implementation-report/)

@tobiasd

This comment has been minimized.

Collaborator

tobiasd commented Mar 27, 2018

So you want rather add issues for all services that support AP instead of having one isse called "implement AP"?

@hoergen

This comment has been minimized.

hoergen commented Mar 27, 2018

There should be only one issue "Implement AcitivityPub". The service behind is for that task not relevant.

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Mar 27, 2018

How would you know?

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Mar 27, 2018

So you want rather add issues for all services that support AP instead of having one isse called "implement AP"?

Yes, you can even see what I mean on the main page of the Bridgy Fed service @hoergen linked. They support Activity Pub for Mastodon and Hubzilla, but not MediaGoblin or postActiv.

I believe there can be a basic shared ActivityPub support, but it wouldn't be enough to fully interface with specific services. Hence the separate tasks.

@hoergen

This comment has been minimized.

hoergen commented Mar 27, 2018

AP as a general protocol isn't bound on a project like Mastodon. Maybe that Mastodon has implementd special things on that, I don't know. But it would make more sense to start working in general on that protocol, to support the standard and then start working on special implementations.

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Mar 27, 2018

ActivityPub is a "loose standard", meaning that it doesn't define precisely each interaction. "Supporting ActivityPub" doesn't make any sense, because, per the standard, implementers are forced to make local decisions about how to convey specific features via ActivityPub because of an intended lack of description.

In this context, you can claim "supporting ActivityPub" and not being able to interact with any other service also claiming "supporting ActivityPub" just because you made different implementation decisions.

Another angle: Mastodon implemented ActivityPub for internal communication between Mastodon instances. So they designed their usage of ActivityPub according to their specific usage and features since the standard actually encourages you to do so by leaving parts intentionally vague. However, if any other project wants to communicate with Mastodon instances, they will have to know which specific decisions were made for the intentionally vague parts of the standard because you can't just guess.

It is different from OStatus where interactions were much more precisely defined, so you actually could rely on the standard definition to implement a single connector that would mostly work with most OStatus services because the vague parts were reduced to a minimum.

@hoergen

This comment has been minimized.

hoergen commented Mar 27, 2018

Hm ok, I didn't know that this standard is more like a "maybe", but I understand now why specifying a service with the issue.

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Mar 27, 2018

Yeah, it's confusing, but it's the least resistance route, if each interaction had to be defined precisely while aiming to federate every single service, they would still be arguing about it and the standard would never had seen the light of day.

So it clearly is a tradeoff, only adoption will tell if it was worth it.

@annando

This comment has been minimized.

Collaborator

annando commented Mar 27, 2018

AFAIK There are two different ways of distributing comments. One is like Friendica and Diaspora are doing (using the thread starter as relay) and the other one is like OStatus and Pump.io are doing (distributing comments to the commenters follower). Additionally linked signatures may be used or not.

Then there can be incompatibilities in the distributed content, since there doesn't seem to be a list of supported HTML elements.

AFAIK there is many room for interpretation.

@strypey

This comment has been minimized.

strypey commented Mar 30, 2018

Yes, you can even see what I mean on the main page of the Bridgy Fed service @hoergen linked. They support Activity Pub for Mastodon and Hubzilla, but not MediaGoblin or postActiv.

I'm not sure that's a fair example @MrPetovan . Bridgy Fed is a multi-protocol bridging service for connecting IndieWeb sites to federation social networking ("fedsocnet") servers using both OStatus and ActivityPub. There's a fair few more moving parts to this than connecting one kind of AP-compliant package to another. Also, I'm guessing MediaGoblin and postActiv aren't supported via AP because AFAIK their own AP implementations aren't finished and deployed yet ;)

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Mar 30, 2018

Please remind me again what is your stake in this internal Friendica discussion?

@strypey

This comment has been minimized.

strypey commented Apr 7, 2018

Sorry @MrPetovan, I'll get off your lawn ;) If these GitHub Issues are not the appropriate place to discuss these issues with the Friendica community, I apologise for butting in, and please let me know where the more appropriate forum can be found.

My only stake is as a fediverse user, who wants to be able to smoothly communicate with users on all other apps that support the same standards. If you're correct that every project that supports ActivityPub is going to have to develop bespoke solutions to connect smoothly with each other, then that doesn't scale, and the standard is pointless in its current form, and needs to be improved until this is no longer the case. Of course, it's also possible that the spec isn't as vague as you (and the Diaspora folks) believe.

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Apr 7, 2018

I appreciate your apology.

After all, I have the same wish as you, that ActivityPub support could be done once and for all, but I trust @annando, our resident protocol expert, with his analysis that it will require additional work for each AP-supporting project we want to communicate with.

@cwebber

This comment has been minimized.

cwebber commented Apr 17, 2018

AFAIK There are two different ways of distributing comments. One is like Friendica and Diaspora are doing (using the thread starter as relay) and the other one is like OStatus and Pump.io are doing (distributing comments to the commenters follower).

In general, a person replying distributes to the participants. The case in which comments are forwarded by parent participants in the thread is the forwarding mechanism in AP and it's for a fairly specific scope: follower collections that you don't have access to because they're another participant's followers. In that case, that participant can forward to their followers.

Additionally linked signatures may be used or not.

When federating content from server to server, HTTP signatures are used to verify that the content came from the actor that was claimed. However for any content that may be forwarded, yes in that case you'll need linked data signatures. Not all implementations are currently using LDS but it's strongly encouraged, though all implementations are using HTTP signatures.

Then there can be incompatibilities in the distributed content, since there doesn't seem to be a list of supported HTML elements.

This bit is true. This was true in OStatus as well IIRC.

It is different from OStatus where interactions were much more precisely defined

Could you be specific? I think we may be retroactively glorifying OStatus a bit? I remember there being a lot of problems getting different OStatus servers to actually interoperate back when people were trying to do it. OStatus was a very meta-standard and my conversations with implementors, including Evan Prodromou who wrote the standard, was that many behaviors were somewhat undefined. However what OStatus did have was a set of well known deployments that one could test against, and so community knowledge of what expected behavior began to gel. I could be wrong, but this is my understanding.

@cwebber

This comment has been minimized.

cwebber commented Apr 17, 2018

BTW I do encourage referring to the ActivityPub specification over Mastodon's particular implementation. Recently one of the Pleroma developers said something like "We will submit an ActivityPub implementation report after we become ActivityPub spec compatible rather than Mastodon compatible". I think that was a good perspective.

If you find that specific things are ambiguous, and these things do happen, try as we might, in standards development (almost any standard that exists has undefined behavior), I'd encourage you to join the SocialCG, of which I'm co-chair. Really! We'd love to have you involved. I'm happy to do any discussion about covering any grey areas, and the community group is able to issue a community report clarifying those. Should we get enough evidence that a new iteration of the specification needs to come out clarifying areas that were previously unclear, we can try to charter a new working group to issue a new standard. This happens with standards all the time... pretty much every protocol you're likely communicating over to have this exchange has gone through such processes, and that's the natural evolution of standards work.

@annando

This comment has been minimized.

Collaborator

annando commented Apr 17, 2018

@cwebber I really don't like the direct distribution. I would always use this forwarding. When working with the different protocols (Diaspora, OStatus and DFRN) I really appreciated the relaying of comments by the thread owner.

When not doing the relay stuff, strange situations can happen. Just imagine that you are in a larger discussion that is to distributed to 300 contacts. You are on a slow machine and the delivery process of the comment takes 5 minutes. In these 5 minutes the first receivers could already have answered your comment - which could lead to the situation that a comment to a comment arrives before the comment had arrived. A relay will always deliver the comments chronically. So your comment will be distributed first, then the comment to the comment will be distributed.

Also I really dislike adding all the receivers of a post in the post itself. This would automatically reveal the list of my contacts to our contacts. We have an option in Friendica to hide our contact list. This would be obsolete then.

Concerning the HTML restrictions in OStatus: This never had been an important issue, since it had been an export to another system. So we simply restrict the HTML that we are sending via OStatus. We never used it for our internal communication.

I'm unsure what to do. For internal communication I would prefer to only use this forwarding and I would completely ignore the direct distribution of comments. Additionally I don't want to add a list of all of my followers in the post. Concerning the content, I would prefer using our own BBCode. Although the default content type is HTML, we can always use our own, can't we?

I guess we really have to detect the target software, so that we adapt to it. Otherwise this will be much "fun". AFAIK there is no way that servers expose their capabilities?

@cwebber

This comment has been minimized.

cwebber commented Apr 17, 2018

I really don't like the direct distribution. I would always use this forwarding. When working with the different protocols (Diaspora, OStatus and DFRN) I really appreciated the relaying of comments by the thread owner.

I understand, there are different tradeoffs. ActivityPub is much more email-like. This has some advantages in that exchanges can also break off, the same way do in email threads... it's very common for myself and another participant to take part of a conversation "off list". But this generality leads to some different choices.

When not doing the relay stuff, strange situations can happen. Just imagine that you are in a larger discussion that is to distributed to 300 contacts. You are on a slow machine and the delivery process of the comment takes 5 minutes. In these 5 minutes the first receivers could already have answered your comment - which could lead to the situation that a comment to a comment arrives before the comment had arrived. A relay will always deliver the comments chronically. So your comment will be distributed first, then the comment to the comment will be distributed.

The most common case like where you're talking about above is very akin to an email mailing list, and indeed there is currently work in the ActivityPub section of the fediverse to use Groups where a Group has an inbox and relays its contents to its subscribers. In this case, the behavior I think would be the same as what you said above, yeah?

Also I really dislike adding all the receivers of a post in the post itself. This would automatically reveal the list of my contacts to our contacts. We have an option in Friendica to hide our contact list. This would be obsolete then.

Well, there are two mitigations to that: bto/bcc for individual folks on the thread, and simply not revealing the contents of a collection such as your followers list. Right now I believe Gargron is implementing support for hiding the followers/following collection contents to anyone but the actor whose followers/following collections those are. That's totally reasonable per the standard. No need to reveal anyone in particular.

Concerning the HTML restrictions in OStatus: This never had been an important issue, since it had been an export to another system. So we simply restrict the HTML that we are sending via OStatus. We never used it for our internal communication.

I'm unsure what to do. For internal communication I would prefer to only use this forwarding and I would completely ignore the direct distribution of comments. Additionally I don't want to add a list of all of my followers in the post. Concerning the content, I would prefer using our own BBCode. Although the default content type is HTML, we can always use our own, can't we?

Good news! We have support for this with the source property, so you can absolutely include your bbcode source and have the "general" content be the html content sent across the wire.

I hope that helps!

@annando

This comment has been minimized.

Collaborator

annando commented Apr 17, 2018

Just some side note: One huge problem I do have with AP is that I have problems understanding the sentences. In the above post for example I had to look for a translation of words like "akin" or "mitigations" - and most longer sentences I have to read multiple times to understand them. And I completely got lost in sentences like "This has some advantages in that exchanges can also break off, the same way do in email threads" or "Gargron is implementing support for hiding the followers/following collection contents to anyone but the actor whose followers/following collections those are.". I know the words, but don't understand the meaning. (This is one of the reasons why I never participated in the SocialWG, although I had been invited - in many discussions on the mailing list I felt like a caveman listening to a group of engineers)

Concerning your post: These groups you mentioned sound like our forums. Nice to hear that something like this is planned as well.

Concerning the different content types: I know that we can use several ones. I think I already discussed about this when AS2 was designed.

BTW: I don't see that much problems in implementing AS2. It is written relatively clear with many examples. AP in difference is much more complicated. Currently I'm not even sure if webfinger is still used or not and if we still could address users in the user@domain.tld format (It is crucial for us to be able to transform the url format of a profile in the address format - and vice versa). Additionally the AP definition refers to other standards like these linked signatures so that these specifications needs to be learned as well.

And it's slightly confusing that the definition for the signed HTTP expired last month: https://tools.ietf.org/html/draft-cavage-http-signatures-08

P.S.: Which is the best place for asking questions about the AP and AS2?

P.P.S: It would have been better if specifications had been written in "easy language".

@strypey

This comment has been minimized.

strypey commented Apr 18, 2018

@annando

It would have been better if specifications had been written in "easy language".

I totally get this concern. I wonder if it's a bit like legal documents, where they must be written in very specific jargon, so there's no room for misinterpretation? Or is just a case of native English speakers writing the specs, and using more flowery language than is strictly necessary? If it's the first option, human-readable interpretations could be drafted. If it's the second, perhaps the spec could be amended to simplify the language used? In either case, it would be great to have the ActivityPub spec (and all open protocol specs) translated into as many languages as possible.

@hoergen

This comment has been minimized.

hoergen commented Jun 2, 2018

@VVelox

This comment has been minimized.

VVelox commented Jul 25, 2018

I am curious, any progress on this?

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Jul 25, 2018

Not yet, we are in the middle of 2-3 big refactoring endeavors at the moment, but it definitely is first on the list once the dust settles.

@annando

This comment has been minimized.

Collaborator

annando commented Jul 25, 2018

@VVelox I have it on my list. Currently I'm busy with rearranging the item storage. This is a task that I postponed for much too long. But after this is done, I want to focus on this.

@tobiasd tobiasd added this to the 2018.11 milestone Aug 19, 2018

@strypey

This comment has been minimized.

strypey commented Sep 10, 2018

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Sep 10, 2018

Diaspora protocol support will be maintained, only DFRN (specific to Friendica) will be deprecated and only kept for backward-compatibility.

@annando

This comment has been minimized.

Collaborator

annando commented Sep 10, 2018

@strypey Once when AP can do everything that we can do with DFRN, there is no need anymore to keep it. Removing it will also slim down the whole code.

But: This will take some time. it will be hard work, to implement all the special features of DFRN. And of course we have to keep the whole stuff for a longer time (> 1 year) to be able to give the admins some time to upgrade their systems.

The Diaspora protocol will be kept as long as Diaspora doesn't move to AP as well. Same is valid for OStatus. As long as there are GNU Social servers (and no working AP plugin), the OStatus protocol is kept.

@m4sk1n

This comment has been minimized.

m4sk1n commented Sep 11, 2018

I guess it would be good to start working on drafts of AP/AS extensions that will be used by Friendica.
Also, if one day ActivityPub will replace DFRN, are there any plans to use its vocab internally?

@annando

This comment has been minimized.

Collaborator

annando commented Sep 11, 2018

I'm not much into "plans". Mostly I intend to do "A" but end up with "B". Once the basic AP communication does work, I will extend it step by step. I will stay in contact with the "litepub" people and the "socialcg" to ensure to be standard compliant and to not replicate stuff that is already possible within the existing standard.

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Sep 11, 2018

Also, if one day ActivityPub will replace DFRN, are there any plans to use its vocab internally?

What are the vocabulary major differences? Inbox/outbox, message?

@annando

This comment has been minimized.

Collaborator

annando commented Sep 11, 2018

I's more like a "notice" should be stored as a notice, an "article" as an "article" and such stuff. Currently we are using some stuff from AS1.

@annando

This comment has been minimized.

Collaborator

annando commented Sep 22, 2018

Like I reported in the developer forum, there are incompatibilities with Mastodon concerning non public posts. I created an issue there: tootsuite/mastodon#8752

@annando

This comment has been minimized.

Collaborator

annando commented Sep 27, 2018

@strypey While working on AP, I had problems with the Mastodon implementation, since they decided to handle only (private) posts to the "followers" collection, see here: tootsuite/mastodon#8752. After this I had a phase where Pleroma hadn't accepted my messages - with the exception of replies. . Then there are different philosophies about the way that comments should be distributed. Also some are signing their content, some don't.

@AlfredSK

This comment has been minimized.

AlfredSK commented Oct 3, 2018

Can this issue be closed now?

@annando

This comment has been minimized.

Collaborator

annando commented Oct 3, 2018

I guess that we can do so. @MrPetovan should decide, it's his issue.

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Oct 4, 2018

So far it's been hit and miss:

Displayed network on the contact page:

  • @amphetamine@social.wxcafe.net: Mastodon (AP)
  • @Angle@anticapitalist.party: Mastodon (AP)
  • @kingu_platypus_gidora@octodon.social: Mastodon (AP)
  • @kingu@pawoo.net: ActivityPub
@annando

This comment has been minimized.

Collaborator

annando commented Oct 4, 2018

I added a to-do for this.

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Oct 5, 2018

Today's report: Out of 272 total OStatus contacts (including followers), 12 of them are now over on ActivityPub. Out of those 12, I can see posts from 6 of them in my network stream filtered by ActivityPub. The 6 others accounts kept posting over the last 48 hours but I didn't receive their posts locally. Among those 6, I updated manually 1 contact.

@annando

This comment has been minimized.

Collaborator

annando commented Oct 5, 2018

Are these bidirectional contacts? If yes, then please try to unfollow and follow again. If this works, then I will see if I can transmit some "follow" request upon changing the network type.

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Oct 5, 2018

Out of the 6 contacts I'm not receiving posts from for 2 days, 4 are bidirectional contacts. I'l try to unfollow/refollow them.

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Oct 5, 2018

I'm now receiving posts from the accounts I've unfollowed/refollowed. It still is a huge caveat of the automatic account conversion.

@annando

This comment has been minimized.

Collaborator

annando commented Oct 5, 2018

That's a great information that it works this way. So I can add now my proposed code.

@AlfredSK

This comment has been minimized.

AlfredSK commented Oct 5, 2018

I have more and more converted contacts (but I haven't nearly as much as @MrPetovan ). So far I didn't notice any troubles with the converted contacts. Bidirectional contacts do work after conversion so far (automatic and manual conversion). I'm not sure about other OStatus contacts because I don't follow them (they only follow me).

@annando

This comment has been minimized.

Collaborator

annando commented Oct 8, 2018

@MrPetovan Does it work now? Or do you think that I should add some resubscribe functionality?

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Oct 8, 2018

I now have 21 ActivityPub contacts, including some new follower-only.

However, I'm following https://mastodon.ar.al/@aral and the last post from him on my node is 6 days old while he kept posting on Mastodon. Not sure if he was converted before or after your fix.

@annando

This comment has been minimized.

Collaborator

annando commented Oct 8, 2018

Okay, so I have to code some functionality like the one for OStatus. Possibly directly on the contact?

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Oct 8, 2018

What functionality are you talking about?

@annando

This comment has been minimized.

Collaborator

annando commented Oct 8, 2018

We have this "repair OStatus". But I guess I will simply add some "resubscribe" at the contact.

@MrPetovan

This comment has been minimized.

Collaborator

MrPetovan commented Oct 8, 2018

Sounds good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment