Consider a plugin architecture

[jcushman]

This is a bit of a random suggestion, but since it's early on in a project that's being very thoughtfully architected it might be a good time to consider: I wonder if it would make sense to implement as much functionality as possible through an internal plugin API to allow for later third party customization.

The reason I'm thinking about this is the Mastodon part of the ActivityPub ecosystem is surprisingly non-plugin-based. Hometown, for example, has to be a fork instead of a plugin to control post visibility. Forks are also required for changes to search and discovery, for moderation tools, for authentication, etc. This is surprising because other ecosystems based on self-hosting, like Wordpress or Minecraft, thrive on their plugin support. And it's unfortunate both because it greatly raises the barriers to maintaining a new moderation/authentication/etc. option, and because even if those things did exist they wouldn't compose -- a server can't adopt more than one fork of the base project even if they deal with unrelated concerns.

There's a four year old open issue on plugins, but no clear statement on the idea in that thread. I imagine Mastodon in particular might be less interested because it's trying to foster a social norm that doesn't benefit from servers behaving differently, but that rationale applies less to alternative ActivityPub implementations.

But another reason Mastodon might be less interested is simply that it's a lot of work to retrofit after the fact. If you were ever to implement search or auth, for example, as plugins installed by default, it might be much more achievable to do that from the beginning.

Just a bit of drive-by musing! Kudos for this project -- I'll enjoy seeing where it goes in any event.

---

Feel free to edit:

Potential areas for a plugin API, without judgment as to desirability:

• discovery
  • search
  • user directory
  • exploration (local timeline, news, for you, etc.)
  • find my friends on other services (server-local version of Fedifinder)
• cosmetic (themes etc.)
• post types (polls, videos, etc.)
• authentication (alternate signup and login methods)
• authorization (e.g. server-local posts, like Hometown)
• moderation tools for admins
• payments (whether to support the server or to users)

[andrewgodwin]

So, this is an interesting one - I was chatting to someone today about making things more pluggable, and it's interesting to think how it might work.

Now, there are downsides to a plugin-based framework - a higher support load on the core project generally seems to be one of them (as you no longer have a single tested configuration everyone is using). That's the reason I haven't pursued it yet, much like why there's still tight coupling between the frontend and backend.

That said, I think it's a reasonable idea to work in earlier. The trick will be working out what sort of hooks need to be placed; as a former Minecraft modder, I can tell you that back in the day it was more force-of-nature rather than a nice plugin API that got a lot of our stuff working, and I'd like to avoid that.

In the interest of using this ticket for progress, is there a list of features that people would like to add to Mastodon with plugins that we can compile in order to get a good idea of the scope and API surface area here?

[jcushman]

Thanks for considering this! If it's helpful to using this ticket for progress, I added an outline of potential pluggable features up at the top. I think you're able to edit my comment; feel free if that's a useful spot to keep a running list.

As far as finding ideas, the one ActivityPub-ish project I've found so far with an active plugin framework is GNU Social -- here's that code in case there's anything useful there. Misskey docs mention plugins but I can't tell if they're real. And searching mastodon issues for plugin or fork finds a bunch of issues that might offer ideas. I put "payments" on the list from there, not because it's necessarily desirable but because it's a good example of something that comes up from time to time in the issues that the core project would have difficulty supporting.

In the interests of keeping a small fun project small and fun, I wonder if one could also bite this off just with one or two core things like authentication and search that touch everything from UI to DB, and get a sense of how tricky it is that way without taking on the whole thing.

[aviflax]

@simonw I wonder if you might have a few thoughts to share on this? My impression is that you’ve been very happy with having gone down this path with Datasette.

[jcushman]

Oh, for what it's worth the Mastodon issue search also surfaces some anti-features, like "we don't want this but have to be aware people could fork the code to support it." The ones I noticed were (a) admin UI access to DMs and (b) a quote feature that might encourage a Twitter pile-on dynamic.

[andrewgodwin]

Yeah, I have also avoided implementing the quote feature for what it's worth (as it's safer to think about that later than blindly implement it now).

Anything UI-only would be relatively easy - Django has a good boundary layer there. Something like auth is a lot harder, though, as writing an API that doesn't let you easily make security holes is difficult.

I suspect we'll never be able to support all of the above, but I'd like to at least support some of the visual stuff.

[simonw]

Adding a plugin mechanism to Datasette was the best decision I ever made for that project.

The thing I love most about plugins is that they give me the freedom to safely experiment. Given the right hooks I can spend a few hours tinkering with almost any idea and then, crucially, ship it to a production Datasette instance somewhere to try it out.

I've used this to explore features which I later rolled into Datasette core - like https://datasette.io/plugins/datasette-insert

I've also used it to explore things that I genuinely thought would turn out to be a bad idea - https://datasette.io/plugins/datasette-graphql for example - and later decided that I actually quite liked them once I'd got a good implementation up and running.

The biggest catch is that you need to design the plugin hooks extremely carefully, because you don't want to change them later in a way that breaks existing plugins.

Pluggy helps a lot here because it supports adding new parameters to existing parameter hooks in a backwards compatible manner - I've done this a bunch of times without breaking anything, e.g. in this issue:

• prepare_jinja2_environment() hook should take datasette argument simonw/datasette#1809

I've been pretty lucky so far in that almost all of my plugin hook designs have survived - but one of the major blockers on me shipping Datasette 1.0 is that I need to do a final review of them to make sure I'm happy supporting their existing shape for many years to come.

I do think plugins for Takahe would be fascinating.

There are actually two categories of plugin to consider here: there are Python-level plugins such as those in Datasette, but there's also potential for JavaScript plugins at the UI level.

Takahe has minimal JavaScript at the moment, which is a decision I fully support! But there's something very interesting about being able to customize the UI using additional scripts - kind of like an official version of Greasemonkey user scripts.

This is an area I want to improve in Datasette: it has plugin hooks for injecting additional JavaScript (used by things like https://datasette.io/plugins/datasette-vega and https://datasette.io/plugins/datasette-cluster-map) but I want to take that a step further, and have those JavaScript plugins able to coordinate with each other in some way - visualization plugins sharing the same area of the page for example. I'm still trying to figure out the best patterns to support this.

[simonw]

A rule that really helped me with this is that I set a policy of never shipping a new plugin hook without also releasing a plugin that used it.

The only hooks that I regret implementing are the ones that I didn't do this for - specifically this one: https://docs.datasette.io/en/stable/plugin_hooks.html#register-facet-classes

[andrewgodwin]

I have now fully considered a plugin architecture and will be adding one soonish, so closing this out. What form it will take will emerge as I work out what we can do as plugins versus core features.