Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-tenancy with custom domains #2668

Open
Gargron opened this issue Apr 30, 2017 · 34 comments
Open

Multi-tenancy with custom domains #2668

Gargron opened this issue Apr 30, 2017 · 34 comments

Comments

@Gargron
Copy link
Member

@Gargron Gargron commented Apr 30, 2017

More technical look at #897

Idea: Let users point DNS records to an existing instance, configure specific user to be served as-if from that domain.

What parts must work correctly for this:

  • Separate field for storing whether an account is remote or local rather than relying on domain being null - store custom domain in that column?
  • All functions that check whether an account is local must check the custom domain as well as the "local domain" of the instance
  • Webfinger must answer queries for the faux-remote account
  • Other accounts must not be served from that custom domain. Accessing the website through the custom domain should be limited to only public content from the owner of the domain.

Concerns:

  • Switching instances while keeping the same domain will present the same problem as erasing/reinstalling an old instance on the same domain - other servers will remember data that you don't, any numeric IDs may reset and overlap (but generated string URIs include not just numeric ID, but also timestamp of creation, so that's not a big concern)
  • Might be the first credible case for exporting/importing data verbatim, e.g. including all small details. Just like with local users having a custom domain, statuses should be amended to be local and able to have verbatim URIs like remote ones
@ZiiX
Copy link
Contributor

@ZiiX ZiiX commented May 1, 2017

Great option to consider! You know I'm for it. With this feature, I'm thinking it would be also awesome to consider personal file/media settings as an option with this. WebDAV, S3, Dropbox, Amazon, Google, etc - this allows their media hierarchy to follow them too.

@ramsey
Copy link

@ramsey ramsey commented May 1, 2017

I feel like there are a lot of security/identity concerns to consider, especially with importing data verbatim. I could forge an import and pretend to be someone else, replaying all their posts into the new server. There’s no mechanism for signing posts and asserting one’s identity.

@saper
Copy link
Contributor

@saper saper commented May 1, 2017

@Gargron what about having a separate domain table and foreign key to that.

Also I think the primary domain should be treated just like all those tenants. Do you have a reason to have it "special"?

@Gargron
Copy link
Member Author

@Gargron Gargron commented May 1, 2017

@saper Initial design decision was:

  • Your domain could be adjusted (even just only at the very start when you're setting up/troubleshooting)
  • Don't want to have 20.000 copies of the same string in the db if it can be avoided by making it null and generating local URIs on the fly
  • Simple distinction for local vs remote content - remote content has a non-null URI

Of course right now we're exploring a use case where, perhaps, that design should be replaced... At the same time, you're right, we could have a separate domain table, then there wouldn't be 20.000 copies of the same string. Though you'd need to join the accounts and domains table on basically every single query.

@saper
Copy link
Contributor

@saper saper commented May 1, 2017

@jeremyBanks
Copy link

@jeremyBanks jeremyBanks commented May 1, 2017

Will it be possible to point to a different user name? If you need to use the same name across instances, that could limit your ability to migrate.

@ClearlyClaire
Copy link
Collaborator

@ClearlyClaire ClearlyClaire commented May 1, 2017

Off the top of my head:

  • Some of this is already handled with the distinction between LOCAL_DOMAIN and WEB_DOMAIN, except that you'd need to use a custom user-specific domain if not null instead of LOCAL_DOMAIN, which is easy when generating something, but can get messy when looking up an user. Also, Mastodon still makes some assumptions that are broken when WEB_DOMAIN is different from the user's domain (see #2012 and #2032), although until now, this can be worked around, but…
  • What about username conflicts? Should user@redirected and user@local conflict or not? If not, the workaround for misdirected webfinger queries (y0t4@46555ce) can't work in this setting, as it would need to disambiguate.
  • Further, should the URIs of redirected accounts follow the same template as local ones? If so, and user@redirected and user@local are different users, how to disambiguate? The easiest way to handle that is probably to have https://web_domain/@user@redirected and https://web_domain/users/user@redirected etc.
  • With that in place, using a remote Mastodon instance for your domain seems easy: just redirect (or proxy, but #2375) /.well-known/host-meta (and possibly /.well-known/webfinger), or use a CNAME RR for your domain, if it's to be only used for Mastodon.
  • Migration probably will be a bit tricky: in OStatus, users are not identified by their acct: (user@redirected) URIs but by their ActivityStream actor URI (which is so far an URL to the user's web profile, and will thus reference the “backend” Mastodon instance). The easiest way to migrate seem to just point to another backend, however, your actor URI will most probably change as a result. I'm not sure how bad a thing that is. Also, it won't be immediate, as remote instances having your account details will need to refresh it from webfinger to switch to the new instance, and that's currently only done in a few situations, and at most once a day.
@semarie
Copy link

@semarie semarie commented May 1, 2017

having this kind of configuration at DNS level would be great. But isn't more simple to have it at webserver level ? it would mean having a minimal code on remote webserver side (could be done with simple PHP script and/or webserver redirection).

As example, considere user@mastodon.social who want to use user@my.domain

On the remote side (my.domain):

  • /.well-known/host_meta : reply with LRDD link to webfinger on my.domain
  • /.well-known/webfinger : get information (mostly magic-public-key) from mastodon.social, and build a reply with acct:user@my.domain. Use local URL for atom, salmon, ...
  • for URL entries points (atom, salmon, ...), use HTTP redirection (Location HTTP Header) to point to mastodon.social

On mastodon side (mastodon.social):

  • permit the user to config is account as user@my.domain
  • use acct:user@my.domain in replies
@russss
Copy link

@russss russss commented May 1, 2017

I'm keen on @semarie's proposal, and in fact I started looking into the feasibility of it a while ago. The advantages as I see it are:

  • You don't need a separate hostname purely for this - which means (for example) my username could be the same as my email address.
  • Multiple users on the same domain could be housed on different Mastodon instances.

This would be similar to how OpenID works, and it feels like a more "federated" situation than pointing domain names around.

It also seems that in the original proposal, the host Mastodon instance would have to obtain an SSL certificate for each domain (or serve them unsecured which is annoying). Even with LetsEncrypt this is fiddly.

@saper
Copy link
Contributor

@saper saper commented May 1, 2017

here is what I understand under domain multi-tenancy:

Ability to point multiple A and AAAA DNS entries, add a new domain to the certificate and make sure the domains do not get mixed up.

  1. Different domain names are treated like independent instances.

1a the simplest implementation would be just to reuse the code base and point to different databases on request. This is probably achievable now with a web server configuration. I think this should be tested prior to proceeding further (how to talk to redis, store media for multiple instances, mail support etc.).

Every instance is completely independent, has its own member namespace and they federate independently. They only get upgraded at the same time. You have to run db:migrate multiple times, with a new RUBY_ENV for ever domain in the loop.

I think there must be some folk trying this right now?

MediaWiki has an interesting hack that allows sharing only some database tables between such instances, not sure it would be helpful here.

A dedicated asset domain could be another thing, but it should be made in a way that does not break the possibility to export the domain to a fully standalone instance.

1b accept multiple WEB_DOMAINS and issues and hacks related to that. This is more tricky in the implementation as it might require a new domain table, new queries etc. This provides little more benefit to 1a but it carries more of the development effort. The web server/PostgreSQL configuration is simplified compared to (1a)and every domain lives independently in the webspace, with its own member namespace.

Interesting questions appear here, like should those instances automatically be federated or not?
(I think not, they should feel independent).

There is a possibility to shortcut the traffic between the domains internally for performance reasons (maybe not necessary).

1c Accept multiple domains under one certificate - basically it is the scheme used by Google multi-tenant Apps. Everything is under google.com/a/domainname/restoftheurl

The problem here is the initial redirect - how to securely redirect domain.fluffy to mastodon.social/a/domain.fluffy on the web? In case of SMTP it works because nobody validates SMTP domains in certs and we have MX records.

One solution could be to have some custom SRV record that is universally accepted (haven't checked the spec yet). StatusNet, bot frameworks etc should also use it.

A quick check with the WebFinger RFC tells me it only supports HTTP redirect to a different domain and not a well known URL but you still need to be able to accept initial request securely.

And here comes another variant:

2a. Some fluffy form of "user@domain" as an identity alias on mastodon.social without DNS entries. I don't know how would that possibly work, except for the user being mrserious@mastodn.socialvall the time on the protocol level and just bring presented as @pinky@fluffyanimal.farm on mastodon.social only.

2b. A variant of (2a) could be considered where mastodon.social tries to convince other instances that mrserious wants to be presented as @pinky@fluffyanimal.farm, which opens a whole can of worms (misrepresentation).

May be this use case should be further elaborated (maybe it is a separate issue).

I think there is a natural development progression to go from 1a via 1b maybe to 1c.

it would be good to refer to one of those variants as we speak (or define additional ones)

@s-leroux
Copy link

@s-leroux s-leroux commented May 1, 2017

Great idea. But let me rephrase to see if I understand the typical use case:

Someone connects to me as @sylvain@yesik.it

First we query the yesik.it DNS for the _mastodon TXT record.
If not found, "yesik.it" is assumed to run its own Mastodon instance.
Otherwise, the TXT record (CNAME?) holds my current instance. Say its "mastodon.social"

So, on the background, Mastodon will understand the physical account name is @sylvain@mastodon.social
But in the UI and for all user interactions, only my logical account name is used @sylvain@yesik.it

Basically we add here a new layer of indirection. Or did I missed the point?


Several remarks:

  1. It doesn't help if someone bookmarked my account URL (https://mastodon.social/@YesIKnowIT). Or does it?
  2. What if several members of the same domain have accounts on different instances?
  3. What if I actually made several domains point toward the same account (aka, @sylvain@chicoree.fr, @sylvain@yesik.it ... being different alias to the same account -- just like I can do it for my email).
@saper
Copy link
Contributor

@saper saper commented May 2, 2017

@s-leroux so what you mean is something like federated alias.

Basically one more assumption had to be broken - not all users from @Domain are hosted on a single instance.

That is another piece of complexity, basically "domain" server would need to provide redirects/pointers,
but I understand that the trouble is for the target host to accept user@otherdomain stuff and somehow treat it's as own

Not sure it's multi-tenant domain handling, though

@alexgleason
Copy link
Contributor

@alexgleason alexgleason commented May 3, 2017

I like this idea a lot, but not sure the benefits outweigh the costs.

I think the goal is to give users more control. The easiest way to do that now is to run your own instance. That's hard though for non-tech users, while setting some DNS values is easy given a tutorial.

Would it be better to make it easy to start your own instance? I feel like a 1-click install on a web host would be ideal. I think IndieHosters is supporting Mastodon, maybe they'd be willing to set something up like this. @pierreozoux

Basically I'm wondering what the goals actually are and how to best reach those goals. I can definitely see the benefits of supporting remote domain names, but running your own instance is more beneficial: you own/control your data, there's no reason to migrate, and it produces no technical debt to the project itself. The paradigm currently in place makes sense: if you're part of a server, you're part of that community and you take their name. If you're remote, you have a remote name. This is intuitive.

On the other hand, email has some similar paradigms and any service can send emails from your domain name with the proper authentication (eg Postmark). But this is usually a generic service and to my knowledge is not built into any email providers themselves.

@pierreozoux
Copy link
Contributor

@pierreozoux pierreozoux commented May 3, 2017

I think it is a tricky discussion.

In one hand, with one postfix, I can host various domains, and it is convenient.
As a user, I can migrate from one hoster to another, it is a bit painful, but it is doable.

On the other hand, on the web, one domain name means the control over your own piece of land, your rules, your design, so it also makes sense to have your own instance (from the indieweb principles).

It is not that hard to host a mastodon instance, or pay somebody to do it. And then, it is easy to just transfer from one server to another, or from one hoster to another. And the change that has to be made to add this feature are quiet heavy I guess. Then, they also have to be maintained.

I think that multitenancy, is always a headache for developers, and it is never really well done, so even if I have the option, as a hoster, I never choose that. It is always easier to migrate people with their own instances.
(All the options I hear, DNS, running a small app server will touch 0.01% of our user base IMHO)

I'm probably not the best judge to help here, but I think the discussion has to evolve around cost/benefit for the software in itself. I'm a fan of unix philosophy, and I think the effort can serve a lot better in other areas :)

And to conclude, I also think that the discussion should happen at the standard itsefl, at ostatus level.

@jeremyBanks
Copy link

@jeremyBanks jeremyBanks commented May 3, 2017

"It's not hard to host a Mastodon instance", said one engineer to the next.

Dozens of projects have told themselves this, but the reality of self-hosting applications like this is that it does not happen for a non-trivial number of users, despite great efforts like Sandstorm. One-click install for end-users is not a responsible solution, unless you also have automatic fast and secure updating on multiple levels. We might get there one day, but don't let that ideal get in the way of a realistic solution.

Configuring DNS entries as a one-time (or at least rare) action is something that a moderately tech-competent person can handle. It will at least get Mastodon on par with email in terms of identifier portability, and is probably the most viable mid-term solution (assuming the current identifier scheme is preserved instead of moving to something cryptographic).

@DanielHeath
Copy link

@DanielHeath DanielHeath commented Jun 29, 2017

Postfix is a good example of something that is easy to operate.
Enable unattended-upgrades and security patches happen magically.
You can automatically transfer all your mail from one provider to another via standard APIs.
This is still not enough - the vast majority of users prefer free-with-ads (eg gmail) or pay-for-hosting (eg fastmail).

Hosting your own Mastodon instance means caring about:

  • Backups (you presumably care about your own history)
  • Patching security bugs quickly (Much as I love rails, it requires more frequent patches than other stuff I run)
  • SSL (letsencrypt has made this easier, but it's still a job to setup)
  • Diagnosing & fixing issues (if something goes wrong, can you pull the relevant bit of the logs to file a bug?)
  • Uptime (ok, it's a personal blog, but it still needs to be fixed when it crashes)

I build and operate production services for work - I don't need or want another thing to carry pager for.

@DanielHeath
Copy link

@DanielHeath DanielHeath commented Jun 29, 2017

@s-leroux I think a mastodon-specific TXT record is unnecessary; a regular old CNAME should do fine.

I think that separate-but-related is the ability to redirect an account from an old name to a new name. It's nice to be able to change your identity (even on the same domain) without breaking all the old URLs and losing your federated followers - there's no reason this sort of redirect shouldn't work across domains too.

@s-leroux
Copy link

@s-leroux s-leroux commented Jun 30, 2017

@DanielHeath The problem with CNAME is cannot coexist with any other record for the same resource (except DNSSEC-related records).

In particular, you cannot have both a CNAME and a MX record for the same domain.
Practically, if you use a CNAME to point your bare domain to some "mastodon provider", you no longer are able to use a mail provider on your bare domain. The workaround being to create a sub-domain per service. But that feels ugly to me.

@DanielHeath
Copy link

@DanielHeath DanielHeath commented Jun 30, 2017

@s-leroux
Copy link

@s-leroux s-leroux commented Jun 30, 2017

@DanielHeath I understand your point of view concerning someone having full access to his server.

But imagine a business or other kind of organization deciding to provide Mastodon access to their team.

However, they already have mail and web site on their primary naked domain (MX and A records). In most cases those services are externalized. The web server having chances of being a mutualized one.
The only standard record that might "bind" their domain to some mastodon provider would be the SRV record. Unless we force the user to create a specific sub-domain? Anyway, in the cases I have in mind, if I explain to them they will now have to host themselves a server just to install a reverse proxy or some other service -- surely they will answer, "never mind, we will continue using TweetDeck".

My point is if we want to target a wide audience, any viable solution should satisfy those two constraints:

  1. It should be possible to co-host Mastodon as well as classical email and web services on the primary naked domain
  2. For the purpose of adding the "Mastodon alias", we should assume the user only has access to his domain DNS configuration.
@DanielHeath
Copy link

@DanielHeath DanielHeath commented Jun 30, 2017

@duck57
Copy link

@duck57 duck57 commented Jul 6, 2017

One other use case for this (assuming I understand correctly) is for a current self-hosting user to add other domains to their existing instance. This probably will mostly be used for novelty accounts, but having the ability to use one instance of Mastodon on your server to serve several domains would be a nice-to-have feature and there are probably better uses than silly account domain names that I'm just not thinking of at the moment.

@azbulutlu
Copy link

@azbulutlu azbulutlu commented Jul 6, 2017

some questions, as a user.

  1. I use shared hosting, I have my domain, that has my website and email address. Can I use this feature? (not everyone can afford to pay for additional vps hosting. not everyone has time/skill/energy to run a server. even if a person has a vps, running a public facing server with good uptime is a lot of overhead on top of that)

  2. What is my profile url? (note that when giving social media links on a website, I use the profile url).

  3. what is my mastodon handle? (if I give someone my profile link, ideally they should be able to add me back, like they do with the profile url of a regular account. So follow button, or typing @username@customdomain or pasting the link and finding the account that way)

  4. (potentially silly question) in terms of the idea for shared certificate, my domain already has a certificate, does that mean I cannot use this domain?

what are the answers in case I use a subdomain that is dedicated to mastodon redirecting instead (e.g. social.mycustomdomain.com)

@duck57 A scenario would be that: you can have 2 domains (e.g. both your personal page and a project/blog) You might want to have "official" accounts for both, without having 2 mastodon installs (which would be a big overhead). It is not uncommon to have separate social media accounts for a personal project vs personal page, even if the project is not that of a group.

@duck57
Copy link

@duck57 duck57 commented Jul 10, 2017

@azbulutlu Thanks for coming up with a much more "professional" use case than I thought of. One install per server makes much more sense than one install per domain. Perhaps you could use some NGINX magic to proxy traffic from one domain to the other as an interim solution, but someone with more NGINX experience than I should comment if this is actually feasible as a solution or not.

@PubliqPhirm PubliqPhirm mentioned this issue Nov 16, 2017
1 of 2 tasks complete
@DanielHeath
Copy link

@DanielHeath DanielHeath commented Jul 20, 2018

@duck57 trying to proxy is fraught - you have to correctly rewrite every instance of the domain name in the body of all resources, but you also have to consider whether you're breaking CSP (since you might have changed the body of a resource which has a hash provided).

It can be made to work but it's not something I'd want to rely on, especially if I'll be updating the backend over time (which might introduce new ways for the nginx config to be wrong).

@duck57
Copy link

@duck57 duck57 commented Jul 25, 2018

@DanielHeath Something I have seen people do is use cnames, but I'm not sure if that solves this issue or simply allows a user to access multiple accounts on a single instance through an app that assumes one account per instance per user.

@DanielHeath
Copy link

@DanielHeath DanielHeath commented Jul 25, 2018

If I cname toots.daniel.heath.cc to mastodon.social, and load up http://toots.daniel.heath.cc, I just get a 'welcome to nginx' page.

@raphaeljolivet
Copy link

@raphaeljolivet raphaeljolivet commented Nov 25, 2020

There are plenty of geeks out there running their own servers and not mutualizing things. I guess this impacts badly the performances of the fediverse as a whole : shared inboxes can't be used and popular toots are replicated to many single-user instances.

To me the solution of @semarie is the way to go.
So let's go.

The first step would be to add a field customDomain in the account table, and spot the places where the username @foo is assumed to be @Foo@instance-domain.fr

For instance the user profile page should accept a full handle like :

Once the WebFinger resolution is done, I guess ActivityPub only refer to full URLs for user IDs, so everything should be fine. right ?

Next step should be to support this in user migration. Typically : migration from the default domain to the custom one, possibly on the same instance. How would migration be handled when moving from one provider to another, with the same handle ?

I would be interested to have the point of view of an active developer on this code base, so we could spot limitations and potential issues with this approach.

@DanielHeath
Copy link

@DanielHeath DanielHeath commented Nov 26, 2020

@semarie I really appreciate the concept of splitting webfinger out from hosting - that's a really nice design that avoids quite a bit of complexity.

@raphaeljolivet
Copy link

@raphaeljolivet raphaeljolivet commented Dec 9, 2020

So ... I have managed use use my custom domain and make it point to a shared instance, without hacking Mastodon
... and, it kinda works ...

I have developed a small python app webfinger-proxy to alias your own domain handles to a shared instance.

I have setup it on my own domain rjo.name.
The handle testrjo@rjo.name points to testrjo@mamot.fr
See the response of the Webfinger https://rjo.name/.well-known/webfinger?resource=testrjo@rjo.name

I have managed to follow this account and send it private messages.
I have another account on the same shared instance, and testrjo@mamot.fr appears as a completely separate remote account though.

I am not sure whether this is only on the UI or if Mastodon now have a duplicate account for this guy ...
I would need some help from the developers to figure this out : Is the handle a primary key ? Or is it the URL ?
Can you see any have adverse effects ?
Do you think the physical migration of accounts from one instance to another would still work ?

I will continue my tests with other implementations :
Pleroma, mobile clients, etc ....

@almet
Copy link

@almet almet commented Dec 9, 2020

Thanks to @raphaeljolivet last message, I took a similar approach, but with an nginx configuration and static files, since I didn't wanted to have a service running just for this.

If you want to do the same, here's a link to my work.

@almet
Copy link

@almet almet commented Dec 9, 2020

This seems to have limitations currently, since there are now ways to define in the settings what our "main" identity is.

I have two accounts : the "front" one which I want others to see and the "back" one. If I interact with users (answer, follow, etc), I should be seen with my front identity, and the back one shouldn't appear anywhere, I don't think that's what's happening here.

@ClearlyClaire
Copy link
Collaborator

@ClearlyClaire ClearlyClaire commented Dec 9, 2020

Please do not do that, and wait for solutions to be implemented in Mastodon itself, since it will require cooperation with the hosting instance.

@raphaeljolivet
Copy link

@raphaeljolivet raphaeljolivet commented Dec 15, 2020

Please do not do that, and wait for solutions to be implemented in Mastodon itself, since it will require cooperation with the hosting instance.

Ok fibne, I have added a warning note to my repo :
https://github.com/raphaeljolivet/webfinger-proxy/blob/master/README.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet