Multisites: feedback

[BWPanda]

After commenting here #3205, I'm wondering what people think of Backdrop's multisite feature...? Do you use it? If not, why not?

Apparently multisites aren't very popular (as indicated by @quicksketch's comment from #2555 (comment) and my own experience with needing to ask for multisite support from modules like Drush).
But I think it's awesome! Allows me to have only one instance of Backdrop (for easier core updates), and I can put modules/themes/layouts that are common to all my sites in one place, rather than duplicating them.

Interested to hear what others think. Are there good reasons not to use multisites that I'm unaware of or ignoring? Or are people just not familiar with the concept? Feedback please!

[Graham-72]

I don't use multisite but I feel sure I should and it could make life a lot easier for core updates.

I have been put off in the past (D7) by the apparent complexity of setting it up. Is there an easy-to-follow step-by-step guide for BD?

[BWPanda]

@Graham-72 There wasn't for a while, but we recently added more instructions to the README.md file in the 'sites' directory: https://github.com/backdrop/backdrop/blob/1.x/sites/README.md

EDIT: Ok, so apparently 'recently' was over 3 years ago! 😁 backdrop/backdrop@9a74777#diff-6a21d5f5af2a4014306eb5eb5229e354

If that's not enough, let me know and I'll write something up for you!

[quicksketch]

In my case, I don't find multisite to be very helpful because all sites I manage have different collections of modules, and most certainly would have small customizations as well. Updating all the sites at the same time would actually be a rather bad thing, as a module update might break one site's functionality but not another. I'd prefer to update them explicitly one at a time, with the ability to roll back to the previous version if it broke one site, but was fine on all the others.

The Pantheon write-up on multisite makes some interesting points: https://pantheon.io/blog/drupal-multisite-much-ado-about-drupal-multisite

[kreynen]

Several members of the higher ed Drupal community (including me) made the case for keeping multisite support in D8/9 in https://www.drupal.org/project/drupal/issues/2306013

While we don't use traditional multisite and can accomplish the same opcode efficiencies without multisites or install profiles using more sophisticated devops deployments, developing for install profiles and maintaining multisites are important stepping stones to understanding what's required to run a CMS as a service.

The developer progression I often see looks like something like...

• hack contib
• replace hacks with alters
• develop custom module for client A's site
• use a variation of client A's module on client B's site
• update module to include configurable variables so the same module can be used on both client A and client B's site
• realize that several sites you've built use much of the same code
• create an install profile
• realize that hosting your install profile in 10 unique, containerized hosting instances costs 10x as much as multisite
• convert sites to a multisite
• feel the pain when a module update might break one site's functionality but not another
• learn more about automated testing and cloning
• grow the sites you run as a service from 10 -> 50 -> 100 -> 1000

Another issue with traditional multisites are that the code for all sites change immediately while updates hooks can take several minutes to run. When the update includes schema changes, you run the risk of putting some sites into a WSOD state until the updates finish. Even if you are updating 4 sites concurrently, if you'd grown a multisite pool to 100 sites the last sites to get updated could be offline for 30+ minutes. Putting the sites into maintenance mode or deploying updates on off hours are both options, but in 2018 clients expect their sites to be up 24/7 and I have no interest in deploying updates at 2AM.

We use https://www.drupal.org/project/dslm to get the advantages to the way opcode caches a single codebase while still being able to do rolling updates and we'll continue to do something like that regardless of whether Backdrop supports multisite.

My concern is that I would have never be able to manage 1000 D7 sites if I didn't learn to manage 10. I can't hire people who have experience managing 1000 or even 100 sites, but I know if they've managed 10 sites in a multisite pool they'll have the skills we need to develop for Drupal or Backdrop at a much larger scale.

[BWPanda]

...all sites I manage have different collections of modules, and most certainly would have small customizations as well.
- @quicksketch

I agree! I currently host 7 Backdrop websites from the one Backdrop instance, and 28 Drupal websites from the one Drupal instance. All my sites are different; from personal blogs and small business sites, to SaaS intranets and e-commerce sites.

Here's my Backdrop setup to help explain (listing only sub-directories where they differ from Backdrop's out-of-the-box setup):

• backdrop/
  • core/
  • files/
  • layouts/
    • pw_layout/
      My custom layout that lays things out the way I like them and is customisable enough for all my sites to use. I can override the CSS in a site-specific theme as necessary.
  • modules/
  • profiles/
    • pw_profile/
      My custom install profile that sets my sites up the way I like them (e.g. configures the modules common to all sites (admin_bar, contact, ckeditor, captcha, devel, libraries, save_draft, simplify, xmlsitemap, etc.), sets up content types I always use, sets up my User 1 account, and (probably one of the reasons I first wrote a custom install profile) configures text formats/CKEditor (I hated doing that over and over for each new site)). As the install profile is only run once during setup of a new site, I can customise each site afterwards as necessary (e.g. uninstall modules, install new ones, setup different content types, etc.).
  • sites/
    • site_1/
      • files/
        Site-specific files and configuration.
      • modules/
        Site-specific modules.
      • themes/
        Site-specific theme (uses pw_theme as a base theme).
    • site_2/
    • etc.
  • themes/
    • pw_theme/
      My custom base theme that styles things the way I like them. I customise each site's theme in its own site-specific theme, which uses this as a base theme.

Note that even though my custom install profile installs and configures a bunch of modules common to all my sites, the modules and their configuration are stored in the site-specific directories, so I can still change everything per-site as necessary.

So while I'm not using multisites to its full potential in terms of having multiple sites using the same collection of modules and themes, etc. (and that's where I think a lot of the problems and criticisms of multisites comes from, case-in-point: the Pantheon article), I am, I believe, using it in a way that is better than having one Backdrop instance per site (quicker and easier Backdrop core updates, if nothing else).

[oadaeh]

Hello all.

I feel like I should be chiming in on this issue.

I have been using the multisite feature since D5 to help with maintaining many small sites. I also presented a talk in Panama, Costa Rica, and Los Angeles that focused on that feature for D8, but I also demonstrated how to do it for both D7 and Backdrop (if it had been recorded at any of those, a walk through of how to set it up for Backdrop would have been available). So I want to add my thoughts and perspective to this.

1. Regarding various site customizations:
   I understand how you feel about this. I even experienced it previously and felt the same way. However, what I found (at least in my situation), was that a high majority (80-90%) of the sites were still similar enough without wacky customizations, that I could place those sites in a multisite installation and take the special snowflakes out into their own installations. I removed (or minimized) that obstacle and still had the benefits for all of the sites in their respective situations.

2. Regarding Pantheon's article:
   I read that article with great interest at a time when I was really questioning whether I should use multisite or not, and while there are some valid points, I feel most of them are simply things you should be aware of when pursuing this path, and that the entire article as a whole was not so much why you should not use multisite, but why you should use Pantheon. I really was looking for valid reasons why I should not use multisite, but after having used it for so long, I knew the points of what Josh was talking about and how they applied to the situation, and so, I was not convinced. Also, many of his technical points are things that could affect non-multisite installations as well, i.e., a traffic spike could affect all sites on a sever that does not have any sort or site resource segregation configured, regardless of whether they are configured with multisite or not.

3. Regarding security:
   While not yet mentioned explicitly, I feel this is the biggest thing to watch for. Michel Hess gave a talk at BADCamp where he demonstrated hacking one site through another, because they were configured in a multisite installation. That really got me until I started thinking it though. The one site wasn't hacked because they were on multisite, but because the other site was insecure. Yes, if they had not be in a multisite installation, that wouldn't have happened, but it also wouldn't have happened if the other site had been secure in the first place. So, the moral of this story is that security is, and always will be, something of utmost importance to be actively managing in all situations, but maybe more so in these.

Like Peter and Kevin, I will most likely continue to use multisite, even if I have to hack it back into core (though they might not), and in fact, I am currently building a platform that makes use of multisite (and gave a presentation around it at Drupal Camp LA yesterday).

[klonos]

@oadaeh was that presentation in LA recorded?

[oadaeh]

@klonos no, it was not. The software they have for their recording system does not work with Linux, and I had nothing installed on my laptop to do it.

[serundeputy]

I do not use or have ever used multisite (on purpose).

• Saves like 128M of disk space on a code repo who cares?
• One site goes down they all go down boo
• One site gets hacked they all get hacked not good

For me the downsides simply outweigh the upsides and the few multisites that i've inherited in my career I've found more difficult to maintain and deliver solid features to the client.

2¢ Geoff.

[kreynen]

@serundeputy (and to a lesser extent @BWPanda),

We don't care about the disk space or the size of the code repos. You are missing out on one of the bigger advantages to multisite (or a symlink/atomic manager like DSLM).

The advantage isn't in the disk space saved in the PHP code on the drives. It's the memory saved caching the "compiled" php in opcode (formerly APC).

Depending on your hosting, you should be able to install https://github.com/rlerdorf/opcache-status and visualize what is actually being stored in opcode. Our opcode looks like this...

For 1000 sites, we have 1 copy of each PHP file in core and our install profile loaded into opcode. If you are running 10 sites without multisite or DSLM, you need 10x the memory for opcode since the file's resolved location is the unique key used to determine what needs to be cached. If the size and number of PHP files you have exceed the opcode size, you'll see more thrashing in opcode as files are added and removed as needed and your php response time will suffer. not every application benefits from being able to store 100% of their PHP in opcode as much as we do, so it doesn't always make sense to monitor for that... but at some point the opcode thrashing gets so bad that opcode is no longer improving performance, but making it worse.

DSLM is our answer to the issues of schema/update_hook errors when rolling out updates, eggs in one basket outages and sharing resources with hackers.

We had been doing "atomic deployments" at CU before I read Rasmus's explanation of how they manage deployments at Etsy. You would have a really hard time convincing me that support for atomic deployments isn't a requirement for a CMS framework used in a service. Our TCO is a fraction of what universities that use containers spend.

We have to limit some of what site owners could to in a more federated approach regarding configuration, but ultimately this results in fewer staff and servers to run more sites. While not an incredibly common use-case, it is one that D8 does not work well for (See https://www.colorado.edu/webcentral/2018/04/11/change-my-view-d8-isnt-best-upgrade-path-1000-d7-edu-sites) and what it motivating us to do a more thorough evaluation of Backdrop.

IMHO for our use case, efficent memory management hosting a large number of sites on the same codebase is a requirement for any CMS product or framework we'd use.

[kreynen]

The further we go with Backdrop, the more likely I'll start porting projects like https://www.drupal.org/project/profile_switcher and https://www.drupal.org/project/profile_status_check. Profile Switcher allows you to alter a site created using a core profile like Standard to leverage code from an install profile and Profile Status Check provides a UI for detecting when a module that exists in /profiles/xxx/modules is being trumped by a version in sites/all/modules. I wrote that to deal with @BWPanda's issues were for reasons I couldn't quickly explain 1 site out of 20 had an issue with the latest version of module X. Rather than leave the other 19 sites on the old version, I'd update the module in the install profile and "fork" the site with issues until I had time to figure out what the problem was.

[laryn]

I've used multisites for a few use cases:

1. Multiple simple personal sites that are not heavily customized and I appreciate only having to update code one time.
2. In development a few times I've used multisite with CiviCRM to host a temporary live donate page (e.g. a donate.orgname.org subdomain that is tied to CiviCRM) and then a development multisite (a separate subdomain also tied to the same CiviCRM installation). This is a bit of a hack but allows the donations to come in and the CRM functionality to work as expected while development of the new site happens behind the scenes. Then when I go live and am not actually using the multisite anymore, the CRM portion already contains all the donations and info that came in during development without needing to do an export/import process.