Make mastodon-streaming systemd unit templated

[e-nomem]

As of #24655 the streaming service no longer runs multiple processes internally so to scale it, multiple separate instances listening on different ports need to be started. This PR updates the systemd unit file for the streaming service to use templating where the argument is the port on which to listen. Multiple streaming instances can then be started. E.g.:

systemctl enable --now mastodon-streaming@4000
systemctl enable --now mastodon-streaming@4001

[ThisIsMissEm]

I'll leave this for others to review, given I'm no systemd expert (thanks for putting this PR together!)

what happens if someone runs just:

systemctl enable --now mastodon-streaming

Will it work / just start one instance on port 4000?

[e-nomem]

No, templated unit files are not usable by themselves/do not have a 'default' arg

[e-nomem]

That being said templated and non-templated unit files can live side-by-side without any problems (it can get a bit confusing from an operational perspective though). Admins can use the existing, non-templated, unit file hard coded to port 4000 and the templated unit files as long as the ports are not conflicting.

[ThisIsMissEm]

@e-nomem hey, I've given this some thought, and I think we'd be better off having two systemd units: one templated, one not. If you give me write permissions to your branch I'll push an update to make it so, now that you've shown the way.

[e-nomem]

I made you a collaborator on my fork which should do the trick. If not let me know if you have a preference on the name of the templated unit and I'll update the PR later today

[ThisIsMissEm]

Okay, I've updated this to support both sudo systemctl start mastodon-streaming and sudo systemctl start mastodon-streaming@{4000,4001,4002}. The former will start just one instance on port 4000, the latter will start three instances on ports 4000, 4001, and 4002.

The streaming servers can also then be restarted in a rolling manner by using sudo systemctl restart mastodon-streaming or stopped entirely with sudo systemctl stop mastodon-streaming regardless of whether templated or non-templated was used to start the streaming processes.

I've also included the changes to the nginx configuration to get streaming to load balance under nginx.

[ThisIsMissEm]

Whilst this isn't strictly necessary, it is very helpful for development to be able to test out the systemd unit files inside the vagrant machine.

[ThisIsMissEm]

I noticed that on multiple vagrant up's and vagrant up --provision calls, if the machine hadn't been destroyed first, then the .bash_profile file grew with repeated lines.

[ThisIsMissEm]

If husky is enabled inside vagrant, it installs a bunch of additional .husky/ scripts, which aren't present when running yarn install outside of the vagrant instance.

[ThisIsMissEm]

This is superseded by setting the packageManager field in the package.json, which corepack automatically detects, by switching to this, we prevent the vagrant instance from creating .yarnrc and .yarn/release/* files.

[ThisIsMissEm]

@renchap see here re you comment

[ThisIsMissEm]

Arguably we could change this to ReadOnlyDirectories=/home/mastodon/live as the streaming server doesn't need to write to disk ever now (unless using sockets, but here we're not using sockets)

[renchap]

I think this would be a nice improvement.

[ThisIsMissEm]

Doing this also solves these issues: mastodon/documentation#931 & #18107

[renchap]

This should be in a separate PR.

I had this in my TODO list, but you can do it as well.

Various docs will need to be updated to use corepack instead of manually installing yarn.

[ThisIsMissEm]

Docs already say to use corepack, but manually enable yarn classic, this just does it automatically — this was a gotcha for me when I was trying to test this out on an ubuntu box

[ThisIsMissEm]

The ignore scripts here will conflict with #24702 which uses a postinstall script to install the streaming server, so we'll need this change here:

Suggested change

	yarn install --ignore-scripts --frozen-lockfile
	yarn install --ignore-scripts --frozen-lockfile
	cd streaming && yarn install --ignore-scripts --frozen-lockfile

[ThisIsMissEm]

This should continue to work post the merge of #24702, but we may wish to update this to:

Suggested change

	ExecStart=/usr/bin/node ./streaming
	WorkingDirectory=/home/mastodon/live/streaming
	ExecStart=/usr/bin/node ./index.js

Such that it's inline with how streaming expects to be run based on the package.json

[renchap]

Looks good to me overall, nice improvements!

[renchap]

I think this would be a nice improvement.

[renchap]

This should be in a separate PR.

I had this in my TODO list, but you can do it as well.

Various docs will need to be updated to use corepack instead of manually installing yarn.

[ThisIsMissEm]

@renchap I can do the separation of the corepack changes post-rebase in the next 1-2 days, then this should be good for the release.

[ClearlyClaire]

I'd prefer if the vagrant and corepack changes were split out, but otherwise it looks good to me!

Upgrading is simply a matter of (mostly writing that as a draft for release notes):

1. cp ~mastodon/live/dist/mastodon-streaming*.service /etc/systemd/system/
2. systemctl daemon-reload
3. systemctl restart mastodon-streaming

And adding other instances:

1. systemctl enable mastodon-streaming@port
2. edit the nginx config to add the new server

[ThisIsMissEm]

The only additional think I'll note id that when using multiple instances, you need to manage them individually with systemctl stop|start mastodon-streaming@xxxx individual as systemctl restart mastodon-streaming will only restart a single instance

[ClearlyClaire]

@ThisIsMissEm can you update the PR without the two middle (Vagrant-related) commits?

[ThisIsMissEm]

Emelia Smith can you update the PR without the two middle (Vagrant-related) commits?

@ClearlyClaire updated, and have pulled the other two commits into a separate branch.

[ClearlyClaire]

Thanks!

[ThisIsMissEm]

(and just to check, I've rebased the branch locally on top of main, and there won't be conflicts — Though I've not pushed that as to not interrupt the tests, as that'd delay merging)

the Vagrant files have been moved to another PR