webproxy/varnish: fix runtime/state dir issues and restart on unit changes

[dpausp]

The first change already would fix our issues with state dir changes but the two other commits are still useful to make things more explicit and less confusing.

1. webproxy/varnish: final version of the fix to manage the varnish state dir properly

a) use the real varnish upstream state directory
b) only reload if it's a pure VCL change, restart otherwise

For channel upgrades (e.g. when this statedir change is rolled out) this
happens in maintenance anyway. We'll provide better public docs about
the restart conditions in the near future.

2. webproxy: varnishncsa should have a runtime dir named after the service

Using /run/varnish for the varnishncsa was quite confusing and can lead
to weird errors when varnish uses the same run time dir and restarting
one of the services clears the runtime directory for both. We don't do
that at the moment, but the varnish work dir can be changed.

3. webproxy: explicitly set work dir for varnishncsa

Before, we relied on varnishncsa's behaviour, looking for the default
varnish work dir at /var/run/varnishd (where /var/run is linked to
/run). /run/varnishd is a symlink created by us, pointing to the real
location /var/spool/varnish/.

Setting the work dir in the varnishncsa command line easier to
understand and less error-prone when the state dir changes.

@flyingcircusio/release-managers

Release process

Impact:

• [NixOS 24.05] varnish will be restarted.

Changelog:

• webproxy/varnish: use varnish default work dir and restart varnish on unit file changes. Before, we always reloaded the service which can cause various problems when changes are not picked up. We ran into this problem when the varnish work dir changed in NixOS. To simplify things, we also override the work dir and use the varnish default of /run/varnishd instead of symlinking it to a different location. This path is checked by CLI tools like varnishadm, for example. (PL-132901).

PR release workflow (internal)

• PR has internal ticket
• internal issue ID (PL-…) part of branch name
• internal issue ID mentioned in PR description text
• ticket is on Platform agile board
• ticket state set to Pull request ready
• if ticket is more urgent than within the next few days, directly contact a member of the Platform team

Design notes

• Provide a feature toggle if the change might need to be adjusted/reverted quickly depending on context. Consider whether the default should be on or off. Example: rate limiting.
• All customer-facing features and (NixOS) options need to be discoverable from documentation. Add or update relevant documentation such that hosted and guided customers can understand it as well.

Security implications

• Security requirements defined? (WHERE)
  • make sure that varnish is reloaded properly when config changes and that other changes to the unit file are picked up by restarting the service
• Security requirements tested? (EVIDENCE)
  • checked on a test VM that varnish is reloaded/restarted as expected