rework rsyslog startup wait for /var/log mounting

[ramonskie]

some iaases are slower to mount the log disk for example aws

solves #241

[ramonskie]

it causes our bats tests to fail https://bosh.ci.cloudfoundry.org/teams/main/pipelines/stemcells-ubuntu-jammy/jobs/test-stemcells-master-ipv4/builds/132

[aramprice]

I'm not familiar with systemd but the spirit / direction of the changes make sense

[cunnie]

Hey @ramonskie — I love this commit! Thank you.

Could you please squash these two commits into one? And re-work the commit message. I'm looking for something like this (you can cut-and-paste if you're feeling lazy):

rsyslogd runs on Jammy

Prior to this commit, `rsyslogd`, the logging daemon, was not running on Jammy-based
stemcells even though it should be running.

This commit addresses the issue by enabling `rsyslogd` via systemd control files. 
Furthermore, we took the opportunity to remove dead code which had been 
commented-out long ago.

We also streamline the start-up process: we no longer have a contorted method
to determine if /var/log is mounted; instead, we use the much-simpler
`After=var-log.mount`.

[rkoster]

@aramprice and @cunnie, forgot to update both of you. This PR was discussed at the last working group meeting. Apparently the changes were tested against the wrong stemcell and do not work with jammy yet. Which is why the PR was converted to draft again.

[ramonskie]

i investigated it further and it seems that journald gets borked on aws stemcells.
ong gcp journaldwork just fine

running a systemctl status systemd-journald* shows all service green
but when running systemctl --verify it fails.
after restarting it it works fine systemctl restart systemd-journald.service

i have no idea why this fails on the first startup.

[ramonskie]

rsyslogd is now enabled by default as we do have a preexec check now before its even able to start.
i think in the past this was disabled as the syslog socket would enable it in a later state.
but it seems this behavior has changed.

i also tried to use the following systemd options for rsyslog.service but i could not get it to behave so that rsyslog would start later than the the mounting of var/log

[Unit]
After=network.target var-log.mount
Require=network.target var-log.mount
RequiresMountsFor=/var/log

some references about journald in bosh eco system:
cloudfoundry/bosh-agent#241

[cunnie]

Hey @ramonskie , thanks for fixing rsyslogd — I was annoyed that I had to manually start it on my newly-deployed Director.

Could you please re-work the first commit message. I'm looking for something like this (you can cut-and-paste if you're feeling lazy):

rsyslogd runs on Jammy

Prior to this commit, `rsyslogd`, the logging daemon, was not running on
Jammy-based stemcells even though it should be running.

We also had to make sure that `rsyslogd` was logging _all_ the logs, especially
the ones emitted by the systemd Journal (`journald` or `systemd-journald`).

A challenge we had to overcome was that `journald` stopped when the BOSH agent
mounted `/var/log` early in the boot process; our fix was to configure
`journald` to write to volatile storage (`/run/log/journal/`) instead of
persistent (`/var/log/journal`). This did not jeopardize the logs; `journald`
forwards its logs to `rsyslogd` via a socket. We also configure `rsyslogd` to
wait until `/var/log` has been mounted before starting.

[ramonskie]

@cunnie i adjusted the commit message. and as your text was already perfect (and im a little lazy)
i just used yours :)

[cunnie]

LGTM