Nixpkgs 2211

[johnalotoski]

Improvements

• Nixpkgs bumped to 22.11
• Nix bumped to 2.13.2
• Monitoring server alertmanager systemd service fix on reboot (it would sometimes not start after reboot due to a race condition dependency).
• NixOS machine configuration is now pinned by default to stateVersion 22.11.
• Equinix provisioning machine profiles and TF lifecyle improved/fixed.
• Nomad exec driver patch updates

Migration

• The bump to Nixpkgs 22.11 in this PR will require a metal deployment to all bitte machines, followed by a reboot to ensure proper kernel is running and firewall rules are behaving normally after the update. To minimize cluster job disruption, clients can be deployed to on a one-by-one or rolling basis with jobs drained from each respective client prior to deployment and reboot. Core nodes should be deploy and rebooted one-by-one to avoid loss of consul, vault or nomad functionality. See known issues below for a caveat.

Known Issues

• If any of your nomad tasks declare a "user" at the task level (ref1, ref2), for example, to control which user a docker image is started with, if the declaration does not contain a standard user name, the nomad job may fail upon restart. An example triggering this would be declaring a user of "0:0". Look for a future commit addressing this.

Testing

• Tested and deployed to bitte-world, ci-world, cardano-world