🐛 BUG: Nebula systemd unit fails to start if the clock is too off

[Fale]

What version of nebula are you using?

1.6.1

What operating system are you using?

Linux

Describe the Bug

When starting nebula with systemd unit, on systems that do not have a clock battery (eg: RPis), if the clock will automatically reset to a moment in time that is prior to the creation of the CA, nebula will fail directly.

This can be fixed by enabling systemd-time-wait-sync and wanting for time-sync.target.

Logs from affected hosts

Note: the logs have been collected on 2022-12-16, even though they have a time that is in July due to the fact that the NTP-based clock-sync daemon did not started yet.

Jul 14 00:00:22 rpi0.fale.io systemd[1]: Started nebula.service - nebula.
Jul 14 00:00:23 rpi0.fale.io nebula[809]: time="2022-07-14T00:00:23Z" level=warning msg="expired certificate present in CA pool" cert="NebulaCertificate {\n\tDetails {\n\t\tName: FalePrivateNet\n\t\tIps: []\n\t\tSubnets: []\n\t\tGroups: []\n\t\tNot before: 2022-09-03 07:54:33 +0000 UTC\n\t\tNot After: 2032-08-31 07:54:33 +0000 UTC\n\t\tIs CA: true\n\t\tIssuer: \n\t\tPublic key: 91c5a8c9aecb98a9152b0171f5f3a8d7dcf4f433a0605dd32aaabc27570a1704\n\t}\n\tFingerprint: 8bfad74452a0073bd716c1d6f5da8de281785857e1ed3a0f02f4de441bcd36b5\n\tSignature: 2c5acfcde6fd42f0dee74fc72a06da0d636c6c0fc40b4034036589f82840d16a2db3c0855f8fd4e664abd233cdc5608690a9a96939c74d1e36901ae431dac90f\n}"
Jul 14 00:00:23 rpi0.fale.io nebula[809]: time="2022-07-14T00:00:23Z" level=error msg="Failed to load ca from config" error="no valid CA certificates present"
Jul 14 00:00:23 rpi0.fale.io systemd[1]: nebula.service: Main process exited, code=exited, status=1/FAILURE
Jul 14 00:00:23 rpi0.fale.io systemd[1]: nebula.service: Failed with result 'exit-code'.
Jul 14 00:00:23 rpi0.fale.io systemd[1]: nebula.service: Scheduled restart job, restart counter is at 1.
Jul 14 00:00:23 rpi0.fale.io systemd[1]: Stopped nebula.service - nebula.
Jul 14 00:00:23 rpi0.fale.io systemd[1]: Started nebula.service - nebula.
Jul 14 00:00:23 rpi0.fale.io nebula[859]: time="2022-07-14T00:00:23Z" level=warning msg="expired certificate present in CA pool" cert="NebulaCertificate {\n\tDetails {\n\t\tName: FalePrivateNet\n\t\tIps: []\n\t\tSubnets: []\n\t\tGroups: []\n\t\tNot before: 2022-09-03 07:54:33 +0000 UTC\n\t\tNot After: 2032-08-31 07:54:33 +0000 UTC\n\t\tIs CA: true\n\t\tIssuer: \n\t\tPublic key: 91c5a8c9aecb98a9152b0171f5f3a8d7dcf4f433a0605dd32aaabc27570a1704\n\t}\n\tFingerprint: 8bfad74452a0073bd716c1d6f5da8de281785857e1ed3a0f02f4de441bcd36b5\n\tSignature: 2c5acfcde6fd42f0dee74fc72a06da0d636c6c0fc40b4034036589f82840d16a2db3c0855f8fd4e664abd233cdc5608690a9a96939c74d1e36901ae431dac90f\n}"
Jul 14 00:00:23 rpi0.fale.io nebula[859]: time="2022-07-14T00:00:23Z" level=error msg="Failed to load ca from config" error="no valid CA certificates present"
Jul 14 00:00:23 rpi0.fale.io systemd[1]: nebula.service: Main process exited, code=exited, status=1/FAILURE
Jul 14 00:00:23 rpi0.fale.io systemd[1]: nebula.service: Failed with result 'exit-code'.
Jul 14 00:00:23 rpi0.fale.io systemd[1]: nebula.service: Scheduled restart job, restart counter is at 2.
Jul 14 00:00:23 rpi0.fale.io systemd[1]: Stopped nebula.service - nebula.
Jul 14 00:00:23 rpi0.fale.io systemd[1]: Started nebula.service - nebula.
Jul 14 00:00:23 rpi0.fale.io nebula[886]: time="2022-07-14T00:00:23Z" level=warning msg="expired certificate present in CA pool" cert="NebulaCertificate {\n\tDetails {\n\t\tName: FalePrivateNet\n\t\tIps: []\n\t\tSubnets: []\n\t\tGroups: []\n\t\tNot before: 2022-09-03 07:54:33 +0000 UTC\n\t\tNot After: 2032-08-31 07:54:33 +0000 UTC\n\t\tIs CA: true\n\t\tIssuer: \n\t\tPublic key: 91c5a8c9aecb98a9152b0171f5f3a8d7dcf4f433a0605dd32aaabc27570a1704\n\t}\n\tFingerprint: 8bfad74452a0073bd716c1d6f5da8de281785857e1ed3a0f02f4de441bcd36b5\n\tSignature: 2c5acfcde6fd42f0dee74fc72a06da0d636c6c0fc40b4034036589f82840d16a2db3c0855f8fd4e664abd233cdc5608690a9a96939c74d1e36901ae431dac90f\n}"
Jul 14 00:00:23 rpi0.fale.io nebula[886]: time="2022-07-14T00:00:23Z" level=error msg="Failed to load ca from config" error="no valid CA certificates present"
Jul 14 00:00:23 rpi0.fale.io systemd[1]: nebula.service: Main process exited, code=exited, status=1/FAILURE
Jul 14 00:00:23 rpi0.fale.io systemd[1]: nebula.service: Failed with result 'exit-code'.
Jul 14 00:00:23 rpi0.fale.io systemd[1]: nebula.service: Scheduled restart job, restart counter is at 3.
Jul 14 00:00:23 rpi0.fale.io systemd[1]: Stopped nebula.service - nebula.
Jul 14 00:00:24 rpi0.fale.io systemd[1]: Started nebula.service - nebula.
Jul 14 00:00:24 rpi0.fale.io nebula[904]: time="2022-07-14T00:00:24Z" level=warning msg="expired certificate present in CA pool" cert="NebulaCertificate {\n\tDetails {\n\t\tName: FalePrivateNet\n\t\tIps: []\n\t\tSubnets: []\n\t\tGroups: []\n\t\tNot before: 2022-09-03 07:54:33 +0000 UTC\n\t\tNot After: 2032-08-31 07:54:33 +0000 UTC\n\t\tIs CA: true\n\t\tIssuer: \n\t\tPublic key: 91c5a8c9aecb98a9152b0171f5f3a8d7dcf4f433a0605dd32aaabc27570a1704\n\t}\n\tFingerprint: 8bfad74452a0073bd716c1d6f5da8de281785857e1ed3a0f02f4de441bcd36b5\n\tSignature: 2c5acfcde6fd42f0dee74fc72a06da0d636c6c0fc40b4034036589f82840d16a2db3c0855f8fd4e664abd233cdc5608690a9a96939c74d1e36901ae431dac90f\n}"
Jul 14 00:00:24 rpi0.fale.io nebula[904]: time="2022-07-14T00:00:24Z" level=error msg="Failed to load ca from config" error="no valid CA certificates present"
Jul 14 00:00:24 rpi0.fale.io systemd[1]: nebula.service: Main process exited, code=exited, status=1/FAILURE
Jul 14 00:00:24 rpi0.fale.io systemd[1]: nebula.service: Failed with result 'exit-code'.
Jul 14 00:00:24 rpi0.fale.io systemd[1]: nebula.service: Scheduled restart job, restart counter is at 4.
Jul 14 00:00:24 rpi0.fale.io systemd[1]: Stopped nebula.service - nebula.
Jul 14 00:00:24 rpi0.fale.io systemd[1]: Started nebula.service - nebula.
Jul 14 00:00:24 rpi0.fale.io nebula[911]: time="2022-07-14T00:00:24Z" level=warning msg="expired certificate present in CA pool" cert="NebulaCertificate {\n\tDetails {\n\t\tName: FalePrivateNet\n\t\tIps: []\n\t\tSubnets: []\n\t\tGroups: []\n\t\tNot before: 2022-09-03 07:54:33 +0000 UTC\n\t\tNot After: 2032-08-31 07:54:33 +0000 UTC\n\t\tIs CA: true\n\t\tIssuer: \n\t\tPublic key: 91c5a8c9aecb98a9152b0171f5f3a8d7dcf4f433a0605dd32aaabc27570a1704\n\t}\n\tFingerprint: 8bfad74452a0073bd716c1d6f5da8de281785857e1ed3a0f02f4de441bcd36b5\n\tSignature: 2c5acfcde6fd42f0dee74fc72a06da0d636c6c0fc40b4034036589f82840d16a2db3c0855f8fd4e664abd233cdc5608690a9a96939c74d1e36901ae431dac90f\n}"
Jul 14 00:00:24 rpi0.fale.io nebula[911]: time="2022-07-14T00:00:24Z" level=error msg="Failed to load ca from config" error="no valid CA certificates present"
Jul 14 00:00:24 rpi0.fale.io systemd[1]: nebula.service: Main process exited, code=exited, status=1/FAILURE
Jul 14 00:00:24 rpi0.fale.io systemd[1]: nebula.service: Failed with result 'exit-code'.
Jul 14 00:00:24 rpi0.fale.io systemd[1]: nebula.service: Scheduled restart job, restart counter is at 5.
Jul 14 00:00:24 rpi0.fale.io systemd[1]: Stopped nebula.service - nebula.
Jul 14 00:00:24 rpi0.fale.io systemd[1]: nebula.service: Start request repeated too quickly.
Jul 14 00:00:24 rpi0.fale.io systemd[1]: nebula.service: Failed with result 'exit-code'.
Jul 14 00:00:24 rpi0.fale.io systemd[1]: Failed to start nebula.service - nebula.

Config files from affected hosts

pki:
  ca: |
    -----BEGIN NEBULA CERTIFICATE-----
    CkAKDkZhbGVQcml2YXRlTmV0KLmQzJgGMLmW/K4HOiCRxajJrsuYqRUrAXH186jX
    3PT0M6BgXdMqqrwnVwoXBEABEkAsWs/N5v1C8N7nT8cqBtoNY2xsD8QLQDQDZYn4
    KEDRai2zwIVfj9TmZKvSM83FYIaQqalpOcdNHjaQGuQx2skP
    -----END NEBULA CERTIFICATE-----
  cert:  [omitted]
  key:  [omitted]

static_host_map:
  192.168.100.1:
    - lh1.fale.io:4242
  192.168.100.2:
    - lh2.fale.io:4242
  192.168.100.3:
    - lh3.fale.io:4242
  192.168.100.4:
    - lh4.fale.io:4242

lighthouse:
  am_lighthouse: true

listen:
  host: 0.0.0.0
  port: 4242

punchy:
  punch: true
  respond: true

tun:
  dev: nebula1

relay:
  am_relay: true

firewall:
  outbound:
    - port: any
      proto: any
      host: any