Testnet nodes generates multiple root certificates - it should only have 1

[CMCDragonkai]

Describe the bug

Connecting to the testnet node 3.106.15.126 shows that it currently has multiple certificates. This shouldn't be the case because we have not executed any sort of key renewal or rotation.

Is it possible that it is using some pre-existing state? But then it shouldn't add certificates that isn't part of its root key chain.

See the certChainPEM output. Also I think this requires a better formatting... I think it is because strings aren't being formatted, they are written like a debug string with \n showing literally instead of being printed. This goes for all the strings actually.

[nix-shell:~/Projects/Polykey-CLI]$ ./dist/polykey.js agent status --client-host 3.106.15.126 --client-port 1315 --node-id v7v9ptvcdbdf8p4upok3prpmu3938ns8v4g45dib7sm5hqvvehv70
✔ Please enter the password … ********************
status	"LIVE"
pid	1
nodeId	"v7v9ptvcdbdf8p4upok3prpmu3938ns8v4g45dib7sm5hqvvehv70"
clientHost	"0.0.0.0"
clientPort	1315
agentHost	"0.0.0.0"
agentPort	1314
publicKeyJWK	{"alg":"EdDSA","kty":"OKP","crv":"Ed25519","x":"P9Oe_Y1bXoyT2cUHnebeGkaL8R8kCFbJZ-WLHX_uj84","ext":true,"key_ops":["verify"]}
certChainPEM	"-----BEGIN CERTIFICATE-----\nMIIC0jCCAoSgAwIBAgIQBlMVYk8ecAC7Ni7+Rza2YjAFBgMrZXAwQDE+MDwGA1UE\nAxM1djd2OXB0dmNkYmRmOHA0dXBvazNwcnBtdTM5Mzhuczh2NGc0NW
RpYjdzbTVo\ncXZ2ZWh2NzAwHhcNMjMxMDE5MTYxNTMyWhcNMjQxMDE4MTYxNTMyWjBAMT4wPAYD\nVQQDEzV2N3Y5cHR2Y2RiZGY4cDR1cG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3Nt\nNWhxdnZlaHY3MDAqMAUGAytlcAMhAD/Tnv2NW16Mk9nFB53m3hpGi/EfJAhWyWfl\nix1/7o/Oo4IBkjCCAY4wDAYDVR0TBAUwAwEB/zALBgNVHQ8EBAMCAf4wRQYDVR0l\nBD4wPAYIKwYBBQUHAwEGCCsGAQUFBwMCBggrBgEFBQcDAwYIKwYBBQUHAwQGCCsG\nAQUFBwMIBggrBgEFBQcDCTCBlgYDVR0RBIGOMIGLgjV2N3Y5cHR2Y2RiZGY4cDR1\ncG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3NtNWhxdnZlaHY3MIY6cGs6Ly92N3Y5\ncHR2Y2RiZGY4cDR1cG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3NtNWhxdnZlaHY3\nMIcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAdBgNVHQ4EFgQUc1eACDFkBPX3ayGV\nWLkeCbK9yC0wHwYLKwYBBAGDvk8CAgEEEBYOMS4yLjEtYWxwaGEuMTQwUQYLKwYB\nBAGDvk8CAgIEQgRAq7lP0qSpl0gGMw9toyMQ/BAkBqJC0ZmvYcvB51AV2Qc5aUZB\nr3HNcUTStm/SVDiymTVEj/hn1STUQuVg+ZLFDzAFBgMrZXADQQDyV7R7HECFhNBF\nhVEvWYS0bnPQ8ULT9FvlIhvGNv9a9D0GxkepojM+EsoNE5Ogg7SvGaG07RNkpvx6\nIVD7Nr0N\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIC0jCCAoSgAwIBAgIQBlMNtLJ+cAC0rE7qLgAxbjAFBgMrZXAwQDE+MDwGA1UE\nAxM1djd2OXB0dmNkYmRmOHA0dXBvazNwcnBtdTM5Mzhuczh2NGc0NWRpYjdzbTVo\ncXZ2ZWh2NzAwHhcNMjMxMDE5MDczMTIzWhcNMjQxMDE4MDczMTIzWjBAMT4wPAYD\nVQQDEzV2N3Y5cHR2Y2RiZGY4cDR1cG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3Nt\nNWhxdnZlaHY3MDAqMAUGAytlcAMhAD/Tnv2NW16Mk9nFB53m3hpGi/EfJAhWyWfl\nix1/7o/Oo4IBkjCCAY4wDAYDVR0TBAUwAwEB/zALBgNVHQ8EBAMCAf4wRQYDVR0l\nBD4wPAYIKwYBBQUHAwEGCCsGAQUFBwMCBggrBgEFBQcDAwYIKwYBBQUHAwQGCCsG\nAQUFBwMIBggrBgEFBQcDCTCBlgYDVR0RBIGOMIGLgjV2N3Y5cHR2Y2RiZGY4cDR1\ncG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3NtNWhxdnZlaHY3MIY6cGs6Ly92N3Y5\ncHR2Y2RiZGY4cDR1cG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3NtNWhxdnZlaHY3\nMIcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAdBgNVHQ4EFgQUc1eACDFkBPX3ayGV\nWLkeCbK9yC0wHwYLKwYBBAGDvk8CAgEEEBYOMS4yLjEtYWxwaGEuMTQwUQYLKwYB\nBAGDvk8CAgIEQgRAfoWjgh9OAgYxQ3cLdMu4eI3M7nY/mXf/YwZsImrJ+e5T8H5Z\nAZj/ivoR+jmbziEEP5VP26szsFBqNM4Hf2FXBjAFBgMrZXADQQDeQsqAB8GU64Tj\nbt7uxtk9Iqk8g8SwSlsSpJY8Crf6zqQ2bBSX/h2eer2h5WbGXhtFeHA/urEg9lco\nwHRKFH8L\n-----END CERTIFICATE-----\n"

We could take the PEM, put into a file, and use step certificate inspect to find out what these certificates are as a sanity check.

To Reproduce

1. Connect to the testnet node using the command pk agent status --client-host 3.106.15.126 --client-port 1315 --node-id v7v9ptvcdbdf8p4upok3prpmu3938ns8v4g45dib7sm5hqvvehv70
2. Observe there being multiple certificates

Expected behavior

It should only have 1 single root certificate.

Especially because we have not called to renew or rotate the certificate. The key generation is supposed to be deterministic and thus produce a deterministic root key from the recovery code and then that should result in a single certificate.

It is possible that there is pre-existing state there... perhaps the same VM and EBS storage means that older certs is being loaded too, we need to make our GC system a bit more robust, especially if a new key is being generated from scratch, old certs should be removed.

Additional context

• 6th Testnet Deployment #551 (comment)

Notify maintainers

• @tegefaulkes

[CMCDragonkai]

I think we have automatic renewal in the certificates. This could be causing this problem. We need automatic renewal when the certificate reaches expiry date. Remember the node ID doesn't change, but there's a new certificate.

[tegefaulkes]

I had a look at the certs we're getting from the seed nodes.

// first seed node
Global Data Dir: /tmp/polykey-test-global-SfVInl
  console.log
    iss: CN=v270ktdd3cs3mp1r3q3dkmick92bn927mii9or4sgroeogd1peqb0
    sub: CN=v270ktdd3cs3mp1r3q3dkmick92bn927mii9or4sgroeogd1peqb0
    notBefore: Fri Nov 10 2023 08:58:33 GMT+1100 (Australian Eastern Daylight Time)
    notAfter: Sat Nov 09 2024 08:58:33 GMT+1100 (Australian Eastern Daylight Time)

      at Object.log (tests/scratch.test.ts:90:11)

  console.log
    iss: CN=v270ktdd3cs3mp1r3q3dkmick92bn927mii9or4sgroeogd1peqb0
    sub: CN=v270ktdd3cs3mp1r3q3dkmick92bn927mii9or4sgroeogd1peqb0
    notBefore: Fri Nov 10 2023 00:14:23 GMT+1100 (Australian Eastern Daylight Time)
    notAfter: Sat Nov 09 2024 00:14:23 GMT+1100 (Australian Eastern Daylight Time)

      at Object.log (tests/scratch.test.ts:92:11)

// 2nd seed node
  console.log
    iss: CN=v7v9ptvcdbdf8p4upok3prpmu3938ns8v4g45dib7sm5hqvvehv70
    sub: CN=v7v9ptvcdbdf8p4upok3prpmu3938ns8v4g45dib7sm5hqvvehv70
    notBefore: Fri Nov 10 2023 08:58:28 GMT+1100 (Australian Eastern Daylight Time)
    notAfter: Sat Nov 09 2024 08:58:28 GMT+1100 (Australian Eastern Daylight Time)

      at Object.log (tests/scratch.test.ts:95:11)

  console.log
    iss: CN=v7v9ptvcdbdf8p4upok3prpmu3938ns8v4g45dib7sm5hqvvehv70
    sub: CN=v7v9ptvcdbdf8p4upok3prpmu3938ns8v4g45dib7sm5hqvvehv70
    notBefore: Fri Nov 10 2023 00:14:19 GMT+1100 (Australian Eastern Daylight Time)
    notAfter: Sat Nov 09 2024 00:14:19 GMT+1100 (Australian Eastern Daylight Time)

      at Object.log (tests/scratch.test.ts:97:11)

It's very likely renewing the certs with the same keys.

[tegefaulkes]

Testing the CertificateManager domain, It seems that the renew task is scheduled for an 9ish hour delay. Seems that when we calculate the time left on the certificate we're using seconds. But the tasks domain uses milliseconds. It's an easy fix.