Skip to content

Fix wildcard cert handling + stale cert detection via TLS serial comparison#25

Merged
ananthb merged 1 commit intomasterfrom
feat/tls-serial-check
Apr 27, 2026
Merged

Fix wildcard cert handling + stale cert detection via TLS serial comparison#25
ananthb merged 1 commit intomasterfrom
feat/tls-serial-check

Conversation

@ananthb
Copy link
Copy Markdown
Member

@ananthb ananthb commented Apr 27, 2026

Summary

  • Fix wildcard cert Vault lookup — HAProxy stores wildcard certs as _.domain.pem on disk, but certificator stores them in Vault as *.domain. shouldUpdateCertificate was doing a Vault lookup with _.domain which always returned not found, so wildcard certs were never updated. Added NormalizeDomainForVault() to convert _.foo → *.foo before any Vault call.

  • Fix stale cert detection (pit138 scenario) — If Vault has a freshly renewed cert but HAProxy is still serving the expired one, shouldUpdateCertificate returned false (the Vault cert isn't expiring). Added checkSerialMismatch() which TLS-dials the HAProxy HTTPS frontend (host:443, same node as DPAPI), reads the live X.509 cert, and compares its serial to the Vault cert. If they differ → push. This bypasses DPAPI storage endpoint path normalization issues entirely.

  • Add certificatee_certificates_wildcard_total metric — Counts how many certs per endpoint still have * in their storage name (pre-migration format). Used on the Grafana dashboard to track the *.domain → _.domain Rundeck migration progress.

Files changed

  • pkg/certmetrics/metrics.go — new CertificatesWildcard gauge
  • pkg/haproxy/client.goGetServedCertificate(), NormalizeDomainForVault(), httpsHost()
  • cmd/certificatee/main.go — domain normalization, wildcard count, serial mismatch check

Test plan

  • CI (nix develop --command check) passes
  • After deploy: certificatee_certificates_wildcard_total appears in VictoriaMetrics
  • Stale cert on pit138 staging gets pushed on next certificatee cycle

🤖 Generated with Claude Code

…l comparison

- Fix: wildcard certs stored as _.domain on disk but *.domain in Vault — the
  Vault KV lookup was always failing, so no wildcard cert ever got updated.
  NormalizeDomainForVault() converts _.foo → *.foo before any Vault call.

- Fix: stale cert detection (pit138 scenario) — if Vault has a freshly renewed
  cert but HAProxy is still serving the old one, shouldUpdateCertificate()
  returned false (Vault cert not expiring). Serial comparison via TLS dial to
  the HAProxy HTTPS frontend now catches this case and triggers a push.

- Add: certificatee_certificates_wildcard_total metric to track how many HAProxy
  endpoints still have wildcard (*) certs in their pre-migration filename format.
  Useful for monitoring the _.domain migration on the Grafana dashboard.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown

Code Coverage Report

Total Coverage: total: (statements) 22.8%

Coverage by function
github.com/vinted/certificator/cmd/certificatee/helpers.go:9:		createHAProxyClients	0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:22:		main			0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:73:		maybeUpdateCertificates	0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:89:		processHAProxyEndpoint	0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:172:		shouldUpdateCertificate	0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:196:		updateCertificate	0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:218:		buildPEMBundle		0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:245:		checkSerialMismatch	0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:268:		endsWith		0.0%
github.com/vinted/certificator/cmd/certificator/main.go:20:		main			0.0%
github.com/vinted/certificator/pkg/acme/acme.go:27:			GetEmail		0.0%
github.com/vinted/certificator/pkg/acme/acme.go:32:			GetRegistration		0.0%
github.com/vinted/certificator/pkg/acme/acme.go:37:			GetPrivateKey		0.0%
github.com/vinted/certificator/pkg/acme/acme.go:42:			NewClient		0.0%
github.com/vinted/certificator/pkg/acme/acme.go:61:			setupClient		0.0%
github.com/vinted/certificator/pkg/acme/acme.go:80:			setupAccount		0.0%
github.com/vinted/certificator/pkg/acme/acme.go:118:			newAccount		0.0%
github.com/vinted/certificator/pkg/acme/acme.go:130:			getAccountKey		0.0%
github.com/vinted/certificator/pkg/acme/acme.go:159:			registerAccount		0.0%
github.com/vinted/certificator/pkg/acme/acme.go:177:			recoverAccount		0.0%
github.com/vinted/certificator/pkg/acme/acme.go:211:			saveAccount		0.0%
github.com/vinted/certificator/pkg/acme/acme.go:221:			saveKey			0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:18:	ObtainCertificate	0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:50:	GetCertificate		0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:68:	NeedsReissuing		0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:96:	arraysEqual		0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:110:	arrayContains		0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:119:	VaultCertLocation	0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:123:	storeCertificateInVault	0.0%
github.com/vinted/certificator/pkg/certmetrics/metrics.go:69:		StartMetricsServer	0.0%
github.com/vinted/certificator/pkg/certmetrics/metrics.go:89:		PushMetrics		0.0%
github.com/vinted/certificator/pkg/config/config.go:73:			LoadConfig		0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:53:		NewClient		100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:88:		NewClients		92.3%
github.com/vinted/certificator/pkg/haproxy/client.go:113:		Endpoint		100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:118:		doRequest		88.9%
github.com/vinted/certificator/pkg/haproxy/client.go:152:		ListCertificates	100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:166:		ListCertificateRefs	85.7%
github.com/vinted/certificator/pkg/haproxy/client.go:204:		UpdateCertificate	80.0%
github.com/vinted/certificator/pkg/haproxy/client.go:240:		CreateCertificate	78.9%
github.com/vinted/certificator/pkg/haproxy/client.go:275:		DeleteCertificate	100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:294:		ExtractDomainFromPath	100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:314:		GetServedCertificate	0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:349:		httpsHost		0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:360:		NormalizeDomainForVault	0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:368:		IsExpiring		100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:379:		NormalizeSerial		100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:389:		Error			100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:393:		Info			0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:397:		Debug			100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:401:		Warn			0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:406:		toLogrusFields		85.7%
github.com/vinted/certificator/pkg/vault/vault.go:18:			NewVaultClient		0.0%
github.com/vinted/certificator/pkg/vault/vault.go:27:			KVWrite			0.0%
github.com/vinted/certificator/pkg/vault/vault.go:41:			KVRead			0.0%
github.com/vinted/certificator/pkg/vault/vault.go:62:			vaultFullPath		0.0%
total:									(statements)		22.8%

@ananthb ananthb merged commit c773c24 into master Apr 27, 2026
1 check passed
@ananthb ananthb deleted the feat/tls-serial-check branch April 27, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant