Skip to content

Gate the HTTPS URL and redirect on real certificate issuance - #59

Merged
ashishmax31 merged 4 commits into
mainfrom
feat/tls-cert-issuance-gating
Aug 2, 2026
Merged

Gate the HTTPS URL and redirect on real certificate issuance#59
ashishmax31 merged 4 commits into
mainfrom
feat/tls-cert-issuance-gating

Conversation

@ashishmax31

Copy link
Copy Markdown
Contributor

The problem

A release would go live, the dashboard showed it ready with a clickable https:// link, and the first person to click it landed on a browser security warning. Worse, it stayed broken — once the certificate finally issued, that user still saw an insecure site. Only a fresh incognito window worked.

This is a sequencing bug, not a reporting one. The Ingress goes out with a TLS block naming a secret that does not exist yet, so Traefik falls back to its built-in self-signed certificate. The HTTP→HTTPS redirect was attached in the same pass, so every visitor was pushed onto that broken endpoint. A user who clicked through the warning had Chrome record a per-origin security decision, and TLS session resumption kept it pinned for the whole profile.

Compounding it, TLSConfigured reported True the moment a ClusterIssuer was found — a statement about configuration, not about any certificate existing.

The fix

  • The redirect waits for a real certificate. The TLS block still goes on immediately, because that is what makes cert-manager's ingress-shim create the Certificate at all. The Traefik middleware annotation does not, so nothing steers a user toward HTTPS until it works.
  • The published URL follows reality. A TLS port publishes nothing while issuance is in flight, https:// once ready, and falls back to http:// if no certificate is ever coming — the site stays reachable instead of becoming permanently unlinkable.
  • Available and Ready are untouched. Release latency does not change. A release goes live on workload facts alone; only the link waits.
  • 301 → 302 on the redirect, so a cached permanent redirect can never outlive the certificate and strand users.

Behaviour

Situation Redirect Published address TLSConfigured reason
Pending, inside 2 min grace no omitted CertificateIssuing
Certificate ready yes (302) https://<fqdn> TLSReady
cert-manager recorded a failure no http://<fqdn> CertificateFailed
Grace elapsed, still nothing no http://<fqdn> CertificateTimedOut
ClusterIssuer missing or absent no http://<fqdn> ClusterIssuerNotConfigured / NotFound
Port without TLS n/a http:// immediately unchanged

In every terminal state the site is reachable, and in no state is a user routed to the self-signed endpoint. A late-issuing certificate still upgrades to https:// — the Ready check runs before any timing logic, so the timeout can never mask it.

Design notes

  • Failure is detected from Certificate.Status.FailedIssuanceAttempts, not the Issuing condition's reason — that condition is removed outright once issuance completes.
  • The grace clock reads TLSConfigured's own LastTransitionTime. meta.SetStatusCondition only moves it when the status changes, so it stays pinned across reason rewrites. No extra fetch, no new status field.
  • The timeout exists because the most common real failure — DNS not pointed at the cluster — produces no terminal condition at all; cert-manager retries forever. Condition-based detection alone would leave those users permanently without a URL.

Accepted tradeoffs

  • The clock can start early. If a resource first fails with ClusterIssuerNotFound and the issuer is added later, the grace is measured from that earlier failure. Result is http:// published early — the safe direction, and issuance still upgrades it.
  • The scheme can flip once on DNS-01, which can exceed the 2-minute grace. A longer global grace would leave the common HTTP-01 stall case with no URL for minutes.
  • A hand-typed https:// during the pending window still reaches the self-signed endpoint. Unavoidable while the TLS block is present, and that block is what triggers issuance.
  • No publish-once latching. A renewal that flips Ready to False reverts the address to http://. An expired certificate is genuinely broken, so that is the honest signal.
  • ObjectStorage has the same defect, untouched. internal/controller/objectstorage/ingress_reconciler.go attaches its redirect just as eagerly. Deliberately out of scope; worth a follow-up.
  • Operational: the 302 applies to every ingress sharing the namespace's redirect Middleware, including ObjectStorage. Clients holding a cached 301 keep following it until their cache expires.

Verification

  • go build ./..., go vet ./..., gofmt — clean.
  • Full unit suite green. Each of the four commits builds and passes on its own.
  • Integration: both new specs pass against a real cert-manager — HTTPS-after-issuance (40s) and the reachability guarantee with a broken issuer (30s).
  • ⚠️ The full 159-spec integration suite is still running at the time of opening. The focused run skipped 157 specs, and this branch touches shared code (the redirect Middleware is namespace-scoped; buildExternalAddresses changed for every exposed port). Please treat that as an open gate — I will report the result on this PR.

No RBAC change needed: config/deploy/01-rbac.yaml:321 already grants get/list/watch on certificates.

Adds a pure classifier over the cert-manager Certificate that ingress-shim
creates for a StackResource's Ingress, plus the lookup and the reverse
name mapping used to watch it.

The stage is one of issuing, ready, or unavailable. Explicit failure and a
grace-period timeout collapse into the same unavailable stage because every
consumer treats them identically; the reason on the result distinguishes
them for the UI.

Failure is detected from Status.FailedIssuanceAttempts rather than the
Issuing condition's reason, since that condition is removed outright once
issuance completes. The grace clock reads TLSConfigured's LastTransitionTime,
which stays pinned at the start of the pending window because
meta.SetStatusCondition only moves it when the status changes.

Nothing consumes the classifier yet.
…e the cert

A 301 is cached by browsers indefinitely. If a certificate later expires or
a renewal breaks, that cached redirect keeps forcing users onto the dead
HTTPS endpoint with no way back to a working HTTP one.

The Middleware CR is namespace-scoped and shared, so this also affects
ObjectStorage ingresses in the same namespace. That is intended.

Costs one extra redirect hop per fresh visit.
…ate issuance

A release went live with a clickable https:// link before any certificate
existed. Traefik served its self-signed default, the redirect pushed every
visitor onto it, and a user who clicked through the browser warning had that
origin's security state pinned for their whole profile — the site still read
as insecure long after the real certificate arrived.

The TLS block still goes on immediately, since that is what makes
ingress-shim create the Certificate at all. The Traefik redirect annotation
now waits for the ready stage, so nothing steers a user toward HTTPS until
it works. The key is in ManagedAnnotations, so SyncManagedAnnotations also
strips it from an existing Ingress that regresses out of ready.

ExternalAddress follows the same stage: nothing published while issuing,
https:// when ready, and http:// once no certificate is coming, so the site
stays reachable rather than becoming permanently unlinkable.

TLSConfigured now reports actual issuance instead of flipping True as soon
as a ClusterIssuer was found, and carries cert-manager's own failure text.

Available and Ready are untouched — release latency does not change.

A Certificate watch drives the upgrade the moment issuance lands. A deferred
requeue covers the silent-stall case, which produces no Certificate event at
all.
Two specs against a real cert-manager. The first uses a self-signed
ClusterIssuer and asserts the resource reaches TLSConfigured=True with an
https:// address and the redirect annotation attached.

The second points at a ClusterIssuer that does not exist and asserts the
resource still becomes Available, publishes http://, and carries no
redirect. That is the guarantee that matters: a certificate that is never
coming must not hold a release hostage or make the site unreachable.

Registers cmv1 in the test client's scheme so the ClusterIssuer can be
created.
@ashishmax31
ashishmax31 merged commit 0322ce3 into main Aug 2, 2026
2 checks passed
@ashishmax31

Copy link
Copy Markdown
Contributor Author

Integration result

Full suite on this branch: 150 passed · 1 failed · 8 skipped (159 specs, 59 min).

Both new specs passed inside that full run:

  • TLS certificate issuance gating — publishes https and attaches the redirect only after issuance
  • TLS with an unresolvable ClusterIssuer — still becomes Available, stays reachable over http

The one failure is pre-existing and unrelated

ObjectStorage credential rotation / should rotate credentials and invalidate old onesobjectstorage_test.go:317.

Attribution, run in isolation both ways:

Run Result
Spec alone on main (157 specs) ✅ passed
Spec alone on this branch (159 specs) ✅ passed

It passes on this branch when run on its own, so it is not a deterministic regression from these changes.

Root cause — a race in the test, not in the operator. The failing assertion is "old credentials no longer work", and the diagnostics captured two rustfs pods Running and ready simultaneously (...-6546cb4c5-j56nw and ...-7ff87b5f8f-kmr8f) — the old ReplicaSet's pod and the new one, mid-rollout. GetPodForDeployment (test/integration/helpers/objectstorage_helpers.go:188) returns the first Running pod matching app=<deployment> with no ReplicaSet filter, so it can hand back the old pod, which still accepts the old credentials.

That spec reaches the pod by port-forward over plain HTTP — no Ingress, no Traefik middleware, no certificate. None of the surfaces this PR touches are in that path.

Suggested follow-up (not in this PR): have the rotation spec wait for the rollout to complete before port-forwarding, or make GetPodForDeployment filter to the current ReplicaSet. Added in 0802d3b, which fits an intermittent race.

Verification summary

  • go build ./..., go vet ./..., gofmt — clean
  • Full unit suite green; each of the four commits builds and passes on its own
  • Full integration suite green except the unrelated flake above

The integration gate flagged when this PR was opened is now closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant