[bug]: All dev and prod endpoints unreachable (HTTP 530 / Cloudflare Error 1033)

[devantler]

Expected Behavior

All dev and prod HTTPS endpoints should be reachable and return valid responses:

URL	Environment	Expected
https://dev.platform.devantler.tech	Dev	Homepage (OAuth2)
https://whoami.dev.platform.devantler.tech	Dev	Whoami echo
https://headlamp.dev.platform.devantler.tech	Dev	Headlamp UI
https://dex.dev.platform.devantler.tech	Dev	Dex OIDC
https://flux.dev.platform.devantler.tech	Dev	Flux status
https://goldilocks.dev.platform.devantler.tech	Dev	Goldilocks (OAuth2)
https://oauth2-proxy.dev.platform.devantler.tech	Dev	OAuth2 proxy
https://platform.devantler.tech	Prod	Homepage (OAuth2)
https://whoami.platform.devantler.tech	Prod	Whoami echo
https://headlamp.platform.devantler.tech	Prod	Headlamp UI
https://dex.platform.devantler.tech	Prod	Dex OIDC
https://flux.platform.devantler.tech	Prod	Flux status
https://goldilocks.platform.devantler.tech	Prod	Goldilocks (OAuth2)
https://oauth2-proxy.platform.devantler.tech	Prod	OAuth2 proxy

HTTP-to-HTTPS redirect (301) should work for all endpoints.

Actual Behavior

HTTPS redirect (301) works because it's handled by Cloudflare, not the origin.

Dev environment root cause:

• The Flux OCIRepository is configured with tag: dev but workload push pushes the artifact with tag latest
• This causes MANIFEST_ Flux cannot pull the manifestsUNKNOWN
• No infrastructure or app workloads are deployed (only Flux and Cilium system pods are running)
• The Cloudflare tunnel (cloudflared) is not running, so all requests through Cloudflare fail
• Upstream KSail bug: [bug]: OCI tag mismatch between workload push and Flux OCIRepository configured by cluster create/update ksail#3927

Prod environment root cause:

• CD workflow (v2.33.3) failed at the workload reconcile step with: wait for oci repository ready: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
• The reconcile step is missing GITHUB_TOKEN env var (see [bug]: CD and CI workflows missing GITHUB_TOKEN in workload reconcile step #1343)
• Likely also affected by the same OCI tag mismatch as dev

Additional observations:

• Dev cluster has a single node named hcloud-prod-cx23-fsn1- the "prod" prefix is misleading for a dev cluster3
• Dev cluster has no HelmReleases, no Gateways, no nothing beyond Flux and CiliumHTTPRoutes
• Only namespaces present in dev: default, flux-system, kube-system, kube-node-lease, kube-public, cilium-secrets

Steps to Replicate

530
530
error code: 1033
MANIFEST_UNKNOWN: manifest unknown
Only flux-system and kube-system pods

[devantler]

Resolution

The root cause is an upstream KSail bug where cluster create/update configures the Flux OCIRepository with a tag derived from metadata.name (e.g., dev, prod), while workload push uses spec.workload.tag (latest). This mismatch causes MANIFEST_UNKNOWN errors, preventing Flux from pulling manifests and deploying any workloads.

Tracked upstream: devantler-tech/ksail#3927

The current ksail.dev.yaml and ksail.prod.yaml configs are already correct for the post-fix state (tag: latest). No changes are needed in this once the upstream KSail fix is released, redeploying will resolve the issue.repository

Additional context:

• The missing GITHUB_TOKEN/GITHUB_ACTOR in the CD workflow reconcile steps was already fixed in fix: add GITHUB_TOKEN and GITHUB_ACTOR to workflow reconcile steps #1345.