Kargo 1.10.0 stage stuck reconcileing

[scottrus]

I was testing 1.10.0 today on a previously working config. This stage accepts downstream freight with autopromotion after a soak time. However I'm seeing a reconcile error on the stage that I'm not sure how to troubleshoot yet. The stage itself isn't providing any details of a promotion or the freight.

I did assemble this freight manually in the UI after warehouse discovery and it passed though several other stages previously.

States the freight 47e9743bac25bb58097713fadc845e280b6d3740 is an invalid value on the stage:

status:
  autoPromotionEnabled: true
  conditions:
    - lastTransitionTime: 2026-04-17T18:18:29Z
      message: 'failed to auto-promote Freight: error creating Promotion for Freight
        "47e9743bac25bb58097713fadc845e280b6d3740" in namespace "kargo-ccs":
        admission webhook "promotion.kargo.akuity.io" denied the request:
        Promotion.kargo.akuity.io
        "aquila-secondary.01kpepwv18c75cep6k16yw28cj.47e9743" is invalid:
        spec.freight: Invalid value: "47e9743bac25bb58097713fadc845e280b6d3740":
        Freight is not available to this Stage'
      observedGeneration: 1
      reason: ReconcileError
      status: "False"
      type: Ready

I can verify that the freight exists and looks good

% kargo get freight -p kargo-ccs                                                    
NAME                                       ALIAS             ORIGIN                     AGE
4841a8f1a51ecf4789ff39a5579213c18cce940a   punk-hound        Warehouse/ccs-sample-app   3h34m
47e9743bac25bb58097713fadc845e280b6d3740   queenly-tuatara   Warehouse/ccs-sample-app   3h34m

% k get freight -n kargo-ccs 47e9743bac25bb58097713fadc845e280b6d3740 -o yaml
alias: queenly-tuatara
apiVersion: kargo.akuity.io/v1alpha1
artifacts: []
charts:
- name: ""
  repoURL: oci://kargo-helm-virtual.jfrog.io/kargo-sample-app
  version: 0.0.9
commits:
- author: Scott Russell <scott.russell@hpe.com>
  branch: main
  committer: GitHub <noreply@github.com>
  id: 1823ae135bcd93c6d596a1d18df9290b869eb787
  message: Add feature flags to values.yaml
  repoURL: https://github.com/owner/kargo-deploy-sample-app.git
  tag: ""
images:
- annotations:
    org.opencontainers.image.created: "2026-03-19T13:52:59.176Z"
    org.opencontainers.image.description: Kargo sample app for pipeline testing
    org.opencontainers.image.licenses: ""
    org.opencontainers.image.revision: e9294f25c83a9768b9d7ee8ae573ab745c2c4036
    org.opencontainers.image.source: https://github.com/owner/kargo-sample-app
    org.opencontainers.image.title: kargo-sample-app
    org.opencontainers.image.url: https://github.com/owner/kargo-sample-app
    org.opencontainers.image.version: v0.0.9
  digest: sha256:e9106923f84dd2339e724d000ea326230884799632421d57809f5f4d520227b4
  repoURL:kargo-docker-virtual.jfrog.io/glcp/kargo-sample-app
  tag: v0.0.9
kind: Freight
metadata:
  creationTimestamp: "2026-04-17T18:21:52Z"
  generation: 1
  labels:
    kargo.akuity.io/alias: queenly-tuatara
  name: 47e9743bac25bb58097713fadc845e280b6d3740
  namespace: kargo-ccs
  resourceVersion: "22306"
  uid: ee71d730-3f56-4b3a-b879-5eb15bea1a2f
origin:
  kind: Warehouse
  name: ccs-sample-app
status:
  currentlyIn:
    aquila-primary:
      since: "2026-04-17T20:57:15Z"
    mira-primary:
      since: "2026-04-17T18:26:01Z"
    mira-secondary:
      since: "2026-04-17T18:26:08Z"
    pavo-primary:
      since: "2026-04-17T18:33:03Z"
    pavo-secondary:
      since: "2026-04-17T19:04:06Z"
    polaris:
      since: "2026-04-17T18:25:44Z"
  verifiedIn:
    aquila-primary:
      verifiedAt: "2026-04-17T20:57:21Z"
    mira-primary:
      longestSoak: 1m7.533296473s
      verifiedAt: "2026-04-17T18:24:17Z"
    mira-secondary:
      longestSoak: 1m1.830435376s
      verifiedAt: "2026-04-17T18:24:30Z"
    pavo-primary:
      verifiedAt: "2026-04-17T18:33:08Z"
    pavo-secondary:
      verifiedAt: "2026-04-17T19:04:10Z"
    polaris:
      longestSoak: 2m22.544758022s
      verifiedAt: "2026-04-17T18:22:43Z"

[scottrus]

These are the first entries I can find where I think it tried to auto-promote after soak time, but I'm not sure how to check that.

kargo-api-7f7c599cdd-kbtct api 2026-04-17T21:49:52Z     ERROR   option/log.go:53        finished unary call: invalid_argument: Freight "47e9743bac25bb58097713fadc845e280b6d3740" is not available to Stage "aquila-secondary"  {"connect.service": "akuity.io.kargo.service.v1alpha1.KargoService", "connect.method": "PromoteToStage", "connect.start_time": "2026-04-17T21:49:52Z", "connect.duration": "467.215µs", "connect.code": "invalid_argument"}
kargo-api-7f7c599cdd-kbtct api 2026-04-17T21:50:16Z     ERROR   option/log.go:53        finished unary call: invalid_argument: Freight "47e9743bac25bb58097713fadc845e280b6d3740" is not available to Stage "aquila-secondary"  {"connect.service": "akuity.io.kargo.service.v1alpha1.KargoService", "connect.method": "PromoteToStage", "connect.start_time": "2026-04-17T21:50:16Z", "connect.duration": "183.544µs", "connect.code": "invalid_argument"}
kargo-api-7f7c599cdd-kbtct api 2026-04-17T21:50:22Z     ERROR   option/log.go:53        finished unary call: invalid_argument: Freight "47e9743bac25bb58097713fadc845e280b6d3740" is not available to Stage "aquila-secondary"  {"connect.service": "akuity.io.kargo.service.v1alpha1.KargoService", "connect.method": "PromoteToStage", "connect.start_time": "2026-04-17T21:50:22Z", "connect.duration": "1.575105ms", "connect.code": "invalid_argument"}

kargo-controller-6b6fb57c-wqnhl controller 2026-04-17T20:57:21Z ERROR   controller/controller.go:474    Reconciler error        {"controller": "stage", "controllerGroup": "kargo.akuity.io", "controllerKind": "Stage", "Stage": {"name":"aquila-secondary","namespace":"kargo-ccs"}, "namespace": "kargo-ccs", "name": "aquila-secondary", "reconcileID": "11826624-7660-4493-9cbd-55774e4ebba6", "error": "failed to auto-promote Freight: error creating Promotion for Freight \"47e9743bac25bb58097713fadc845e280b6d3740\" in namespace \"kargo-ccs\": admission webhook \"promotion.kargo.akuity.io\" denied the request: Promotion.kargo.akuity.io \"aquila-secondary.01kpekqrz584bqsejdvt6fzpw2.47e9743\" is invalid: spec.freight: Invalid value: \"47e9743bac25bb58097713fadc845e280b6d3740\": Freight is not available to this Stage"}
kargo-controller-6b6fb57c-wqnhl controller 2026-04-17T20:57:21Z ERROR   controller/controller.go:474    Reconciler error        {"controller": "stage", "controllerGroup": "kargo.akuity.io", "controllerKind": "Stage", "Stage": {"name":"aquila-secondary","namespace":"kargo-ccs"}, "namespace": "kargo-ccs", "name": "aquila-secondary", "reconcileID": "723ae9b9-405a-4207-8148-7d67d28aafc6", "error": "failed to auto-promote Freight: error creating Promotion for Freight \"47e9743bac25bb58097713fadc845e280b6d3740\" in namespace \"kargo-ccs\": admission webhook \"promotion.kargo.akuity.io\" denied the request: Promotion.kargo.akuity.io \"aquila-secondary.01kpekqrzvzdkmn5ebngm7wcc1.47e9743\" is invalid: spec.freight: Invalid value: \"47e9743bac25bb58097713fadc845e280b6d3740\": Freight is not available to this Stage"}
kargo-controller-6b6fb57c-wqnhl controller 2026-04-17T20:57:21Z ERROR   controller/controller.go:474    Reconciler error        {"controller": "stage", "controllerGroup": "kargo.akuity.io", "controllerKind": "Stage", "Stage": {"name":"aquila-secondary","namespace":"kargo-ccs"}, "namespace": "kargo-ccs", "name": "aquila-secondary", "reconcileID": "8654896b-733b-4156-aba4-0517f4878065", "error": "failed to auto-promote Freight: error creating Promotion for Freight \"47e9743bac25bb58097713fadc845e280b6d3740\" in namespace \"kargo-ccs\": admission webhook \"promotion.kargo.akuity.io\" denied the request: Promotion.kargo.akuity.io \"aquila-secondary.01kpekqs0rz11mznfw1vdnfspb.47e9743\" is invalid: spec.freight: Invalid value: \"47e9743bac25bb58097713fadc845e280b6d3740\": Freight is not available to this Stage"}

[scott-hpe]

I just ripped down my POC env and started fresh with Kargo 1.10.3. It seems to have the same issue.

1. Auto-promotion after soak to Pavo works
2. After manual promotion to Aquila, the Aquila-secondary stage starts up with reconciliation errors before the soak time expires.

Any hints on how to troubleshoot this?