!terraform.state in catalog causes plan-diff failure in github-action-atmos-terraform-apply@v7

[Juan A. (jaguer0)]

We're running into an issue with !terraform.state in catalog defaults when using the Atmos Pro managed GitHub Actions (apply). Looking for guidance on whether migrating to native CI is the right fix.

Environment

• Ref arch quickstart
• Atmos v1.208.0
• Atmos Pro with cloudposse/github-action-atmos-terraform-plan@v5 / apply@v7.0.0
• Atmos Auth with GitHub OIDC profiles (github-plan, github-apply)
• runs-on self-hosted EC2 runners

What happens
Our component uses !terraform.state in catalog defaults:

  components:
    terraform:
      honeycomb-otlp-collector:
        vars:
          vpc_id: !terraform.state vpc vpc_id
          ecs_cluster_name: !terraform.state ecs/cluster cluster_name
          alb_listener_arn: !terraform.state ecs/cluster .alb.private.https_listener_arn

• atmos tf plan succeeds — Atmos Auth assumes the per-stack identity (plat-dev/terraform via OIDC), resolves all lookups, produces a valid plan.
• plan-diff fails — the post-plan diff step inside github-action-atmos-terraform-apply@v7.0.0 resolves the same !terraform.state expressions but runs under the runner's EC2 instance role, not the Atmos Auth identity.

  User: arn:aws:sts::111111111111:assumed-role/acme-core-use2-auto-runs-on-EC2InstanceRole/i-0af140e378470b994
  is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/acme-core-gbl-root-tfstate

  Error: error processing stacks for plan-diff: failed to read Terraform state
  for component ecs/cluster in stack plat-use2-dev in YAML function:
  !terraform.state ecs/cluster cluster_name

The runner (111111111111, core-auto account) can't assume the tfstate role in the root account (222222222222) to read the S3 state backend. The plan step worked because Atmos Auth handled it, but plan-diff runs outside of Atmos Auth.

Workaround

Applying via CLI locally works fine — Atmos Auth handles identity correctly throughout the entire lifecycle.

Questions

1. Is native CI + Atmos 1.211.0 the fix? We read the native CI announcement. Since atmos terraform deploy --ci runs the entire lifecycle as a single CLI invocation, we assume plan-diff would run under the same Atmos Auth identity as the plan. Is that correct?
2. Atmos Pro compatibility? We're using Atmos Pro with the orchestration flow (atmos-pro.yaml → dispatch → plan/apply workflows). If we replace the managed actions with native CI calls, is there anything extra needed to maintain Pro integration (run tracking, workspace sync, ATMOS_PRO_WORKSPACE_ID)? The native CI blog doesn't mention Pro. - or any migration GHA we should use instead

[Dan Miller (milldr)]

This looks like a bug in plan-diff — it should assume the same identity as the plan step. I'll raise that internally, but in the meantime, native CI is the right path forward and is fairly straightforward to adopt.

To enable native CI:

1. Add the ci settings to your atmos.yaml
2. Replace your three Atmos GitHub Action workflows with these:

• atmos-pro.yaml
• atmos-terraform-plan.yaml
• atmos-terraform-apply.yaml

A few things to keep in mind:

• Workflow names must match exactly what's specified in your Pro settings (likely under stacks/orgs/acme/_defaults.yaml).
• Atmos Pro always dispatches workflows from the default branch, so merge these to main before testing.

On Atmos Pro compatibility: these workflows are fully compatible. Atmos Pro doesn't care what's inside the dispatched workflows — but there are two things required to keep Pro integration working: include --upload on describe affected to trigger Pro, and include --upload-status on plan and apply to report results back (beyond a simple "completed" status).

We'll get documentation out for this soon. Let me know if it works!

[Dan Miller (milldr)]

Toolchain is supposed to install tofu for you, which should just require the .tool-versions file. Try adding this step:

      - name: Install Toolchain
        shell: bash
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          atmos toolchain install
          atmos toolchain env --format github

[Juan A. (jaguer0)]

Dan Miller (@milldr) thanks , I'll give that a try.

I saw this p.r. earlier so didn't think to add it,

Latest Atmos versions handle toolchain installation automatically, so the
explicit `atmos toolchain install` and `atmos toolchain env` steps are no
longer needed in the CI workflows.

I'll let you know if that fixes it 🤞

[Juan A. (jaguer0)]

looks like it got further, however I hit this back to back now. The plan is successful but action fails with

 ERRO  Pro API Error operation=UploadInstanceStatus request="" status=500 success=false trace_id=1ce73517530576b5d889812eff75fae9 error_message="" context=map[]
🧪 ci is an experimental feature. Learn more https://atmos.tools/experimental
# Error
**Error:** failed to upload instance status failed to upload instance status API
response error: API request failed with status 500 (trace_id:
1ce73517530576b5d889812eff75fae9)
Error: Process completed with exit code 1.

I hit this earlier this a.m as well but a retry fixed it, I'll keep triggering it to see if I can get it to complete

[Dan Miller (milldr)]

I checked the trace ID in logs and found the issue. This error means that this specific instance does not exist yet on Atmos Pro server. Was this a new component?

We have a release going out today to fix that, but in the meanwhile, rerun list instances to upload the new component. You can run that from the branch of the PR with workflow dispatch

[Juan A. (jaguer0)]

I kicked off the list instance for both the PR branch and main, but the same issue persists.
For context, this is an existing stack — I only updated it to Valkey.

I verified in the Atmos Pro dashboard that both atmos plan and atmos apply completed successfully prior to this, FWIW.

I also kicked off a drift detection in Atmos Pro — the majority of runs failed with similar API errors (either 500 or 401), even though the underlying plan was successful with no changes.

For example, this 401 was for atmos terraform plan vpc -s plat-use2-sandbox, which I know is already deployed:

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

ERRO  Pro API Error operation=UploadInstanceStatus request="" status=401 success=false trace_id=660d9b4e66e3fe4c2ad2cc54eb10d74e error_message="" context=map[]

🧪 ci is an experimental feature. Learn more https://atmos.tools/experimental

# Error
**Error:** failed to upload instance status failed to upload instance status API
response error: API request failed with status 401 (trace_id:
660d9b4e66e3fe4c2ad2cc54eb10d74e)

So it looks like the plans themselves are succeeding, but the Atmos Pro API calls to upload status are failing.

I don't recall seeing these prior to updating from atmos v1.208.0 -> 1.211.0 & doing the native CI change this morning, I did both of those ~3 hours ago FWIW

[Juan A. (jaguer0)]

Thanks Dan Miller (@milldr)! Implemented the native CI migration following your guidance.

Here's what we changed:

1. Added ci: section to atmos.yaml

ci:
  enabled: true
  output:
    enabled: true
  summary:
    enabled: true
  checks:
    enabled: true
    context_prefix: ""
    statuses:
      component: true
      add: true
      change: true
      destroy: true

2. Replaced atmos-pro.yaml (affected stacks)

Swapped cloudposse/github-action-atmos-affected-stacks@v6 for native CLI:

- name: Install Atmos
  uses: cloudposse/github-action-setup-atmos@v3
  with:
    install-wrapper: false
    atmos-version: ${{ vars.ATMOS_VERSION }}

- name: Determine Affected Stacks
  shell: bash
  env:
    ATMOS_PRO_WORKSPACE_ID: ${{ vars.ATMOS_PRO_WORKSPACE_ID }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    ATMOS_PROFILE: "github-plan"
  run: |
    atmos describe affected \\
    --process-functions=false \\
    --include-settings=false \\
    --skip=terraform.output \\
    --upload \\
    --identity=core-auto/terraform \\
    --sha ${{ ... }}

3. Replaced plan workflow

Swapped cloudposse/github-action-atmos-terraform-plan@v5 for:

- name: Plan Atmos Component
  shell: bash
  env:
    ATMOS_PRO_WORKSPACE_ID: ${{ vars.ATMOS_PRO_WORKSPACE_ID }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    ATMOS_PROFILE: "github-plan"
    GITHUB_SHA: ${{ inputs.sha }}
  run: |
    atmos terraform plan ${{ inputs.component }} -s ${{ inputs.stack }} --upload-status

4. Replaced apply workflow

Swapped cloudposse/github-action-atmos-terraform-apply@v7.0.0 for:

- name: Apply Atmos Component
  shell: bash
  env:
    ATMOS_PRO_WORKSPACE_ID: ${{ vars.ATMOS_PRO_WORKSPACE_ID }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    ATMOS_PROFILE: "github-apply"
    GITHUB_SHA: ${{ inputs.sha }}
  run: |
    atmos terraform deploy ${{ inputs.component }} -s ${{ inputs.stack }} --upload-status

Key adaptations for our setup:

• Kept separate github-plan / github-apply profiles (planner vs terraform IAM roles)
• Kept workflow filenames matching our _defaults.yaml Pro settings (atmos-pro-terraform-plan.yaml, atmos-pro-terraform-apply.yaml)
• Used --identity=core-auto/terraform for the affected stacks step
• Added checks: write and statuses: write permissions for CI features

Will report back after merging and testing with a !terraform.state component.

[Erik Osterman (Cloud Posse) (osterman)]

Great to hear!

[Erik Osterman (Cloud Posse) (osterman)]

Juan A. (@jaguer0) reach out via email if you need any assistance and we'll be happy to schedule complementary assistance for Native CI onboarding, if that would be helpful.

[Juan A. (jaguer0)]

Erik Osterman (Cloud Posse) (@osterman) Yes! Thank you . I'll def take you up on that for the native CI. I'll send over an email