Daily AI Credits guardrail permanently fails open under org-level required workflows

[alvistar]

Summary

When an agentic workflow reaches a repository through a GitHub organization-level
required workflow (ruleset-injected), the daily AI Credits guardrail runs, 404s on
its own run-history query, and fails open — on every single run, permanently.

The guardrail is not being skipped by shouldSkipDailyAICGuardrail(); the event name
is pull_request, so that check correctly declines to short-circuit. The failure is
one call later: the run's workflow_id is not resolvable within the repository the
run executes in, so listWorkflowRuns returns 404, the top-level catch treats it as
a transient fault, and daily_ai_credits_exceeded stays at its default "false".

Net effect: the daily credit ceiling does not exist for this deployment model, and
the only signal is a warning worded to look like a passing network hiccup.

Impact

• The daily AIC ceiling never applies. Setting GH_AW_DEFAULT_MAX_DAILY_AI_CREDITS
  has no effect — the log below shows threshold: 5000 resolved correctly and then
  discarded when the query fails.
• No headroom telemetry is produced, so the summary table that would normally disclose
  24h spend, threshold used %, and remaining headroom is never written.
• A permanent, structural failure is reported through the channel reserved for
  transient ones, so it is indistinguishable from noise in run logs.
• This is precisely the deployment model that most needs a ceiling: one agentic review
  workflow rolled out across many repositories, where no single repository owner is
  watching aggregate spend.

The per-run cap (apiProxy.maxAiCredits) is unaffected — it is written into the
firewall config inside the agent job and does not depend on this query. So a single
run is still bounded, but the number of runs per day is not bounded at all.

Production evidence

Three consecutive runs, 2026-08-03 at 07:14, 07:25 and 07:36 UTC, byte-identical:

[daily-workflow-aic] Resolved current workflow AI Credits guardrail context:
  {"owner":"ORG","repo":"target-repo","currentRunId":30794198089,
   "workflowId":324645976,"workflowName":"PR Quality Review",
   "threshold":5000,"rateLimitRemaining":4992,"rateLimitLimit":5000}
[daily-workflow-aic] Querying completed workflow runs:
  {"workflowId":324645976,"page":1,"perPage":100,"cutoff":"2026-08-02T07:36:35.516Z"}
##[warning]Daily workflow AI Credits guardrail encountered an unexpected error
           and will be skipped: Not Found -
           https://docs.github.com/rest/actions/workflow-runs#list-workflow-runs-for-a-workflow

Why the 404 — the workflow is not a member of the repository's workflow collection:

$ gh api /repos/ORG/target-repo/contents/.github/workflows
ci.yml
release.yml

$ gh api /repos/ORG/target-repo/actions/workflows
318081782  CI       .github/workflows/ci.yml   [active]
318081783  Release  .github/workflows/release.yml  [active]

The repository contains no agentic workflow file and lists two workflows, yet the runs
execute under workflow_id: 324645976. That id resolves in neither the target
repository nor the repository that defines the workflow — which is characteristic of
ruleset-injected required workflows: they execute in the target repo's context
without being members of its workflow collection.

Root cause

actions/setup/js/check_daily_aic_workflow_guardrail.cjs, in main():

const { owner, repo } = context.repo;            // the repo the run executes in

const currentRun = await githubClient.rest.actions.getWorkflowRun({
  owner, repo, run_id: context.runId,            // OK — the RUN is in this repo
});

// ...

const response = await githubClient.rest.actions.listWorkflowRuns({
  owner, repo,                                   // this repo...
  workflow_id: currentRun.data.workflow_id,      // ...but a workflow that is not in it
  status: "completed", per_page: 100, page,
});                                              // -> 404 Not Found

Swallowed by the top-level handler:

} catch (error) {
  // Treat unexpected guardrail execution errors as non-blocking skips so transient
  // API/runtime issues do not fail activation. The output stays at the default "false",
  // allowing the agent to run. Legitimate threshold exceedance still fails via setFailed.
  core.warning(`Daily workflow AI Credits guardrail encountered an unexpected error and will be skipped: ${getErrorMessage(error)}`);
}

That comment describes a sound policy for a transient fault. A 404 here is neither
transient nor unexpected — it will recur on every run for the lifetime of the
deployment.

Reproduction

Attached test file: actions/setup/js/check_daily_aic_workflow_guardrail.required_workflow.test.cjs

$ cd actions/setup/js && npx vitest run check_daily_aic_workflow_guardrail

 Test Files  1 failed | 1 passed (2)
      Tests  1 failed | 37 passed (38)

Three cases. The existing 35 tests are untouched and still pass.

Test	State	Establishes
REPRO: a permanent 404 on run history silently disables the daily ceiling	passes	the bug
CONTRAST: a transient 500 should fail open	passes	fail-open on real transients is correct and must be preserved
REGRESSION: a permanent 404 must be distinguishable from a transient failure	fails	the defect

The mock encodes the exact asymmetry that required workflows create:

// Succeeds: the RUN does live in this repo.
getWorkflowRun: async () => ({ data: { workflow_id: 324645976, ... } }),
// Fails: the WORKFLOW does not live in this repo.
listWorkflowRuns: async () => { throw httpError(404, "Not Found"); },

The repro runs with GH_AW_MAX_DAILY_AI_CREDITS = "10" — a deliberately absurd
ceiling — and still asserts daily_ai_credits_exceeded === "false" with setFailed
never called. A limit of ten credits cannot fire.

Note: GITHUB_EVENT_NAME is set to pull_request in the test, matching production.
shouldSkipDailyAICGuardrail() correctly does not short-circuit; reaching the
catch proves the guardrail genuinely ran and then failed.

One caveat in the interest of accuracy: the 404 in the mock is synthesized from the
production log message rather than captured from a live Octokit rejection. The status
code and message match what GitHub returned, and the code path only reads the message
via getErrorMessage, but a recorded fixture would be strictly more faithful.

Full test file — actions/setup/js/check_daily_aic_workflow_guardrail.required_workflow.test.cjs

import { beforeEach, describe, expect, it, vi } from "vitest";

/**
 * Repro + regression coverage for the daily AI Credits guardrail under
 * GitHub organization-level *required workflows*.
 *
 * Background
 * ----------
 * A required workflow is injected by an org ruleset. It EXECUTES in the target
 * repository's context, but it is NOT a member of that repository's workflow
 * collection. Observed in production:
 *
 *   $ gh api /repos/OWNER/REPO/actions/workflows
 *   318081782  CI       .github/workflows/ci.yml
 *   318081783  Release  .github/workflows/release.yml
 *
 * ...yet review runs execute under workflow_id 324645976, which is in neither
 * that list nor the defining repo's list.
 *
 * main() reads the workflow id off the current run and then queries run history
 * scoped to context.repo:
 *
 *   const { owner, repo } = context.repo;
 *   const currentRun = await ...getWorkflowRun({ owner, repo, run_id: context.runId });
 *   const response  = await ...listWorkflowRuns({ owner, repo,
 *                            workflow_id: currentRun.data.workflow_id, ... });
 *
 * The second call 404s permanently, the top-level catch treats it as transient,
 * and daily_ai_credits_exceeded stays "false" -- so the ceiling never applies.
 *
 * Real log line, identical on three consecutive runs (2026-08-03 07:14/07:25/07:36 UTC):
 *
 *   [daily-workflow-aic] Querying completed workflow runs: {"workflowId":324645976,...}
 *   ##[warning]Daily workflow AI Credits guardrail encountered an unexpected
 *              error and will be skipped: Not Found
 */

let exports;

/** Octokit surfaces HTTP failures as an error carrying a numeric `status`. */
function httpError(status, message) {
  const err = new Error(message);
  err.status = status;
  return err;
}

function makeCore() {
  const outputs = {};
  const warnings = [];
  const errors = [];
  return {
    outputs,
    warnings,
    errors,
    failures: [],
    core: {
      setOutput: (k, v) => {
        outputs[k] = v;
      },
      info: () => {},
      warning: m => warnings.push(String(m)),
      error: m => errors.push(String(m)),
      setFailed: function (m) {
        this._failures.push(String(m));
      },
      _failures: [],
      summary: {
        addDetails: function () {
          return this;
        },
        addRaw: function () {
          return this;
        },
        write: async () => {},
      },
    },
  };
}

/**
 * A guardrail run where the workflow executes in the repo but its id is not
 * resolvable there -- i.e. an org-level required workflow.
 *
 * @param {Error} listError error thrown by listWorkflowRuns
 */
function makeGithub(listError) {
  return {
    rest: {
      rateLimit: {
        get: async () => ({
          data: {
            resources: {
              core: {
                limit: 5000,
                remaining: 4992,
                used: 8,
                reset: Math.floor(Date.UTC(2026, 7, 3, 9) / 1000),
              },
            },
          },
          headers: {},
        }),
      },
      actions: {
        // Succeeds: the RUN does live in this repo.
        getWorkflowRun: async () => ({
          data: {
            workflow_id: 324645976,
            actor: { login: "octocat" },
            triggering_actor: { login: "octocat" },
          },
          headers: {},
        }),
        // Fails: the WORKFLOW does not live in this repo.
        listWorkflowRuns: async () => {
          throw listError;
        },
      },
    },
  };
}

describe("daily AIC guardrail under org-level required workflows", () => {
  beforeEach(async () => {
    vi.resetModules();
    // The production event name really is pull_request: the required-workflow
    // stub triggers on pull_request, so shouldSkipDailyAICGuardrail() does NOT
    // short-circuit. The guardrail genuinely runs and then fails.
    process.env.GITHUB_EVENT_NAME = "pull_request";
    process.env.GH_AW_WORKFLOW_DISPATCH_AW_CONTEXT = "";
    process.env.GH_AW_HAS_SLASH_COMMAND = "false";
    process.env.GH_AW_HAS_LABEL_COMMAND = "false";
    process.env.GH_AW_GITHUB_TOKEN = "fake-token";
    process.env.GH_AW_WORKFLOW_NAME = "PR Quality Review";
    // A deliberately tight ceiling: if the guardrail could evaluate at all,
    // any real spend would breach it.
    process.env.GH_AW_MAX_DAILY_AI_CREDITS = "10";
    const mod = await import("./check_daily_aic_workflow_guardrail.cjs");
    exports = mod.default || mod;
  });

  async function runGuardrail(listError) {
    const h = makeCore();
    global.core = h.core;
    global.github = makeGithub(listError);
    global.context = { repo: { owner: "test-org", repo: "target-repo" }, runId: 30794198089 };
    try {
      await expect(exports.main()).resolves.toBeUndefined();
    } finally {
      delete global.core;
      delete global.github;
      delete global.context;
    }
    return h;
  }

  // ---------------------------------------------------------------- repro ---
  // PASSES on current code. This is the bug, not the fix.
  it("REPRO: a permanent 404 on run history silently disables the daily ceiling", async () => {
    const h = await runGuardrail(httpError(404, "Not Found"));

    // The agent is allowed to run...
    expect(h.outputs["daily_ai_credits_exceeded"]).toBe("false");
    // ...and nothing failed the step, despite a ceiling of 10 AIC.
    expect(h.core._failures).toEqual([]);
    // The only trace is the generic transient-error warning.
    expect(h.warnings.some(w => /unexpected error.*skipped/i.test(w))).toBe(true);
  });

  // PASSES on current code. Establishes that fail-open on a genuine transient
  // fault is correct and must be preserved by any fix.
  it("CONTRAST: a transient 500 should fail open (existing, desired behaviour)", async () => {
    const h = await runGuardrail(httpError(500, "Internal Server Error"));

    expect(h.outputs["daily_ai_credits_exceeded"]).toBe("false");
    expect(h.warnings.some(w => /unexpected error.*skipped/i.test(w))).toBe(true);
  });

  // ----------------------------------------------------------- regression ---
  // FAILS on current code. Passes once a permanent, structural failure is
  // reported distinguishably from a transient one.
  //
  // The assertion is deliberately about the OBSERVABLE CONTRACT, not a specific
  // implementation: "the guardrail could not evaluate" must not be reported
  // through the same channel, and with the same output, as "the guardrail
  // evaluated and you are under budget". Any distinguishable signal satisfies
  // this -- core.error, a dedicated status output, or a non-"false" value.
  it("REGRESSION: a permanent 404 must be distinguishable from a transient failure", async () => {
    const structural = await runGuardrail(httpError(404, "Not Found"));
    const transient = await runGuardrail(httpError(500, "Internal Server Error"));

    const structuralSignal = JSON.stringify({
      errors: structural.errors,
      outputs: structural.outputs,
    });
    const transientSignal = JSON.stringify({
      errors: transient.errors,
      outputs: transient.outputs,
    });

    expect(structuralSignal).not.toEqual(transientSignal);
  });
});

Suggested fix

Two parts, the second mattering more than the first.

1. Resolve run history in a way that survives required-workflow injection. When the
   workflow_id lookup 404s, fall back to listing the repository's runs and filtering
   by workflow name (already available as GH_AW_WORKFLOW_NAME).

2. Stop reporting a permanent failure as a transient one. A structural error that
   disables a spend guard for the lifetime of a deployment should be distinguishable
   from a network blip — core.error, a dedicated status output, or any signal that
   separates "could not evaluate" from "evaluated, under budget". Today those two
   states are byte-identical to a consumer, which is what turns this from a bug into an
   unbounded bill.

The attached regression test asserts only the contract in (2), not any particular
implementation, so it should pass under whichever mechanism you prefer.

Environment

• gh-aw v0.83.4 (compiler + github/gh-aw-actions/setup@v0.83.4) in production
• Reproduced against main at bd16174
• Engine: codex / gpt-5.6-sol; self-hosted runners
• Distribution: org ruleset injects a stub that calls the compiled workflow via
  workflow_call; target repositories contain no agentic workflow files