Skip to content

fix(ci): evaluate deterministic hourly gates before NIM - #15

Open
seonghobae wants to merge 4 commits into
mainfrom
fix/hourly-deterministic-gates
Open

fix(ci): evaluate deterministic hourly gates before NIM#15
seonghobae wants to merge 4 commits into
mainfrom
fix/hourly-deterministic-gates

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Fleet incident

OriginWeave's protected-main Hourly Product Development workflow checked NVIDIA_NIM_API_KEY before the model-independent open-PR, release-blocker, and dry-run decisions. That makes deterministic governance depend on an optional live-model credential even when no model call should occur.

Repair

  • preserve Harden Runner egress-policy: block and the current exact endpoint allowlist;
  • keep api.github.com:443 as the GitHub API endpoint proven by the latest inspected OriginWeave protected-main run;
  • do not add cafe.github.com:443 without OriginWeave runner evidence requiring that alias;
  • evaluate open PR, release blocker, and dry-run gates before checking NVIDIA_NIM_API_KEY;
  • require the NVIDIA credential only after those deterministic stop gates permit the model-backed path;
  • add a repository contract that fixes the gate ordering and prevents broad *.github.com:443 egress.

TDD evidence

The first commit (73dd8a27a7e926b9059d77d3be85e7dcb63aa5e1) added the ordering/egress regression while the inherited workflow still checked nim_api_key_unavailable first. The production commit then made only the four-line gate-order move required by that contract.

Closure criteria

This PR is not operational closure by itself. Merge only after exact-head CI, security, review, branch protection, and qualifying independent non-author approval pass. After protected merge, a scheduled/manual Hourly Product Development run must demonstrate that an existing open PR exits through open_pull_request without requiring the NVIDIA credential and without GitHub API egress regression. A later zero-PR run must reach the model-backed path or an explicit deterministic product/release gate before this fleet path is considered fully closed.

No invented credential, no COPILOT_GITHUB_TOKEN, no broadening of fail-closed egress, and no temporary write-capable repair workflow.

@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@seonghobae, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 1 minute

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a152a00e-be1d-4e7e-bcac-d56a849794dd

📥 Commits

Reviewing files that changed from the base of the PR and between b228693 and 3cf49fd.

📒 Files selected for processing (3)
  • .github/workflows/hourly-product-development.yml
  • CHANGELOG.md
  • tests/test_hourly_product_development_contract.py

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@opencode-agent @cwl-noema-review

Review-only request for exact current head 3cf49fd8e849236ad131c08057a439ea4ccb74d0 against protected base b228693f9e0a11d1794c1e4b0f7a3a38559a5491. Do not mutate the branch or merge. Independently verify that fail-closed Harden Runner remains egress-policy: block with the exact reviewed endpoint set, allowed-endpoints remains the runtime-safe folded scalar, the actual gh api open-PR/release-blocker gates and dry-run decision execute before NVIDIA_NIM_API_KEY is required, the NVIDIA credential is required only for the model-backed path, no broad GitHub wildcard or invented credential/COPILOT token is introduced, exact-head CI/SAST/Security Scan are successful, and no unresolved valid review finding remains. Submit a formal non-author GitHub APPROVE review only if this exact head satisfies repository policy.

@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo="ContextualWisdomLab/OriginWeave"
pr_number="15"
expected_head="3cf49fd8e849236ad131c08057a439ea4ccb74d0"
expected_base="b228693f9e0a11d1794c1e4b0f7a3a38559a5491"

echo "== Checked-out commit =="
git rev-parse HEAD

echo "== Requested commits exist locally =="
git cat-file -t "$expected_head"
git cat-file -t "$expected_base"

echo "== PR metadata =="
gh pr view "$pr_number" --repo "$repo" \
  --json number,state,isDraft,headRefOid,baseRefOid,mergeCommit,statusCheckRollup,reviews \
  --jq '{
    number, state, isDraft, headRefOid, baseRefOid, mergeCommit,
    statusCheckRollup: [.statusCheckRollup[] | {
      name: .name,
      status: .status,
      conclusion: .conclusion,
      detailsUrl: .detailsUrl
    }],
    reviews: [.reviews[] | {
      author: .author.login,
      state: .state,
      commit: .commit.oid,
      submittedAt: .submittedAt
    }]
  }'

echo "== Changed files for exact requested range =="
git diff --name-status "$expected_base" "$expected_head"

echo "== Candidate workflow and contract files =="
fd -HI -t f .github . 2>/dev/null | \
  rg '(^|/)([^/]*[Ww]orkflow[^/]*|.*\.(ya?ml|md|json|sh))$' | \
  head -n 300

echo "== Gate and egress references in the exact requested range =="
git diff --unified=80 "$expected_base" "$expected_head" -- \
  ':!**/package-lock.json' ':!**/pnpm-lock.yaml' | \
  rg -n -C 8 \
    'egress-policy|allowed-endpoints|api\.github\.com|cafe\.github\.com|\*\.github\.com|NVIDIA_NIM_API_KEY|COPILOT_GITHUB_TOKEN|nim_api_key_unavailable|open_pull_request|release_blocker|dry.run|gh api' \
  || true

echo "== Exact-head check runs =="
gh api --paginate \
  "repos/$repo/commits/$expected_head/check-runs?per_page=100" \
  --jq '.check_runs[] | [
    .name, .status, .conclusion, .details_url,
    (.output.title // ""), (.output.summary // "")
  ] | `@tsv`'

echo "== Open review threads =="
gh api graphql -F owner="ContextualWisdomLab" -F name="OriginWeave" -F number="$pr_number" -f query='
query($owner: String!, $name: String!, $number: Int!) {
  repository(owner: $owner, name: $name) {
    pullRequest(number: $number) {
      reviewThreads(first: 100) {
        nodes {
          isResolved
          isOutdated
          comments(first: 20) {
            nodes {
              author { login }
              body
              commit { oid }
              path
              line
            }
          }
        }
      }
    }
  }
}' --jq '
  .data.repository.pullRequest.reviewThreads.nodes[]
  | select(.isResolved == false)
  | {
      isOutdated,
      comments: [
        .comments.nodes[] | {
          author: .author.login,
          body,
          commit: .commit.oid,
          path,
          line
        }
      ]
    }'

Length of output: 8589


🏁 Script executed:

#!/bin/bash
set -uo pipefail

repo="ContextualWisdomLab/OriginWeave"
pr_number="15"
head="3cf49fd8e849236ad131c08057a439ea4ccb74d0"
base="b228693f9e0a11d1794c1e4b0f7a3a38559a5491"
workflow=".github/workflows/hourly-product-development.yml"
contract="tests/test_hourly_product_development_contract.py"

echo "== Workflow structure =="
ast-grep outline "$workflow" --items all 2>&1 || true

echo "== Changed workflow =="
cat -n "$workflow"

echo "== Contract test structure =="
ast-grep outline "$contract" --items all 2>&1 || true

echo "== Contract test =="
cat -n "$contract"

echo "== Exact-range diff =="
git diff --check "$base" "$head"
git diff --unified=12 "$base" "$head" -- "$workflow" "$contract" CHANGELOG.md

echo "== Sensitive tokens in exact head =="
git grep -n -I -E \
  'COPILOT_GITHUB_TOKEN|NVIDIA_NIM_API_KEY|nim_api_key_unavailable|allowed-endpoints|egress-policy|cafe\.github\.com|\*\.github\.com|api\.github\.com|open_pull_request|release_blocker|dry_run|dry-run' \
  "$head" -- . ':!CHANGELOG.md' || true

echo "== Required GitHub API gate calls in workflow =="
rg -n -C 6 -P \
  '\bgh\s+api\b|NVIDIA_NIM_API_KEY|nim_api_key_unavailable|open_pull_request|release_blocker|dry.run|allowed-endpoints|egress-policy' \
  "$workflow" || true

echo "== Exact-head check-run conclusions =="
gh api --paginate "repos/$repo/commits/$head/check-runs?per_page=100" \
  --jq '.check_runs[] | select(.name != null) | [.name, .status, .conclusion, .details_url] | `@tsv`' \
  || true

echo "== Unresolved review threads =="
gh api graphql -F owner="ContextualWisdomLab" -F name="OriginWeave" -F number="$pr_number" -f query='
query($owner: String!, $name: String!, $number: Int!) {
  repository(owner: $owner, name: $name) {
    pullRequest(number: $number) {
      reviewThreads(first: 100) {
        nodes {
          isResolved
          isOutdated
          comments(first: 20) {
            nodes {
              author { login }
              body
              commit { oid }
              path
              line
            }
          }
        }
      }
    }
  }
}' --jq '
  [.data.repository.pullRequest.reviewThreads.nodes[]
   | select(.isResolved == false)
   | {
       isOutdated,
       comments: [
         .comments.nodes[] | {
           author: .author.login,
           body,
           commit: .commit.oid,
           path,
           line
         }
       ]
     }]'
echo "== Branch protection and rulesets =="
gh api "repos/$repo/branches/main/protection" --jq '{
  required_pull_request_reviews,
  required_status_checks,
  enforce_admins,
  restrictions
}' 2>&1 || true
gh api --paginate "repos/$repo/rulesets?includes_parents=true" \
  --jq '.[] | {id, name, target, enforcement, conditions, rules}' 2>&1 || true

Length of output: 62592


🏁 Script executed:

#!/bin/bash
set -uo pipefail

repo="ContextualWisdomLab/OriginWeave"
owner="ContextualWisdomLab"
name="OriginWeave"
pr_number="15"
head="3cf49fd8e849236ad131c08057a439ea4ccb74d0"
ruleset_id="18156473"

echo "== Pull request author and review decision =="
gh pr view "$pr_number" --repo "$repo" \
  --json author,reviewDecision,reviews,headRefOid,baseRefOid \
  --jq '{
    author: .author.login,
    reviewDecision,
    headRefOid,
    baseRefOid,
    reviews: [.reviews[] | {
      author: .author.login,
      state,
      commit: .commit.oid,
      submittedAt
    }]
  }' 2>&1 || true

echo "== Commit status contexts =="
gh api "repos/$repo/commits/$head/status" \
  --jq '{
    state,
    statuses: [.statuses[] | {
      context, state, target_url, description, created_at, updated_at
    }]
  }' 2>&1 || true

echo "== Active ruleset detail =="
gh api "repos/$repo/rulesets/$ruleset_id" \
  --jq '{
    id, name, target, enforcement, source_type, source,
    conditions, rules, bypass_actors
  }' 2>&1 || true

echo "== Repository merge-policy flags =="
gh api "repos/$repo" \
  --jq '{
    default_branch,
    allow_merge_commit,
    allow_squash_merge,
    allow_rebase_merge,
    allow_auto_merge,
    allow_update_branch,
    use_squash_pr_title_as_default,
    has_discussions
  }' 2>&1 || true

echo "== Exact head and range reconfirmation =="
printf 'checked_out='; git rev-parse HEAD
printf 'range_files=\n'
git diff --name-only \
  b228693f9e0a11d1794c1e4b0f7a3a38559a5491 \
  3cf49fd8e849236ad131c08057a439ea4ccb74d0

Length of output: 3993


@seonghobae Review complete for exact head 3cf49fd8e849236ad131c08057a439ea4ccb74d0 against base b228693f9e0a11d1794c1e4b0f7a3a38559a5491.

No valid unresolved finding was identified.

Verified:

  • Harden Runner keeps egress-policy: block.
  • allowed-endpoints remains a folded scalar with >-.
  • The endpoint set includes api.github.com:443.
  • The endpoint set does not include cafe.github.com:443 or *.github.com:443.
  • Open-PR, release-blocker, and dry-run gates execute before nim_api_key_unavailable.
  • NVIDIA_NIM_API_KEY is required only after deterministic stop gates permit the model-backed path.
  • The workflow does not introduce COPILOT_GITHUB_TOKEN or another model credential.
  • The added contract test protects both gate ordering and the exact endpoint set.
  • Exact-head CI, SAST, and security checks report success.
  • No unresolved review threads exist.
  • The active organization ruleset requires central workflows and resolved threads. It requires zero approving reviews.

I did not mutate the branch or merge the PR.

I did not submit a formal GitHub APPROVE review. Approval requires an explicit top-level @coderabbitai approve command. The post-merge scheduled/manual operational evidence remains required for fleet closure.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant