sec: require authenticated invocation for gmail-scraper Cloud Run service

[avireddy0]

Security fix — 2026-08-02 AGENTS.md coverage sweep finding (tier 1)

Finding: the gmail-scraper Cloud Run service is deployed --allow-unauthenticated. A POST to the public URL triggers a domain-wide-delegation Gmail scrape of every Workspace user into BigQuery gmail_analytics.messages (or an AlloyDB backfill via {"action": "backfill"}). Anyone holding the URL can invoke it. This is live in production right now.

This PR requires authentication on the service without breaking the 5-minute Cloud Scheduler cadence, per the sweep's remediation directive.

What actually blocked the fix — the pipeline has been dead since March

Deploy history (deploy.yml runs): the last green deploy is run 12, 2026-03-26 (dee6b4e). Every push since (runs 13–16, incl. PR #4, PR #5, and today's AGENTS.md commit) failed at workflow startup with:

The actions actions/checkout@v4, google-github-actions/auth@v2, and google-github-actions/setup-gcloud@v2 are not allowed in Envision-Construction/gmail because all actions must be pinned to a full-length commit SHA.

So production today = the run-12 deploy: public service, plus the tokenless gmail-scraper-5min job created by the old setup_scheduler.sh (still firing every 5 minutes — later failed runs never touched it). The PR #4 Cloud Build routing (cloudbuild.yaml) has never executed successfully — and it carries the vulnerable --allow-unauthenticated plus a regression of the dee6b4e fix (an invalid --attempt-deadline=3600s; Cloud Scheduler HTTP max is 1800s, so its create step would delete the job and then fail to recreate it).

Changes

1. .github/workflows/deploy.yml — restore the proven run-12 direct-deploy path, hardened:
   • All actions pinned to full-length commit SHAs (org policy): checkout v4.4.0, auth v2.1.13, setup-gcloud v2.2.1, deploy-cloudrun v2.7.6 — SHAs verified against upstream tags via git ls-remote.
   • Cloud Run deploy now passes --no-allow-unauthenticated (removes the allUsers → roles/run.invoker binding).
   • Grants roles/run.invoker on the service to claude-service-account@claude-mcp-457317.iam.gserviceaccount.com (the scheduler's OIDC identity).
   • Tripwire step: deploy fails if a public binding (allUsers / allAuthenticatedUsers) is ever present — the exact regression class that resurrected this flag once already.
   • Scheduler job now sends OIDC tokens (--oidc-service-account-email, audience = service URL) with the legal 1800s attempt-deadline. The step never deletes the job: it guards against an empty deploy-URL output, then updates-in-place (or creates only if missing). Any failure — earlier step, permissions, API error — leaves the previous job running, so the 5-minute cadence cannot be stranded.
   • concurrency group (no cancel-in-progress) so racing pushes can't interleave scheduler mutations; persist-credentials: false on checkout.
2. cloudbuild.yaml deleted — ⚠️ this reverts PR chore: align CI with Cloud Build + rewrite README_CLOUDRUN #4's deploy routing. Rationale: that path never ran green, its in-build IAM prerequisites are unproven (first-ever builds submit would gate the security fix on unknown permissions), and every permission the restored direct path needs was exercised successfully by run 12 under the same WIF identity. If Cloud Build routing is still wanted, it should come back as its own change with the IAM verified — say the word and I'll prep it.
3. Docs aligned (AGENTS.md, README.md, README_CLOUDRUN.md, deploy.sh): authenticated-invocation posture, OIDC scheduler, SHA-pin policy, removal of stale setup_scheduler.sh references, removal of the false "secrets are read at runtime from Secret Manager" claim, and authenticated curl examples.

Secondary finding — verified, deliberately NOT changed here

The sweep asked to confirm how production authenticates the Gmail scrape before touching it. Confirmed from repo + deploy evidence:

• gmail_scraper.py builds credentials only via from_service_account_file(SERVICE_ACCOUNT_FILE) (default ./service-account-key.json) — no ADC fallback, no Secret Manager integration (the old deploy.sh claim was false).
• The image (multi-stage since f35498a) copies only main.py + gmail_scraper.py; CI builds from a WIF checkout that cannot contain a key; the deploy sets no SERVICE_ACCOUNT_FILE and mounts no secret.
• Therefore every scheduled POST since 2026-03-26 has returned {"status": "failed"} (HTTP 200) without scraping. Corroborating: the Envision-MCP gateway's gmail-analytics tools were repointed away from this table (Phase 268), and the AlloyDB gmail_messages table the PR feat: AlloyDB dual-write for gmail_messages #5 dual-write would have created does not exist.
• This PR does not touch the runtime auth path (gmail_scraper.py, main.py, Dockerfile unchanged), and deploy-cloudrun's default env-var merge strategy plus untouched volumes mean any out-of-band runtime config would survive this deploy. Fixing the scrape (keyless DWD via IAM signJwt, or a Secret Manager mount) is an owner decision with Workspace-admin implications — documented as a Known Issue in README_CLOUDRUN.md.

Deploy behavior on merge (heads-up)

• This will be the first successful deploy since March: it ships everything on main since dee6b4e, including PR feat: AlloyDB dual-write for gmail_messages #5's AlloyDB dual-write (inert — ALLOYDB_URL is not set) and the AGENTS.md docs.
• Expect a ≤1–2 tick 403 window: the service flips to authenticated a few minutes before the scheduler job gains its OIDC token. No data impact (the scrape path is already non-functional, see above).
• secrets.WIF_PROVIDER was last exercised in run 12; if it has been rotated since, the run fails at the auth step with nothing mutated.
• First scheduler update exercises iam.serviceAccounts.actAs on the OIDC SA (an SA does not implicitly hold actAs on itself). Run 12's deploy (--service-account=<self> under the same identity) exercised the same permission successfully; if it has since been revoked, the update fails leaving the old job intact — red run; the legacy tokenless job keeps firing and 403s against the now-authenticated service (harmless — the scrape path is already non-functional) until the grant is fixed and the workflow re-run. Optional pre-merge check: confirm roles/iam.serviceAccountUser for the SA on itself (or project-level).

Post-merge verification

gcloud run services get-iam-policy gmail-scraper --project=claude-mcp-457317 --region=us-central1
# expect: run.invoker only for serviceAccount:claude-service-account@…; NO allUsers/allAuthenticatedUsers

gcloud scheduler jobs describe gmail-scraper-5min --project=claude-mcp-457317 --location=us-central1
# expect: schedule "*/5 * * * *", attemptDeadline 1800s, oidcToken.serviceAccountEmail=claude-service-account@…

curl -s -o /dev/null -w '%{http_code}\n' -X POST "$SERVICE_URL/"    # expect 403

Security review

Ran the security-reviewer agent pattern (global/rules/security.md) over the diff: initial verdict BLOCK (HIGH: delete-before-create could strand the scheduler job; MED: tripwire missed allAuthenticatedUsers; MED: README overclaimed invoker exclusivity; LOW: unguarded delete masking, missing concurrency group). All findings fixed as above; scoped re-review passed (APPROVE-WITH-NITS, nits applied). Checked per the pre-commit checklist: no secrets in the diff, no ${{ }} interpolation into run: blocks (URL flows through step env:), runtime code untouched, prod env vars byte-identical.

Follow-ups (out of scope here)

• Dedicated gmail-scraper-invoker@ SA for the scheduler (shrinks the invoke surface off the shared god-SA that also holds BigQuery write + domain-wide delegation).
• Fix runtime Gmail auth (keyless DWD or Secret Manager mount) — restores actual scraping; owner decision.
• main.py returns HTTP 200 on status: failed, so Scheduler counts dead scrapes as successes — worth flipping to 5xx for alerting.
• Sweep other repos for allow-unauthenticated (the "review for similar issues" step) — this pattern reached production once, and AGENTS.md here even documented it as intended state.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpPDLUg5HWuvS7FrfZFdjG

---

Generated by Claude Code

---

Note

High Risk
Changes production Cloud Run IAM and the only deploy pipeline; misconfiguration could block Scheduler or briefly leave public invoke until OIDC is applied, though tripwire and in-place scheduler updates reduce blast radius.

Overview
Locks down production by deploying gmail-scraper with --no-allow-unauthenticated, granting roles/run.invoker to the runtime service account for Cloud Scheduler, and failing the workflow if allUsers or allAuthenticatedUsers bindings appear on the service.

The deploy path is restored from the broken Cloud Build route: direct source deploy via pinned GitHub Actions (org policy requires full commit SHAs), plus serialized workflow concurrency so scheduler updates don’t race. The gmail-scraper-5min job is updated in place (or created if missing) with OIDC (--oidc-service-account-email, audience = service URL) and a 1800s attempt deadline—no delete-before-create.

cloudbuild.yaml is removed (the PR #4 path that never ran green and still allowed unauthenticated deploy). Docs (AGENTS.md, READMEs, deploy.sh) now describe authenticated invocation, OIDC scheduler, SHA pinning, and the separate known issue that scrape POSTs still fail at runtime without a key file (not changed in this PR).

^{Reviewed by Cursor Bugbot for commit 0d520a6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Hardens the gmail-scraper ingestion service’s production deployment by requiring authenticated Cloud Run invocation and ensuring Cloud Scheduler continues to trigger the job via OIDC, while also restoring a working deploy pipeline compliant with the org’s SHA-pinning policy.

Changes:

• Updates the GitHub Actions deploy workflow to use SHA-pinned actions, deploy Cloud Run with --no-allow-unauthenticated, add a non-public IAM “tripwire”, and update/create the Scheduler job in-place with OIDC.
• Removes the unused/broken Cloud Build deployment route (cloudbuild.yaml).
• Aligns docs (AGENTS.md, README*, deploy.sh) with the authenticated invocation + OIDC scheduler posture and documents the known runtime credential issue.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
.github/workflows/deploy.yml	Restores direct Cloud Run deploy (SHA-pinned), removes unauth invocation, grants invoker, enforces “no public principals”, and updates Scheduler job with OIDC.
cloudbuild.yaml	Deleted to remove the unproven Cloud Build routing path.
README.md	Updates top-level deploy/invocation description to reflect authenticated invocation + OIDC scheduler.
README_CLOUDRUN.md	Expands deploy/security posture docs and adds “known issue” note about runtime Gmail credentials.
deploy.sh	Updates informational deployment notes and adds an authenticated invocation example.
AGENTS.md	Updates repo agent guidance to reflect authenticated invocation + in-place scheduler updates and removal of Cloud Build path.

Suppressed comments (1)

README_CLOUDRUN.md:123

• Same as the GET example: for Cloud Run IAM, the identity token should be minted with an audience matching the service URL, otherwise authorized callers may still get 401/403 due to audience mismatch.

curl -X POST https://YOUR-SERVICE-URL/ \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  -H 'Content-Type: application/json' \
  -d '{"incremental": true, "max_per_user": 100}'

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.