[codex] acknowledge terminal Gmail OAuth watch failures

[litang9]

Summary

• classify expired or revoked Gmail OAuth credentials as terminal processing failures
• persist a durable auth-recovery marker before acknowledging terminal Pub/Sub deliveries
• preserve the last successfully processed Gmail history cursor across watch start and watch renew
• use Gmail's immediate watch-registration notification to catch up after re-authentication
• keep transient Gmail API, rate-limit, persistence, and downstream hook failures retryable

Root cause

The Gmail watch consumers treated every non-rate-limit processing failure as retryable. When the OAuth refresh token returned invalid_grant, push delivery responded with HTTP 500 and pull delivery nacked the message. Pub/Sub then redelivered a failure that cannot recover without operator re-authentication, producing an unbounded retry and log storm.

Recovery contract

Terminal OAuth failures are acknowledged only after authRecoveryPending and the failure timestamp/reason are saved without advancing historyId or lastPushMessageId. A state write failure remains a normal processing error, so Pub/Sub keeps the delivery retryable.

After re-authenticating, run gog gmail watch renew or rerun gog gmail watch start. Successful registration preserves the old successful cursor while recovery is pending. Gmail sends an immediate notification for a successful watch registration, so the consumer catches up from that cursor without waiting for another mailbox change. Advancing history successfully clears the recovery marker. gog gmail watch status exposes the pending recovery state.

Temporary API failures, rate limits, and hook delivery failures retain their existing retry behavior.

Validation

• go test ./internal/gmailwatch
• focused internal/cmd tests covering start, renew, status, push acknowledgement, pull acknowledgement, persistence failure, and catch-up
• make fmt-check
• make lint
• make deadcode
• make build

The broader internal/cmd Gmail-watch test selection still has pre-existing shared-state failures when run as one package on this machine. The same failures reproduce on unmodified upstream/main; the focused regression tests above pass.

Redacted live Gmail/Pub/Sub evidence

Run on 2026-07-19 with commit 7b32e33be8a5. The run used:

• a real Gmail account and Gmail-generated history event
• a temporary real Pub/Sub pull subscription on the configured topic
• an isolated --home and encrypted file keyring
• a temporary Gmail label added to and then removed from one existing INBOX message
• the real watch start, pull acknowledgement, watch renew, immediate Gmail notification, history fetch, and hook delivery paths

Only the isolated refresh token was replaced with an invalid value. The production token was never revoked or modified.

runtime_binary=7b32e33be8a5
delivery=real_gmail_pubsub_pull
terminal_oauth=invalid_grant pull_acknowledged=yes
recovery_pending=true reason=reauthentication_required cursor_preserved=yes
watch_renewed=yes expiration_updated=yes cursor_preserved=yes
recovery_pending_after_renew=true
renew_notification=real_gmail_pubsub_pull
hook_received=true message_count=3 history_set=true
cursor_advanced=yes recovery_pending_after=false
cleanup=label_removed,label_deleted,subscription_deleted,isolated_credentials_deleted

The invalid-token pull logged Gmail authorization requires re-authentication; acknowledging pull without advancing history after Google OAuth returned invalid_grant. After restoring the isolated token, watch renew retained the old cursor and pending marker. Gmail's immediate registration notification arrived through the temporary Pub/Sub subscription; the consumer fetched the retained history, delivered three message records to a local 204 hook, advanced the cursor, cleared the marker, and acknowledged the notification.

Account addresses, project/topic/subscription names, history IDs, Pub/Sub message IDs, Gmail message IDs, label IDs, endpoints, and all credential material are omitted. Cleanup was verified: no temporary process, directory, subscription, label, or credential copy remains.

Related OpenClaw lifecycle fix: openclaw/openclaw#96202.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed July 27, 2026, 1:16 PM ET / 17:16 UTC.

ClawSweeper review

What this changes

The PR records terminal Gmail OAuth refresh failures as recovery state, acknowledges the affected Pub/Sub delivery without advancing the Gmail history cursor, and preserves that cursor across watch renewal for catch-up after reauthentication.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

This PR remains necessary: current main does not contain the terminal-OAuth classification or durable recovery-state path. The patch has focused coverage and credible redacted live evidence, but acknowledging a Pub/Sub delivery after a credential failure deliberately changes the recovery contract and needs maintainer acceptance before merge.

Priority: P1
Reviewed head: 647d762128c7bb249a4fc0571749da8abd6ece79
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	The patch is focused, covered across push, pull, persistence, registration, and state transitions, with strong real behavior evidence; the remaining blocker is maintainer acceptance of the delivery-policy tradeoff.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (live_output): The PR body contains redacted live Gmail/Pub/Sub pull evidence showing the after-fix acknowledgement, cursor preservation, post-reauth renewal, immediate catch-up notification, hook delivery, and recovery-marker clearing.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (live_output): The PR body contains redacted live Gmail/Pub/Sub pull evidence showing the after-fix acknowledgement, cursor preservation, post-reauth renewal, immediate catch-up notification, hook delivery, and recovery-marker clearing.
Evidence reviewed	5 items	Current-main gap: The current-main Gmail watch server delegates processing to the shared Gmail-watch handler, but the current internal/cmd/gmail_watch_server.go does not contain the PR's OAuth terminal-error classifier or its processor wiring. The central requested behavior is therefore not already implemented on main. Proposed recovery transition: The PR head adds TerminalAuthError, persists authRecoveryPending before returning that error, preserves the existing history cursor during registration, and clears the marker only after a later successful history advance. Delivery acknowledgement coverage: The PR adds explicit push and pull tests verifying that a terminal invalid_grant acknowledgement does not advance HistoryID or LastPushMessageID, while persistence failure remains retryable.
Findings	None	None.
Security	None	None.

How this fits together

Gmail watch receives Gmail/Pub/Sub notifications, fetches Gmail history, and sends resulting messages to an optional hook. Its persistent watch state tracks the last successful history cursor so retries, registrations, and recovery survive process restarts.

flowchart LR
  A[Gmail mailbox change] --> B[Pub/Sub notification]
  B --> C[Gmail watch consumer]
  C --> D[OAuth and Gmail history fetch]
  D --> E[Persistent watch state]
  D --> F[Configured hook delivery]
  E --> G[Retry or reauthentication recovery]
  F --> H[Downstream automation]

Loading

Decision needed

Question	Recommendation
Should Gmail watch acknowledge an unrecoverable OAuth refresh failure after durably recording recovery state, rather than continue Pub/Sub redelivery until an operator reauthenticates?	Accept durable recovery acknowledgement: Merge the proposed contract, using the preserved cursor and immediate watch-registration notification to catch up after reauthentication.

Why: This is an intentional delivery-policy tradeoff: it prevents unbounded retry storms but makes reauthentication plus watch renewal the required operator recovery path.

Before merge

• Resolve merge risk (P1) - Merging intentionally stops Pub/Sub redelivery for expired or revoked credentials; users must reauthenticate and run gog gmail watch renew or gog gmail watch start to resume catch-up.
• Resolve merge risk (P1) - The recovery guarantee depends on persisting the marker before acknowledgement and on Gmail history remaining available until reauthentication; the implementation and tests cover the persistence boundary, while the acceptable operator contract requires maintainer approval.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Recovery surface	19 files affected; 535 added, 20 removed	The change spans persistent state, push and pull delivery handling, command output, tests, and user documentation, so the delivery contract deserves an explicit maintainer decision.
Runtime validation	1 redacted real Gmail/Pub/Sub pull run	The PR body demonstrates acknowledgement, renewal, catch-up, hook delivery, and recovery-marker clearing in a live setup.

Merge-risk options

Maintainer options:

1. Approve the durable recovery contract (recommended)
   Accept terminal OAuth acknowledgement after durable recovery-state persistence, with the documented reauthenticate-and-renew workflow as the supported recovery path.
2. Pause for a different recovery policy
   Keep the PR unmerged if maintainers require Pub/Sub retries or another operator-facing recovery mechanism instead of terminal acknowledgement.

Technical review

Best possible solution:

Adopt the durable-marker acknowledgement contract if maintainers accept explicit reauthentication and watch renewal as the recovery path, retaining the documented cursor-preservation and retry-on-state-write-failure safeguards.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: an OAuth refresh returning invalid_grant enters the Gmail-watch processing path and is currently treated as a retryable failure on main. The PR also supplies a redacted live Gmail/Pub/Sub reproduction using an isolated invalid refresh token, but this review did not execute that account-bound flow.

Is this the best way to solve the issue?

Unclear: the implementation is a focused and well-tested solution to retry storms, but whether terminal credential failures should be acknowledged rather than redelivered is an operator-facing delivery-policy choice that maintainers must explicitly accept.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 83cc6dd34d8a.

Labels

Label justifications:

• P1: Terminal OAuth failures currently cause an unbounded Gmail/Pub/Sub retry path, and this PR changes recovery behavior for that active delivery workflow.
• merge-risk: 🚨 compatibility: Existing operators will move from automatic Pub/Sub redelivery to an explicit reauthentication and watch-renewal recovery procedure after terminal credential failure.
• merge-risk: 🚨 message-delivery: The PR deliberately acknowledges a terminal delivery before processing its notification, relying on the preserved Gmail history cursor and later watch renewal to recover messages.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body contains redacted live Gmail/Pub/Sub pull evidence showing the after-fix acknowledgement, cursor preservation, post-reauth renewal, immediate catch-up notification, hook delivery, and recovery-marker clearing.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body contains redacted live Gmail/Pub/Sub pull evidence showing the after-fix acknowledgement, cursor preservation, post-reauth renewal, immediate catch-up notification, hook delivery, and recovery-marker clearing.

Evidence

What I checked:

• Current-main gap: The current-main Gmail watch server delegates processing to the shared Gmail-watch handler, but the current internal/cmd/gmail_watch_server.go does not contain the PR's OAuth terminal-error classifier or its processor wiring. The central requested behavior is therefore not already implemented on main. (internal/cmd/gmail_watch_server.go:619, 83cc6dd34d8a)
• Proposed recovery transition: The PR head adds TerminalAuthError, persists authRecoveryPending before returning that error, preserves the existing history cursor during registration, and clears the marker only after a later successful history advance. (internal/gmailwatch/processor.go:176, 647d762128c7)
• Delivery acknowledgement coverage: The PR adds explicit push and pull tests verifying that a terminal invalid_grant acknowledgement does not advance HistoryID or LastPushMessageID, while persistence failure remains retryable. (internal/cmd/gmail_watch_pull_test.go:288, 647d762128c7)
• Real behavior proof: The PR body reports a redacted real Gmail account and Pub/Sub pull run: an isolated invalid token was acknowledged, the cursor was preserved, renewal generated an immediate notification, the retained history was delivered to a local hook, and successful processing cleared recovery state. (7b32e33be8a5)
• Recent ownership signal: The current PR head was force-pushed on July 27, 2026 and is authored by steipete with the terminal OAuth recovery implementation, indicating active ownership rather than an abandoned branch. (internal/gmailwatch/state.go:36, 647d762128c7)

Likely related people:

• steipete: Authored the current PR head implementing the recovery-state transition and force-pushed the active branch on July 27, 2026. (role: recent Gmail-watch contributor; confidence: medium; commits: 647d762128c7; files: internal/gmailwatch/processor.go, internal/gmailwatch/state.go, internal/cmd/gmail_watch_server.go)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Obtain explicit maintainer approval for the terminal-OAuth acknowledgement and reauthentication recovery contract.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (5 earlier review cycles)

• reviewed 2026-07-19T10:53:46.482Z sha 554028d :: needs real behavior proof before merge. :: [P1] Guarantee catch-up before acknowledging terminal OAuth failures
• reviewed 2026-07-19T11:35:15.643Z sha 554028d :: needs real behavior proof before merge. :: [P1] Guarantee catch-up before acknowledging terminal OAuth failures
• reviewed 2026-07-19T11:44:25.952Z sha 7b32e33 :: needs real behavior proof before merge. :: none
• reviewed 2026-07-19T12:26:10.550Z sha 7b32e33 :: needs real behavior proof before merge. :: none
• reviewed 2026-07-19T12:52:51.904Z sha 7b32e33 :: needs maintainer review before merge. :: none

[steipete]

Maintainer decision accepted: terminal Gmail OAuth failures are acknowledged only after the recovery marker is durably persisted, with the last successfully processed history cursor retained for catch-up after reauthentication.

Landed candidate: 647d762128c7bb249a4fc0571749da8abd6ece79

Proof on that exact candidate:

• go test ./internal/gmailwatch -count=1
• focused internal/cmd Gmail watch regression selection
• make fmt-check
• make test (all Go packages and 11 Node tests)
• local pinned lint: 0 issues
• AutoReview: clean, no accepted/actionable findings (confidence 0.99)
• built-CLI contract check: watch help exposes start/status/renew/serve/pull; an empty isolated home reports no stored watch state and creates none
• contributor live proof remains applicable to the unchanged runtime patch: real Gmail/Pub/Sub pull delivery acknowledged invalid_grant only after durable recovery state, preserved the cursor through renew, then caught up and cleared the marker after the immediate registration notification
• exact-head CI: https://github.com/openclaw/gogcli/actions/runs/30288279428
• exact-head Docker: https://github.com/openclaw/gogcli/actions/runs/30288279407

The commit preserves contributor credit with Co-authored-by: Alex Tang <alextangli@outlook.com> and the changelog thanks @litang9.