fix(deploy): retry shadow-gate OSS readback on transient publication-race 404

[proerror77]

Change contract

Shadow-gate OSS triplet readback retries each download up to 6× with 10s backoff (deadline-bounded), so a one-second publication race (data object commits between the uploader's sequential writes) can no longer fail the whole gate.

Disclosure (evidence parity)

The on-host release copy of polymarket-raw-ops-shadow-gate.sh (candidate cb55ac1d) is being patched with this exact file content to unblock the in-flight cutover; bundle hash recomputed. This PR makes it official.

Out of scope

• Uploader-side publication ordering (separate improvement)
• The legacy-health admission change (fix(deploy): drop legacy-Python health admission from polymarket shadow gate #553)

Dependency / merge order

None.

Focused validation

• Production race documented in fix(deploy): retry shadow-gate OSS readback on transient publication-race 404 #556 (404 at 07:47:39 == object Last-Modified second)
• test-polymarket-raw-ops-control-plane.sh PASS (new contract pins the retry wrapper on all three downloads)
• test-polymarket-raw-ops-stage.sh PASS; bash -n clean

Rollout / rollback impact

Takes effect with the next release bundle; host interim state matches. Rollback = revert.

Issue relationship

Refs #556

Summary by CodeRabbit

• Bug Fixes

  • Improved reliability of data downloads by automatically retrying failed object-storage transfers.
  • Applied retry handling consistently to data objects, manifests, and completion markers while honoring existing deadlines.

• Tests

  • Updated control-plane validation to confirm retry behavior is used for all required downloads.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[coderabbitai]

Warning

Review limit reached

@proerror77, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 9 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a4589652-8503-4428-a1e3-5b8275731edc

📥 Commits

Reviewing files that changed from the base of the PR and between 0379c02 and 1bc0346.

📒 Files selected for processing (2)

• deployment/aliyun/polymarket-raw-ops-shadow-gate.sh
• deployment/aliyun/test-polymarket-raw-ops-control-plane.sh

📝 Walkthrough

Walkthrough

The shadow gate now retries OSS downloads up to 15 times with 12-second delays within the existing deadline. Data, manifest, and _SUCCESS downloads use the helper. The control-plane test validates the helper and its three invocations.

Changes

OSS download retry flow

Layer / File(s)	Summary
Retry helper and OSS triplet integration deployment/aliyun/polymarket-raw-ops-shadow-gate.sh	Adds bounded retries for OSS copies and uses the helper for the data, manifest, and _SUCCESS downloads.
Control-plane retry contract validation deployment/aliyun/test-polymarket-raw-ops-control-plane.sh	Extracts the retry helper and verifies that the OSS retry wrapper is invoked three times.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

• proerror77/monday#556 — Requests the retry wrapper for shadow-gate OSS triplet readbacks and corresponding contract tests.

Possibly related PRs

• proerror77/monday#54 — Modifies the same shadow gate and control-plane tests, but addresses memory-pressure validation.
• proerror77/monday#376 — Adds bounded retry handling for related Polymarket OSS downloads in another code path.
• proerror77/monday#422 — Introduced the real-market preflight workflow that includes the modified shadow-gate OSS triplet downloads.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is structured, but it states 6 retries with 10-second backoff while the implementation uses 15 retries with 12-second backoff.	Update the Change contract to match the implementation and add the required Scope exception section with None.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the deployment fix: retrying shadow-gate OSS readback after transient publication-race 404 errors.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/pm-gate-readback-retry

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

deployment/aliyun/test-polymarket-raw-ops-control-plane.sh (1)

3846-3851: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Test the retry boundary instead of only matching source text.

These checks only prove that the gate contains one helper definition and three literal call strings. They do not prove that the data, manifest, and _SUCCESS downloads use the helper, or that retries stop at 15 attempts and at the deadline.

If the extracted helper runs with the no-op sleep() stub at deployment/aliyun/test-polymarket-raw-ops-control-plane.sh:4677, the test cannot detect elapsed-time or post-deadline behavior. Add executable failure-then-success, attempt-exhaustion, and expired-deadline counterexample tests with controlled command stubs.

As per coding guidelines, safety boundaries require a targeted counterexample test, not only workspace compilation.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deployment/aliyun/test-polymarket-raw-ops-control-plane.sh` around lines 3846
- 3851, Replace the grep-only assertions around the OSS retry checks with
executable counterexample tests for the extracted oss_download_with_retry
helper: use controlled command stubs to verify data, manifest, and _SUCCESS
downloads retry and eventually succeed, stop after 15 failed attempts, and do
not execute after an expired deadline. Ensure the test setup overrides the no-op
sleep and relevant download/time commands so attempt counts and deadline
behavior are observable, while preserving the existing gate validation.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deployment/aliyun/polymarket-raw-ops-shadow-gate.sh`:
- Around line 694-695: Update the retry loop in run_before_deadline so the sleep
backoff is executed only while the deadline guard still permits it. Ensure a
failed run_before_deadline after expiration exits without sleeping or continuing
retries, while preserving the existing attempt limit and 12-second backoff
before the deadline.

---

Nitpick comments:
In `@deployment/aliyun/test-polymarket-raw-ops-control-plane.sh`:
- Around line 3846-3851: Replace the grep-only assertions around the OSS retry
checks with executable counterexample tests for the extracted
oss_download_with_retry helper: use controlled command stubs to verify data,
manifest, and _SUCCESS downloads retry and eventually succeed, stop after 15
failed attempts, and do not execute after an expired deadline. Ensure the test
setup overrides the no-op sleep and relevant download/time commands so attempt
counts and deadline behavior are observable, while preserving the existing gate
validation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f08f5f61-25b4-4457-b01f-850bd7347708

📥 Commits

Reviewing files that changed from the base of the PR and between 98e0b80 and 0379c02.

📒 Files selected for processing (2)

• deployment/aliyun/polymarket-raw-ops-shadow-gate.sh
• deployment/aliyun/test-polymarket-raw-ops-control-plane.sh