Skip to content

ci: raise publish retry timeout to give slow Nexus uploads headroom#15826

Merged
jamesfredley merged 1 commit into
8.0.xfrom
ci/publish-timeout-headroom
Jul 4, 2026
Merged

ci: raise publish retry timeout to give slow Nexus uploads headroom#15826
jamesfredley merged 1 commit into
8.0.xfrom
ci/publish-timeout-headroom

Conversation

@jamesfredley

Copy link
Copy Markdown
Contributor

What

Raise the per-attempt timeout_seconds from 1200 (20 min) to 1800 (30 min) on both publish retry steps in .github/workflows/gradle.yml — the publish job and the publishMicronaut job.

Why

CI run 28709355300 failed with only the publish job red. Tracing the full log of that job (85155475687):

  • The publish step wraps ./gradlew publish aggregateChecksums aggregatePublishedArtifacts --no-build-cache --rerun-tasks in nick-fields/retry with timeout_seconds: 1200 / max_attempts: 3.
  • All 3 attempts hit the 20-minute wall (Timeout of 1200000ms hit). There was no BUILD FAILED, no exception, and no Nexus Read timed out — a pure per-attempt timeout.
  • Because of --no-build-cache --rerun-tasks, every attempt spins a fresh Gradle daemon and does a full from-scratch recompile + groovydoc/javadoc for 60+ modules + upload to Apache Nexus, which legitimately runs ~14-20 min (the existing comment says "normal range 14min"). The last successful publish (28537677757) ran the whole job in 19m48s — essentially zero headroom.
  • Recurring ~3-min stalls during Nexus uploads (varying per attempt, so upload latency, not a deadlock) pushed each attempt just past 20 min. Since every retry repeats the identical full uncached build, the retries can't rescue it — they all hit the same wall.

Raising the per-attempt timeout to 30 min gives headroom for slow Nexus uploads so a single attempt is not killed mid-publish, while keeping max_attempts: 3 / retry_wait_seconds: 180 for genuine transient read-timeouts. Worst case (3 × 30 min + 2 × 180 s waits) is ~96 min, well under the GitHub Actions job limit.

Notes

  • CI-config only; no source/behavior change.
  • The same failed run was also re-run to confirm the diagnosis (transient slow-Nexus timing vs. a persistent defect).

The `publish` and `publishMicronaut` jobs wrap the Gradle publish command
(`--no-build-cache --rerun-tasks`) in the `nick-fields/retry` action with a
per-attempt `timeout_seconds: 1200` (20 min). That command does a full
from-scratch recompile plus groovydoc/javadoc for 60+ modules and uploads to
Apache Nexus, which legitimately runs ~14-20 min. When Nexus uploads are slow,
a single attempt exceeds 20 min and is killed mid-publish; because every retry
repeats the identical full uncached build, all 3 attempts hit the same wall and
the job fails on a pure timeout (no build error, no Nexus exception).

Raise `timeout_seconds` to 1800 (30 min) on both publish retry steps to give
headroom for slow uploads so a single attempt is not killed mid-publish, while
keeping `max_attempts: 3` / `retry_wait_seconds: 180` for genuine transient
read-timeouts. Worst case (3 x 30 min + 2 x 180 s waits) is ~96 min, well under
the GitHub Actions job limit.

Assisted-by: opencode:claude-opus-4-8
Copilot AI review requested due to automatic review settings July 4, 2026 22:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts the CI publish retry configuration to better tolerate slow Apache Nexus uploads, preventing otherwise-successful snapshot publishes from being killed by an overly tight per-attempt timeout.

Changes:

  • Increased nick-fields/retry timeout_seconds from 1200s to 1800s for the main publish job’s snapshot publish step.
  • Increased timeout_seconds from 1200s to 1800s for the publishMicronaut job’s snapshot publish step.
  • Expanded inline comments to clarify the rationale (uncached build duration and Nexus upload headroom).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@jamesfredley jamesfredley merged commit 9702f00 into 8.0.x Jul 4, 2026
39 of 40 checks passed
@jamesfredley jamesfredley deleted the ci/publish-timeout-headroom branch July 4, 2026 22:33
@codecov

codecov Bot commented Jul 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.5810%. Comparing base (c88070e) to head (82946a2).
⚠️ Report is 2 commits behind head on 8.0.x.

Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff              @@
##                8.0.x     #15826   +/-   ##
=============================================
  Coverage     49.5810%   49.5810%           
  Complexity      16695      16695           
=============================================
  Files            1947       1947           
  Lines           92598      92598           
  Branches        16179      16179           
=============================================
  Hits            45911      45911           
  Misses          39589      39589           
  Partials         7098       7098           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

MAVEN_PUBLISH_PASSWORD: ${{ secrets.NEXUS_PW }}
with:
timeout_seconds: 1200
timeout_seconds: 1800 # headroom for slow Nexus uploads so a single attempt is not killed mid-publish (see `publish` job)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jamesfredley why are we even defining timeout_seconds? I get that we used to do this b/c of RAO misperforming, but shouldn't we just remove teh timeout_seconds?

Also, we should merge changes without reviews ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants