Retry wget downloads in Ubuntu and distroless Dockerfiles

[groeneai]

The Docker server image job intermittently fails when fetching the prebuilt .deb packages from clickhouse-builds.s3.amazonaws.com:

ERROR 500: Internal Server Error.
ERROR 503: Service Unavailable.

Three unrelated PRs hit this S3 download flake over three days — #100173 on 2026-05-15 (500), #104694 on 2026-05-14 (503), #104853 on 2026-05-13 (503). The 14-day CIDB pattern is below.

day         master_hits  pr_hits  distinct_prs
2026-05-15           0        1            1
2026-05-14           0        2            1
2026-05-13           0        1            1
2026-05-08           0        6            1
2026-05-07           0        1            1
2026-05-05           0       12            5   ← apt mirror outage, different shape
2026-05-04           1       20           10   ← apt mirror outage, different shape

(Most of 2026-05-04/2026-05-05 was a different, apt-ca-certificates failure mode; the recent 500/503 shape on May 13–15 is the one this PR addresses.)

docker/server/Dockerfile.alpine already wraps wget with a retry helper (added in #100380 to absorb the same kind of transient errors during Alpine cross-architecture builds via QEMU). The Ubuntu and distroless flavours still call wget ... || exit 1 on the very first attempt, so a single 500/503 from S3 kills the whole build.

This change ports the same wget_with_retry wrapper to docker/server/Dockerfile.ubuntu and docker/server/Dockerfile.distroless for the DIRECT_DOWNLOAD_URLS block (the path used by CI). WGET_RETRIES (default 5) and WGET_RETRY_DELAY (default 1s) match the Alpine version and remain overridable via --build-arg.

Triggered by @alexey-milovidov's directive on #100173:

@groeneai, could you investigate the latter two (Docker server image S3 500 / Stress test (arm_tsan) global timeout) and provide a fix in a separate PR if a fix is needed?

CI report (#100173, sha 3ceac71): https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=100173&sha=3ceac71a43a5fd4fc1a7859937b23e71b8fe42ae&name_0=PR&name_1=Docker%20server%20image

Changelog category (leave one):

• CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

...

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

Version info

• Merged into: 26.5.1.797

[groeneai]

cc @alexey-milovidov @Algunenano — could you review this? It ports the wget_with_retry wrapper from Dockerfile.alpine (which @alexey-milovidov added in #100380) to Dockerfile.ubuntu and Dockerfile.distroless, so a transient 500/503 from clickhouse-builds.s3.amazonaws.com no longer kills the Docker server image build on the first try. Triggered by @alexey-milovidov's directive on #100173.

[groeneai]

Pre-PR validation gate (per worker policy):

This is an infrastructure / Dockerfile fix, not a flaky-test fix, so the deterministic-repro gate doesn't apply 1:1. The relevant checks:

• a) Deterministic repro? Not on-demand reproducible (the trigger is an external S3 outage), but the failure shape is reproduced 3× across 3 unrelated PRs in 3 days (Add user_files_policy server setting for custom disks in user_files directory #100173 / 500, Wire functions stress test to ProcessList for KILL-QUERY-style timeouts #104694 / 503, Fix block structure mismatch in UnionStep after filter push-down with NULL constants #104853 / 503) — concrete signal, not anecdote.
• b) Root cause explained? Yes. Dockerfile.ubuntu and Dockerfile.distroless run wget ... || exit 1 once. A single 5xx from clickhouse-builds.s3.amazonaws.com then fails the whole docker build step. Dockerfile.alpine already wraps wget with wget_with_retry (added in Add wget retry logic to Alpine Docker image build #100380); Ubuntu/distroless were just never updated.
• c) Fix matches root cause? Yes — adds the same wget_with_retry wrapper around the DIRECT_DOWNLOAD_URLS wget call in both Dockerfiles. Same approach @alexey-milovidov used for Alpine in Add wget retry logic to Alpine Docker image build #100380.
• d) Test intent preserved? No tests changed. Build behaviour on success is unchanged (wrapper succeeds on first attempt → same as before). On transient failure, wrapper retries up to WGET_RETRIES=5 times (overridable) instead of failing immediately.
• e) Both directions demonstrated? Shell function syntax checked under sh/bash/dash (all parse clean). Retry behaviour validated locally against an unreachable URL — confirmed it retries WGET_RETRIES times with WGET_RETRY_DELAY between attempts, then exits non-zero.
• f) Fix is general? Yes — wrapper is applied in both Ubuntu and distroless Dockerfiles in one go. Alpine already has it. The deb_location_url/single_binary_location_url blocks are untouched because CI doesn't take those paths (DIRECT_DOWNLOAD_URLS is what fails). Happy to extend the wrapper to those blocks too if you'd prefer one consistent style across all install paths.

Cross-PR check at PR-creation time: no competing open PR touching Dockerfile.ubuntu/Dockerfile.distroless or mentioning wget_with_retry/DIRECT_DOWNLOAD_URLS.

[clickhouse-gh]

Workflow [PR], commit [394cd58]

Summary: ✅

---

AI Review

Summary

This PR adds wget_with_retry to docker/server/Dockerfile.ubuntu and docker/server/Dockerfile.distroless for the DIRECT_DOWNLOAD_URLS install path, aligning behavior with the existing Alpine retry approach so transient S3 500/503 responses do not fail the image build on the first attempt. I reviewed the current diff and prior discussion, and I did not find unresolved correctness, safety, or contract violations in the changed code.

Final Verdict

• Status: ✅ Approve