Improve jdk download retries

[alpar-t]

Look like the --retry argument is not always effective.

Closes #40531

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[dliappis]

Thanks! I left a few comments, mainly highlighting the need to emit the error at the end.

[dliappis]

We need to emit the error here (similar to how we do it for yum install later on) using:

 \
    (exit $exit_code)

otherwise as it is now, if all retries are exhausted it will continue to the next step.

Unrelated, but while testing this, I observed that (when running ./gradlew with --info) retry messages are shown as:

download error: retry $iter in 10s

instead of

download error: retry 1<or 2, 3 and so on> in 10s

I thought this was ok and tested in the past in the yum section, but apparently \ shouldn't be used inside the double quoted section.

In other words the complete fixed command should be:

RUN for iter in {1..10}; do curl -L -s -S ${jdkUrl} | tar -C /opt -zx && \
    exit_code=0 && break || exit_code=\$? && echo "download error: retry $iter in 10s" && sleep 10; done; \
    (exit $exit_code)

[dliappis]

nit: let's have a space between && and \ for consistency i.e. && \

[dliappis]

One new line is enough, I think we can remove this.

[mark-vieira]

LGTM I suppose. Pretty annoying that --retry doesn't do what we want. Did we discover what kind of errors we are encountering that aren't caught by --retry?

[dliappis]

Did we discover what kind of errors we are encountering that aren't caught by --retry?

Connection resets -- which is something that occasionally happens from the GitHub URL we are relying -- aren't being retried (others reporting the same here).

For the record curl already covers connection refused (via curl/curl#1036) and will also retry connection timeouts (if --connect-timeout has been set).

[alpar-t]

but --connection-timeoutis not available in the curl version we use here.

[dliappis]

LGTM. I left a minor comment, there's no need for another review cycle.

[dliappis]

(super) nit: we could remove the 4 space characters in line 22 (so that there's just a newline).

[dliappis]

but --connection-timeoutis not available in the curl version we use here.

Good point and additionally, my earlier comment attempts to show that there are other errors, like connection resets, that curl won't catch, even with the latest version, so I am in favor of wrapping such operations in this kind of retry logic -- even if it looks a bit awkward due to the shell syntax.