[install-plugins.sh] Try other url when 1 fails

[elouanKeryell-Even]

For a reason that I don't understand, I am having connectivity problems with one of the servers hosting jenkins plugins, namely icm.edu.pl.

When downloading plugins, mirrors.jenkins-ci.org shows a list of several urls for the resource I am looking for (HTTP Link headers), and chooses one for me (HTTP Location header).

Sometimes it works fine:

# curl -S -D - --connect-timeout 20 --retry 5 --retry-delay 0 --retry-max-time 60 -s -L https://updates.jenkins.io/download/plugins/matrix-auth/1.3.2/matrix-auth.hpi -o /dev/null
HTTP/1.1 302 Found
Date: Fri, 21 Oct 2016 15:19:58 GMT
Server: Apache/2.4.7 (Ubuntu)
Location: http://mirrors.jenkins-ci.org/plugins/matrix-auth/1.3.2/matrix-auth.hpi
Content-Length: 255
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 302 Found
Date: Fri, 21 Oct 2016 15:19:58 GMT
Server: Apache/2.4.7 (Ubuntu)
X-MirrorBrain-Mirror: ftp.halifax.rwth-aachen.de
X-MirrorBrain-Realm: region
Link: <http://mirrors.jenkins-ci.org/plugins/matrix-auth/1.3.2/matrix-auth.hpi.meta4>; rel=describedby; type="application/metalink4+xml"
Link: <http://ftp.halifax.rwth-aachen.de/jenkins/plugins/matrix-auth/1.3.2/matrix-auth.hpi>; rel=duplicate; pri=1; geo=de
Link: <http://jenkins.mirror.isppower.de/plugins/matrix-auth/1.3.2/matrix-auth.hpi>; rel=duplicate; pri=2; geo=de
Link: <http://ftp.icm.edu.pl/packages/jenkins/plugins/matrix-auth/1.3.2/matrix-auth.hpi>; rel=duplicate; pri=3; geo=pl
Link: <http://ftp-nyc.osuosl.org/pub/jenkins/plugins/matrix-auth/1.3.2/matrix-auth.hpi>; rel=duplicate; pri=4; geo=us
Link: <http://mirror.xmission.com/jenkins/plugins/matrix-auth/1.3.2/matrix-auth.hpi>; rel=duplicate; pri=5; geo=us
Location: http://ftp.halifax.rwth-aachen.de/jenkins/plugins/matrix-auth/1.3.2/matrix-auth.hpi
Content-Length: 267
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 200 OK
Server: nginx/1.9.10
Date: Fri, 21 Oct 2016 15:19:59 GMT
Content-Type: application/octet-stream
Content-Length: 77026
Last-Modified: Tue, 19 Apr 2016 20:44:21 GMT
Connection: keep-alive
Accept-Ranges: bytes

Other times, when mirrors.jenkins-ci.org tells me to download from the particular server that I can't reach, it will fail:

# curl -S -D - --connect-timeout 20 --retry 5 --retry-delay 0 --retry-max-time 60 -s -L https://updates.jenkins.io/download/plugins/matrix-auth/1.3.2/matrix-auth.hpi -o /dev/null
HTTP/1.1 302 Found
Date: Fri, 21 Oct 2016 15:20:00 GMT
Server: Apache/2.4.7 (Ubuntu)
Location: http://mirrors.jenkins-ci.org/plugins/matrix-auth/1.3.2/matrix-auth.hpi
Content-Length: 255
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 302 Found
Date: Fri, 21 Oct 2016 15:20:00 GMT
Server: Apache/2.4.7 (Ubuntu)
X-MirrorBrain-Mirror: icm.edu.pl
X-MirrorBrain-Realm: region
Link: <http://mirrors.jenkins-ci.org/plugins/matrix-auth/1.3.2/matrix-auth.hpi.meta4>; rel=describedby; type="application/metalink4+xml"
Link: <http://ftp.icm.edu.pl/packages/jenkins/plugins/matrix-auth/1.3.2/matrix-auth.hpi>; rel=duplicate; pri=1; geo=pl
Link: <http://ftp.halifax.rwth-aachen.de/jenkins/plugins/matrix-auth/1.3.2/matrix-auth.hpi>; rel=duplicate; pri=2; geo=de
Link: <http://jenkins.mirror.isppower.de/plugins/matrix-auth/1.3.2/matrix-auth.hpi>; rel=duplicate; pri=3; geo=de
Link: <http://ftp-nyc.osuosl.org/pub/jenkins/plugins/matrix-auth/1.3.2/matrix-auth.hpi>; rel=duplicate; pri=4; geo=us
Link: <http://ftp-chi.osuosl.org/pub/jenkins/plugins/matrix-auth/1.3.2/matrix-auth.hpi>; rel=duplicate; pri=5; geo=us
Location: http://ftp.icm.edu.pl/packages/jenkins/plugins/matrix-auth/1.3.2/matrix-auth.hpi
Content-Length: 264
Content-Type: text/html; charset=iso-8859-1

curl: (7) Failed to connect to 2001:6a0:0:31::2: Network is unreachable

I was thinking maybe the downloading section of the script could be strengthened by trying those other urls in the Link headers, when the 1st one does not work.

I know this might be a little over-the-top, since nearly nobody should be affected by those kind of rare connectivity problems, but I thought I should mention it anyway :)

[elouanKeryell-Even]

Here is the little bash function I use as a work-around:

strengthenedDownload() {
    local plugin version url jpi
    plugin="$1"
    version="$2"
    url="$3"
    jpi="$4"

    # Retrieve the "proxy location". It is a location that contains, in its metadata, all the "real" locations where the resource can be found.
    proxy_location=$(curl -D - --connect-timeout ${CURL_CONNECTION_TIMEOUT:-20} --retry ${CURL_RETRY:-5} --retry-delay ${CURL_RETRY_DELAY:-0} --retry-max-time ${CURL_RETRY_MAX_TIME:-60} -s "$url" -o /dev/null | tr -d '\r' | sed -En 's/^Location: (.*)/\1/p')

    # Retrieve all "real" locations, and put them into array "locations". Those locations should normally be sorted by priority.
    mapfile -t locations < <(curl -D - --connect-timeout ${CURL_CONNECTION_TIMEOUT:-20} --retry ${CURL_RETRY:-5} --retry-delay ${CURL_RETRY_DELAY:-0} --retry-max-time ${CURL_RETRY_MAX_TIME:-60} -s $proxy_location -o /dev/null | tr -d '\r' | sed -En 's/^Link: <(.*)>; rel=duplicate; pri=[0-9]; geo=.*/\1/p')

    for location in ${locations[@]}; do
        echo "[${plugin}:${version}] Trying location $location ..."
        set +e
        curl --connect-timeout ${CURL_CONNECTION_TIMEOUT:-20} --retry ${CURL_RETRY:-5} --retry-delay ${CURL_RETRY_DELAY:-0} --retry-max-time ${CURL_RETRY_MAX_TIME:-60} -s "$location" -o "$jpi"
        if (( $? == 0)); then
            set -e
            echo "[${plugin}:${version}] Download succeeded"
            return 0
        else
            set -e
            echo '[${plugin}:${version}] Download failed'
        fi
    done

    >&2 echo '[${plugin}:${version}] All download locations failed.'
    return 1
}

I just replaced the curl line in the script by a call to this function, and it works in my case. Might be very breakable though, I haven't tested it thoroughly.

[Chrislevi]

having same problem. seems like the install-plugins.sh needs more refinement, i myself had to alter the script due to its curl command lacking extended timeout and retry enhancment.