Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-31200][K8S] Enforce to use
https
in /etc/apt/sources.list
…n progress errors. ### What changes were proposed in this pull request? Switching to `https` instead of `http` in the debian mirror urls. ### Why are the changes needed? My ISP was trying to intercept (or trying to serve from cache) the `http` traffic and this was causing a very confusing errors while building the spark image. I thought by posting this, I can help someone save his time and energy, if he encounters the same issue. ``` bash-3.2$ bin/docker-image-tool.sh -r scrapcodes -t v3.1.0-f1cc86 build Sending build context to Docker daemon 203.4MB Step 1/18 : ARG java_image_tag=8-jre-slim Step 2/18 : FROM openjdk:${java_image_tag} ---> 381b20190cf7 Step 3/18 : ARG spark_uid=185 ---> Using cache ---> 65c06f86753c Step 4/18 : RUN set -ex && apt-get update && ln -s /lib /lib64 && apt install -y bash tini libc6 libpam-modules krb5-user libnss3 procps && mkdir -p /opt/spark && mkdir -p /opt/spark/examples && mkdir -p /opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh && ln -sv /bin/bash /bin/sh && echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && chgrp root /etc/passwd && chmod ug+rw /etc/passwd && rm -rf /var/cache/apt/* ---> Running in 96bcbe927d35 + apt-get update Get:1 http://deb.debian.org/debian buster InRelease [122 kB] Get:2 http://deb.debian.org/debian buster-updates InRelease [49.3 kB] Get:3 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB] Err:3 http://deb.debian.org/debian buster/main amd64 Packages File has unexpected size (13217 != 7906744). Mirror sync in progress? [IP: 151.101.10.133 80] Hashes of expected file: - Filesize:7906744 [weak] - SHA256:80ed5d1cc1f31a568b77e4fadfd9e01fa4d65e951243fd2ce29eee14d4b532cc - MD5Sum:80b6d9c1b6630b2234161e42f4040ab3 [weak] Release file created at: Sat, 08 Feb 2020 10:57:10 +0000 Get:5 http://deb.debian.org/debian buster-updates/main amd64 Packages [7380 B] Err:5 http://deb.debian.org/debian buster-updates/main amd64 Packages File has unexpected size (13233 != 7380). Mirror sync in progress? [IP: 151.101.10.133 80] Hashes of expected file: - Filesize:7380 [weak] - SHA256:6af9ea081b6a3da33cfaf76a81978517f65d38e45230089a5612e56f2b6b789d Release file created at: Fri, 20 Mar 2020 02:28:11 +0000 Get:4 http://security-cdn.debian.org/debian-security buster/updates InRelease [65.4 kB] Get:6 http://security-cdn.debian.org/debian-security buster/updates/main amd64 Packages [183 kB] Fetched 419 kB in 1s (327 kB/s) Reading package lists... E: Failed to fetch http://deb.debian.org/debian/dists/buster/main/binary-amd64/by-hash/SHA256/80ed5d1cc1f31a568b77e4fadfd9e01fa4d65e951243fd2ce29eee14d4b532cc File has unexpected size (13217 != 7906744). Mirror sync in progress? [IP: 151.101.10.133 80] Hashes of expected file: - Filesize:7906744 [weak] - SHA256:80ed5d1cc1f31a568b77e4fadfd9e01fa4d65e951243fd2ce29eee14d4b532cc - MD5Sum:80b6d9c1b6630b2234161e42f4040ab3 [weak] Release file created at: Sat, 08 Feb 2020 10:57:10 +0000 E: Failed to fetch http://deb.debian.org/debian/dists/buster-updates/main/binary-amd64/by-hash/SHA256/6af9ea081b6a3da33cfaf76a81978517f65d38e45230089a5612e56f2b6b789d File has unexpected size (13233 != 7380). Mirror sync in progress? [IP: 151.101.10.133 80] Hashes of expected file: - Filesize:7380 [weak] - SHA256:6af9ea081b6a3da33cfaf76a81978517f65d38e45230089a5612e56f2b6b789d Release file created at: Fri, 20 Mar 2020 02:28:11 +0000 E: Some index files failed to download. They have been ignored, or old ones used instead. The command '/bin/sh -c set -ex && apt-get update && ln -s /lib /lib64 && apt install -y bash tini libc6 libpam-modules krb5-user libnss3 procps && mkdir -p /opt/spark && mkdir -p /opt/spark/examples && mkdir -p /opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh && ln -sv /bin/bash /bin/sh && echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && chgrp root /etc/passwd && chmod ug+rw /etc/passwd && rm -rf /var/cache/apt/*' returned a non-zero code: 100 Failed to build Spark JVM Docker image, please refer to Docker build output for details. ``` ### Does this PR introduce any user-facing change? No ### How was this patch tested? Manually by switching to `https` mirrors on the offending ISP (I am already on). Closes #27966 from ScrapCodes/docker-mirror. Authored-by: Prashant Sharma <prashsh1@in.ibm.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
- Loading branch information