FROM remote URL not working

[aniongithub]

When using FROM with the official URL for a lite image (https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip) from this page, I get

### FROM https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
Fetching remote: https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
--2021-04-26 17:01:43--  https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip%0D
Resolving downloads.raspberrypi.org (downloads.raspberrypi.org)... 93.93.130.212, 93.93.135.141, 46.235.230.122, ...
Connecting to downloads.raspberrypi.org (downloads.raspberrypi.org)|93.93.130.212|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-04-26 17:01:44 ERROR 404: Not Found.

but a wget with the same url gives me

wget https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
--2021-04-26 10:03:01--  https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
Resolving downloads.raspberrypi.org (downloads.raspberrypi.org)... 176.126.240.86, 176.126.240.84, 93.93.135.118, ...
Connecting to downloads.raspberrypi.org (downloads.raspberrypi.org)|176.126.240.86|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 463405209 (442M) [application/zip]
Saving to: ‘2021-03-04-raspios-buster-armhf-lite.zip.1’

buster-armhf-lite.zip.1           11%[=====>                                                    ]  51.60M  7.53MB/s    eta 58s

[oxzi]

Thanks for trying pimod and report your experience.

First, could you please provide your Pifile or at least its head regarding the FROM command?

Internally pimod also uses wget to fetch remote images, have a look at the from_remote_fetch function:

pimod/modules/from_remote.sh

Lines 42 to 44 in dd10904

	echo "Fetching remote: ${url}"
	mkdir -p "$(dirname "${download_path}")"
	wget --progress=dot:giga -O "${download_path}" "${url}"

However, the IP addresses are indicating that you have used two different mirror servers from the CDN. I successfully tried to access the file from your faulty mirror with:

curl \
  --header 'Host:downloads.raspberrypi.org' \
  --insecure \
  -o foo \
  https://93.93.130.212/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip

Thus, I would suggest that either the CDN server was faulty at that specific time or there was some something wrong with your Pifile.

[aniongithub]

FROM https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip

RUN curl -sSL get.docker.com | sh
RUN usermod -aG docker pi

Here is my pifile. I can't imagine that the CDN being down makes sense because I tried subsequent invocations of wget and docker run in the same console, right after one another (no time-gap) - multiple times. The wget always succeeds, but the

docker run --rm --privileged -v $PWD:/pimod pimod pimod.sh examples/Rpi-Docker.Pifile

always fails with the log shown above. Also note that both invocations used the exact same URL. How it resolves to 2 different CDNs is a mystery to me (not something I did) - maybe a side-effect of using Docker?

[aniongithub]

I also confirm that

running

root@dc9cb31b2187:/# wget https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
--2021-04-26 20:17:21--  https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
Resolving downloads.raspberrypi.org (downloads.raspberrypi.org)... 176.126.240.86, 176.126.240.84, 46.235.230.122, ...
Connecting to downloads.raspberrypi.org (downloads.raspberrypi.org)|176.126.240.86|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 463405209 (442M) [application/zip]
Saving to: '2021-03-04-raspios-buster-armhf-lite.zip'

                2021-03-04-ra  15%[======>                                           ]  68.55M  4.71MB/s    eta 81s

works fine, which is a docker container I ran using: docker run -it --rm ubuntu. I did this to eliminate docker being the cause.

[aniongithub]

This was my own fault - some quirk of the WSL (Windows Subsystem for Linux caused VS code to use CRLF line endings. Closing.

[oxzi]

I added a check to warn about potentially wrong encodings in 8b9db07.

Thanks a lot for reporting this issue!