Cargo build failed with spurious network error on docker hub auto builder and AWS T2 instance

[iav]

Have similar error trying to do a cross-compilation in docker.
I got it in automatic builder on a Docker Hub, then on a wps in Amazon S3.

This dockerfile (scenario) works on my x64 pc.
before run cross-platform image, qemu need to be added to host system.
Fast and easy for container runner docker run --rm --privileged multiarch/qemu-user-static:register --reset

It's not a "simple" network error, like dns not work, site unreachable, proxy should be used, proxy auth etc. Problem occurs on machine free internet access, and target sites reachable from same sessions with wget

error:
(in docker container)

cargo --version
cargo 1.37.0 (9edd08916 2019-08-02)

uname -a
Linux b584dc1b7555 5.0.0-1012-aws #13-Ubuntu SMP Fri Aug 2 12:25:32 UTC 2019 armv7l GNU/Linux

# qemu-arm-static --version
qemu-arm version 4.1.0 (qemu-4.1.0-1.fc31)

outside container —  aws instance:
$ uname -a
Linux ip-172-30-0-82 5.0.0-1012-aws #13-Ubuntu SMP Fri Aug 2 12:25:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux


# cargo run --verbose
    Updating crates.io index
warning: spurious network error (2 tries remaining): could not read directory '/usr/local/cargo/registry/index/github.com-1ecc6299db9ec823/.git//refs': Value too large for defined data type; class=Os (2)
warning: spurious network error (1 tries remaining): could not read directory '/usr/local/cargo/registry/index/github.com-1ecc6299db9ec823/.git//refs': Value too large for defined data type; class=Os (2)
error: failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  could not read directory '/usr/local/cargo/registry/index/github.com-1ecc6299db9ec823/.git//refs': Value too large for defined data type; class=Os (2)

Start of Dockerfile to reproduce inside AWS t2.medium instance, image id ubuntu/images/hvm-ssd/ubuntu-disco-19.04-amd64-server-20190819 (ami-00290d66f0da25f73)

FROM multiarch/qemu-user-static as qemu
FROM arm32v7/rust:1.37-buster as rust
COPY --from=qemu /usr/bin/qemu-arm-static /usr/bin
WORKDIR /
RUN git clone https://github.com/rustapp.git app
WORKDIR /app
RUN cargo build

[ehuss]

I'm not sure exactly what's wrong, but this seems to be an issue with the qemu environment and maybe its interaction with docker. The readdir C function fails inside any directory that you create. A repro:

Inside the qemu environment, create a file, a.c:

#include <dirent.h>
#include <stdio.h>
#include <sys/errno.h>

int main() {
    DIR *dirp;
    struct dirent *dp;
    dirp = opendir(".");
    if (dirp == NULL) {
        perror("opendir failed");
        return 1;
    }
    while ((dp = readdir(dirp)) != NULL) {
        printf("%s\n", dp->d_name);
    }
    if (errno) {
        perror("readdir failed");
        return 1;
    }

    closedir(dirp);
}

The run:

gcc -o a a.c
mkdir foo
cd foo
../a

should result in:

readdir failed: Value too large for defined data type

Something is maybe wrong with the qemu environment? My suspicion is that it could be related to large file support. However, getconf FILESIZEBITS / is returning 32, so I'm not sure why it is overflowing. Maybe something weird with the "overlay" filesystem?

[iav]

Run your sample on my PC (where initial case work ok):

$ cat Dockerfile.aarch64
# contains qemu-*-static for cross-compilation
FROM multiarch/qemu-user-static as qemu


FROM arm64v8/rust:1.37-buster as rust

COPY --from=qemu /usr/bin/qemu-aarch64-static /usr/bin
RUN /usr/bin/qemu-aarch64-static --version

# Install musl
RUN apt-get update && apt-get install -y gcc
WORKDIR /app

COPY a.c .
RUN uname -a
RUN gcc -o a a.c
RUN file a
RUN mkdir foo
#RUN cd foo
WORKDIR /app/foo
RUN ../a

Result:

<...cut...>
Step 7/13 : COPY a.c .
 ---> c48861c3edd6
Step 8/13 : RUN uname -a
 ---> Running in 6d22ac39ce27
Linux 6d22ac39ce27 5.0.0-29-generic #31-Ubuntu SMP Thu Sep 12 13:05:32 UTC 2019 aarch64 GNU/Linux
Removing intermediate container 6d22ac39ce27
 ---> 6d6f6e3d780d
Step 9/13 : RUN gcc -o a a.c
 ---> Running in 8468861460e9
Removing intermediate container 8468861460e9
 ---> 37c302460ddc
Step 10/13 : RUN file a
 ---> Running in 904c26d28431
a: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=190c550fac9b87cfb8b1ffdeb9b88bb3af8b3345, not stripped
Removing intermediate container 904c26d28431
 ---> 1df42604d446
Step 11/13 : RUN mkdir foo
 ---> Running in 2fcf5b826bbe
Removing intermediate container 2fcf5b826bbe
 ---> 523f68c96743
Step 12/13 : WORKDIR /app/foo
 ---> Running in 2a465ebb0234
Removing intermediate container 2a465ebb0234
 ---> ccfa88d170ed
Step 13/13 : RUN ../a
 ---> Running in ce01f2fd09c1
readdir failed: Cannot allocate memory
.
..
The command '/bin/sh -c ../a' returned a non-zero code: 1

[iav]

Command inside container from previous test:

# getconf FILESIZEBITS /
32

[iav]

Now run same on docker hub. (https://github.com/iav/test1)
Looks the same:

Step 7/14 : COPY a.c .
---> 21134c8a0cc4
Step 8/14 : RUN uname -a
---> Running in b0a6e9b1db7e
Linux b0a6e9b1db7e 4.4.0-1060-aws #69-Ubuntu SMP Sun May 20 13:42:07 UTC 2018 aarch64 GNU/Linux
Removing intermediate container b0a6e9b1db7e
---> b1c217408040
Step 9/14 : RUN getconf FILESIZEBITS /
---> Running in 3c10228f0154
32
Removing intermediate container 3c10228f0154
---> 011e820067ee
Step 10/14 : RUN gcc -o a a.c
---> Running in 6ac952ede364
Removing intermediate container 6ac952ede364
---> dae8b2fc6ca7
Step 11/14 : RUN file a
---> Running in 30586b566d0d
a: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=190c550fac9b87cfb8b1ffdeb9b88bb3af8b3345, not stripped
Removing intermediate container 30586b566d0d
---> 1ed8140bf56c
Step 12/14 : RUN mkdir foo
---> Running in dd20edb9120a
Removing intermediate container dd20edb9120a
---> 6ac9d1643728
Step 13/14 : WORKDIR /app/foo
Removing intermediate container 938df7afe618
---> d45435af9fad
Step 14/14 : RUN ../a
---> Running in 1e53921925ae
readdir failed: Cannot allocate memory
..
.
Removing intermediate container 1e53921925ae
The command '/bin/sh -c ../a' returned a non-zero code: 1

[splitice]

I too am getting this issue.

Anyone got a workaround?
"Value too large for defined data type; class=Os"

QEMU emulating Arm here.

[splitice]

This thread is for the underlying bug: https://lkml.org/lkml/2018/12/28/461

[ehuss]

I'm going to close this, as it seems unrelated to Cargo.

[iav]

Cargo could provide more descriptive diagnostic messages.
Current messages fit only for google feeding.

[Biswa96]

I have this same issue. I asked in qemu IRC and there is already a bug report here https://bugs.launchpad.net/qemu/+bug/1805913.

[technimad]

I experienced this issue today on macOS when using buildx to build for linux/arm/v7.
Can someone point me to a recommended work around?

[JonasAlfredsson]

For those searching for this issue and might stumble across this in the future: discussion moved over here, and the workaround found was to mount a tmpfs on ~/.cargo since this seems to not trigger the bug.

[t-moe]

I also tried to crosscompile some rust project for the raspberry pi and came across this issue when using qemu to run an arm docker image.

In my case the workaround was something like this:

# Install docker and qemu-user-static first

# then on the host:
mkdir /tmp/cargo
sudo mount -t tmpfs -o size=10G tmpfs /tmp/cargo
docker run -it -v /tmp/cargo:/usr/local/cargo/git --platform linux/arm/v7 rust:latest /bin/bash

# then inside the docker
cargo install ...