"sh: write error: Invalid argument" errors when register

[heyi-arm]

Hi, on my x86_64 server:

$ uname -a
Linux a010291 4.4.0-127-generic #153-Ubuntu SMP Sat May 19 10:58:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:17:20 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:15:30 2018
OS/Arch: linux/amd64

Run the command "docker run --rm --privileged multiarch/qemu-user-static:register" encounters errors:

sh: write error: Invalid argument
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
sh: write error: Invalid argument
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
sh: write error: Invalid argument
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
sh: write error: Invalid argument
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
sh: write error: Invalid argument

But run command ""docker run --rm --privileged multiarch/qemu-user-static:register --reset" actually installs all the interpreters successfully:

Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa

[junaruga]

@heyi-arm hi the behavior you reported is expected behavior.

The error message sh: write error: Invalid argument is not understandable.
But the actual meaning is when "docker run --rm --privileged multiarch/qemu-user-static:register" try to add "binfmt_misc" entry files /proc/sys/fs/binfmt_misc/qemu-$arch, those files already exist.
/proc/sys/fs/binfmt_misc/qemu-$arch files are special files . [1]
When updating the file, we have to delete the file at once.

I am not sure which Linux distribution you are using.
When installing RPM qemu-user-staticon CentOS or Fedora, it adds the /proc/sys/fs/binfmt_misc/qemu-$arch files as a post script after yum install qemu-user-static.
There is deb package qemu-user-static package on Debian or Ubuntu.
It is possible those files already exist when you run "docker run --rm --privileged multiarch/qemu-user-static:register".

Here is the way to change flags: to flags: F for /proc/sys/fs/binfmt_misc/qemu-aarch64.

# prompt is executed by root user. sudo does not work for the operation.
But if you try this operation, that is by your own responsibility.

If the environment is really clean without binfmt_misc files. Maybe condition is like this.

$ ls /proc/sys/fs/binfmt_misc
register  status

# echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64

$ ls /proc/sys/fs/binfmt_misc/qemu-aarch64
ls: cannot access '/proc/sys/fs/binfmt_misc/qemu-aarch64': No such file or directory

# echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:" > /proc/sys/fs/binfmt_misc/register

$ ls /proc/sys/fs/binfmt_misc/qemu-aarch64
/proc/sys/fs/binfmt_misc/qemu-aarch64

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags:  
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

# echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64

$ ls /proc/sys/fs/binfmt_misc/qemu-aarch64
ls: cannot access '/proc/sys/fs/binfmt_misc/qemu-aarch64': No such file or directory

# echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:F" > /proc/sys/fs/binfmt_misc/register

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /qus/bin/qemu-aarch64-static
flags: F
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

I would close this ticket, as it is expected behavior.
But you like to reopen, you can do it.
Thanks for reporting!

By the way today we release a new image "multiarch/qemu-user-static" that can build and run standard architecture specific container instead of multiarch compatible imagtes.

Like this.

$ uname -m
x86_64

$ docker run --rm -t arm64v8/ubuntu uname -m
standard_init_linux.go:211: exec user process caused "exec format error"

$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

$ docker run --rm -t arm64v8/ubuntu uname -m
aarch64

• [1] https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html

[junaruga]

The register image is using below QEMU's scripts/qemu-binfmt-conf.sh script internally.
So, if the message is not understandable, you might be able to report to QEMU project.

Below is the logic to add binfmt_misc entry.

https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh#L269-L270

qemu_register_interpreter() {
    echo "Setting $qemu as binfmt interpreter for $cpu"
    qemu_generate_register > /proc/sys/fs/binfmt_misc/register
}

[hoshsadiq]

@junaruga I'm not sure I follow all of what you said. Could you please expand a little bit on it?

My build output can be found here: https://travis-ci.com/hoshsadiq/qemu-testing/builds/124639278

As you can the /proc/sys/fs/binfmt_misc/ is empty and doesn't have the files as suggested in your comment (unless I misunderstood).

[junaruga]

https://travis-ci.com/hoshsadiq/qemu-testing/builds/124639278

@hoshsadiq in your Travis log, below command outputs error message then finishes with exit status: 0. that is what we have to fix.

$ docker run --rm --privileged multiarch/qemu-user-static:$QUS_VERSION --reset -p yes
Status: Downloaded newer image for multiarch/qemu-user-static:v4.0.0-4
sh: write error: Invalid argument
...

That means docker run --rm --privileged multiarch/qemu-user-static:$QUS_VERSION --reset -p yes failed to create /proc/sys/fs/binfmt_misc/qemu-* files.
I do not know why for now. To know it, we need to debug /qemu-binfmt-conf.sh injecting set -x in it.

But in your case I found why the issue happened.

Here is my repository forked from your qemu testing repository.
https://github.com/junaruga/qemu-testing/tree/feature/debug
https://travis-ci.org/junaruga/qemu-testing/builds/577460704

diff --git a/.travis.yml b/.travis.yml
index 8a2fc7a..b31a117 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,4 +1,4 @@
-dist: trusty
+dist: xenial
 # sudo: required
 language: minimal

After changing dist from trusty to xenial, you can run it without errors.

[junaruga]

This repository's .travis.yml might be also useful.
https://github.com/junaruga/ci-multi-arch-test

[hoshsadiq]

Ah. I was being stupid. I was meant to be using bionic. Seems this is supported on Xenial and above. Thank you!

[lugeng]

echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:F" > /proc/sys/fs/binfmt_misc/register
bash: line 0: echo: write error: Invalid argument

ls /proc/sys/fs/binfmt_misc/qemu-aarch64
ls: cannot access /proc/sys/fs/binfmt_misc/qemu-aarch64: No such file or directory

But
echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:" > /proc/sys/fs/binfmt_misc/register

ls /proc/sys/fs/binfmt_misc/qemu-aarch64
/proc/sys/fs/binfmt_misc/qemu-aarch64 works

cannot set flag with F

[yuanzhedong]

@lugeng I got the same error, did you find a solution? Thanks!

[junaruga]

The F flag is a relatively new feature of the kernel binfmt_misc. Maybe it does not work if the host os kernel version (= uname -r) < 4. See #100 (comment) for the solution.

[Deepak275]

facing the same issue, kernel is 4.4, OS: Ubuntu 18.04.4 LTS. Tried running both, with and without -p yes, but no progress. Any help is appreciated ?

[lqhandsome]

must uname -r > 4.8