exec user process caused "exec format error" when run container with CMD on RHEL

[gouyang]

Download below two files to a directory and build an image from it.

https://github.com/docker-library/hello-world/tree/master/amd64/hello-world
https://github.com/docker-library/hello-world/blob/master/amd64/hello-world/hello

on RHEL 7.4, it throws an error as below. No this error on Fedora 27.

# buildah bud -t testing/hello .
STEP 1: FROM scratch
STEP 2: COPY hello /
STEP 3: CMD ["/hello"]
STEP 4: COMMIT containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]docker.io/testing/hello:latest

# buildah images 
IMAGE ID             IMAGE NAME                                               CREATED AT             SIZE
f4ff539da64e         docker.io/testing/hello:latest                           Feb 13, 2018 08:27     3.699 KB
# buildah from docker.io/testing/hello
hello-working-container
# buildah run hello-working-container
standard_init_linux.go:190: exec user process caused "exec format error"

[TomSweeneyRedHat]

@gouyang Thanks for the issue report, we'll take a look at it.

[rhatdan]

@gouyang Could you try to build buildah from master and see if you get this issue. I just built buildah-0.15 for RHEL, but it will take a few weeks to get released.

[gouyang]

The issue is still existing by building buildah from master.

# ./buildah run hello-working-container
standard_init_linux.go:190: exec user process caused "exec format error"
# ./buildah version
Version:       0.15
Go Version:    go1.8.1
Image Spec:    1.0.0
Runtime Spec:  1.0.0
Git Commit:    d1330a5
Built:         Wed Feb 28 19:59:37 2018
OS/Arch:       linux/amd64

[rhatdan]

THis could be a runc version error.
runc version

[gouyang]

It's the same result with runc build from master.

# runc --version
runc version 1.0.0-rc5+dev
commit: aada2af1b2ffb3538d967878ad55f9365c77a0a8
spec: 1.0.0

[TomSweeneyRedHat]

I finally had some time to play and wasn't able to duplicate this. I tried both Buildah 0.12 and Buildah 0.15 and everything worked for me. Our runc versions are different, I just used what ever was installed with RHEL rather than updating that.

# buildah bud -t testing/hello .
STEP 1: FROM scratch
STEP 2: COPY hello /
STEP 3: CMD ["/hello"]
STEP 4: COMMIT containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]docker.io/testing/hello:latest
# buildah --version
buildah version 0.15 (image-spec 1.0.0, runtime-spec 1.0.0)
[root@localhost tmp]# cat /etc/*release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.4 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.4"
PRETTY_NAME="OpenShift Enterprise"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.4"
Red Hat Enterprise Linux Server release 7.4 (Maipo)
Red Hat Enterprise Linux Server release 7.4 (Maipo)
[root@localhost tmp]# runc --version
runc version spec: 1.0.0

I then upgraded runc to the latest version and had the same result:

# buildah bud -t testing/hello .
STEP 1: FROM scratch
STEP 2: COPY hello /
STEP 3: CMD ["/hello"]
STEP 4: COMMIT containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]docker.io/testing/hello:latest
[root@localhost tmp]# runc version
No help topic for 'version'
[root@localhost tmp]# runc --version
runc version 1.0.0-rc5+dev
commit: ce80fa0a64803d52883955cb77b2708b438a0b28
spec: 1.0.0

If possible, could you retest on a new VM with RHEL @gouyang ?

[gouyang]

It's not reproduced on a new VM RHEL guest. It's strange it 's always reproduced on one of my RHEL system no matter reboot the system or reinstall buildah+runc on it.

[gouyang]

It seems the problem of the hello file, built a binary locally works fine.

# cat hello.go 
package main

import "fmt"

func main() {
	fmt.Println("hello world")
}
# go build -o hello .
# ./hello 
hello world
# buildah bud -t hello2 .
STEP 1: FROM scratch
STEP 2: COPY hello /
STEP 3: CMD ["/hello"]
STEP 4: COMMIT containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]docker.io/library/hello2:latest
# buildah from hello2
hello2-working-container
# buildah run hello2-working-container
hello world

[TomSweeneyRedHat]

@gouyang thanks for the follow up. That's strange that the binary was problematic on just that one machine. If you run into that again, please reopen or open a new issue.

[momocow]

@gouyang Make sure if there is a shebang in hello.

Though I'm not familiar with go, I happen to find here since I met with the same problem (standard_init_linux.go:190: exec user process caused "exec format error").

I think it is because the system does not know how to execute the file.
FYI: What's the appropriate Go shebang line?

Also be aware of the difference between the shell form and exec form of CMD in Dockerfile.

[khalidumarr]

just adding #!/bin/bash to my entry point file fixed the issue.

[pixdrift]

@khalidumarr, which version of buildah are you using? Did you execute with buildah run and you received this error?

[khalidumarr]

i was using docker run directly.

[ealexhaywood]

Why should there be a shebang in hello? It's a compiled binary

[TrayserCassa]

We build with an wrong architecture...
CGO_ENABLED=0
GOOS=linux
GOARCH=amd64

and then:
go build -o bin/serviceXYZ_linux_amd64

Our Image is for godeployment and we get this error. Maybe it will help somebody..

[rhatdan]

@TrayserCassa What error?

[TrayserCassa]

@rhatdan
Sorry for the late answering...
Like in the title in this issue: 'exec format error'

[rhatdan]

There is probably a good change the runc version is out of date.
rpm -q runc

[adambant]

just adding #!/bin/bash to my entry point file fixed the issue.

Turns out the #!/bin/bash was in my entry point file, but since I did a copy and paste into that file, the first line was a newline, not the #!/bin/bash, effectively ignoring it. If this helps anyone as well. Deleted the empty line and all worked fine.

[TomSweeneyRedHat]

@adambant TYVM for the follow up and info on your work around.

[Kiran01bm]

@gouyang Make sure if there is a shebang in hello.

Though I'm not familiar with go, I happen to find here since I met with the same problem (standard_init_linux.go:190: exec user process caused "exec format error").

I think it is because the system does not know how to execute the file.
FYI: What's the appropriate Go shebang line?

Also be aware of the difference between the shell form and exec form of CMD in Dockerfile.

FWIW - I had the same error (copied below) with a ruby container and the problem was due to a missing shebang in my ruby program. I was using CMD instruction in a exec form (CMD instruction copied below)

Error

standard_init_linux.go:190: exec user process caused "exec format error"

CMD:

CMD ["somefile.rb"]

[k4korea]

good. best quality .. answer very very good

[jomach]

in my case the entrypoint.sh had the wrong shebang

[fractos]

In case anyone finds this useful, you can get this issue if your shell script uses CRLF for line endings and/or UTF-8 with BOM (e.g. if you created a shell script file in Visual Studio). Changing to LF only and straight UTF-8 fixed it for me.

[limup]

just adding #!/bin/bash to my entry point file fixed the issue.

Thanks soooooo much!!!!

Now... I have a new error:

/bin/bash: /docker-entrypoint.sh: Permission denied

=( =(

[TomSweeneyRedHat]

@limup can you open a new issue please and send us details if you've not figured out the issue yet?

[limup]

@limup can you open a new issue please and send us details if you've not figured out the issue yet?

Well, I fixed the problem. I added the command in the Dockerfile and execute:

`ENTRYPOINT ["/docker-entrypoint.sh"]
RUN chmod +x /docker-entrypoint.sh`

[~See ya]

[TomSweeneyRedHat]

Thanks for the follow up and solution @limup !

[rhatdan]

@limup I would rather you called it container-entrypoint, and ran it with podman :^)

[limup]

@limup I would rather you called it container-entrypoint, and ran it with podman :^)

@rhatdan I'm curious with the podman... what's it? And what is it useful for?

[TomSweeneyRedHat]

@limup Podman is a related project to Buildah. It lives under the libpod project at: https://github.com/containers/libpod and has it's own website https://podman.io. In short, it's a OCI compliant container system. buildah run emulates what happens with the RUN command in a Dockerfile. podman run is the equivalent of the cli docker run command. In fact you can alias podman for docker and it should be gtg on your system. For more details, please see this blog: https://developers.redhat.com/blog/2018/11/20/buildah-podman-containers-without-daemons/

[limup]

Thanks for sharing it.

[akshayeshenoi]

just adding #!/bin/bash to my entry point file fixed the issue.

For anyone who got a standard_init_linux.go:190: exec user process caused "no such file or directory" error after applying this fix, you're probably on an alpine base image which does not come with bash.

Replacing #!/bin/bash with #!/bin/sh will do the trick!

[kazizi-swe]

just adding #!/bin/bash to my entry point file fixed the issue.

By entry point, it means to the top of the .sh file itself. For some people, it might not be clear.

[rsmithlal]

just adding #!/bin/bash to my entry point file fixed the issue.

Turns out the #!/bin/bash was in my entry point file, but since I did a copy and paste into that file, the first line was a newline, not the #!/bin/bash, effectively ignoring it. If this helps anyone as well. Deleted the empty line and all worked fine.

Ahhh! This was my problem! Thank you so much!!!

[ronak34]

just adding #!/bin/bash to my entry point file fixed the issue.

For anyone who got a standard_init_linux.go:190: exec user process caused "no such file or directory" error after applying this fix, you're probably on an alpine base image which does not come with bash.

Replacing #!/bin/bash with #!/bin/sh will do the trick!

Thanks, this worked for me.

[INTK-S]

just adding #!/bin/bash to my entry point file fixed the issue.

where is this file？

[jomach]

just adding #!/bin/bash to my entry point file fixed the issue.

where is this file？

entrypoint.sh

[zakkg3]

In my case i needed to install https://github.com/multiarch/qemu-user-static