can not find the executable file under rootfs while launching an image via rkt

[chadcao]

running environment: coreos (VERSION_ID=522.2.0)

1. the image is like this:
   $tar tvf hello-app.aci
   drwxr-xr-x 500/500 0 2014-12-19 05:44 rootfs
   -rwxr-xr-x 500/500 34936 2014-12-19 05:44 rootfs/echo
   -rw-r--r-- root/root 390 2014-12-19 05:44 manifest

Here echo is the common linux command copied for /bin

1. the manifest is like this:
   $ more manifest
   {
   "acVersion": "0.1.1",
   "acKind": "ImageManifest",
   "name": "coreos.com/hello",
   "app": {
   "exec": [
   "/echo hello rocket"
   ]
   },
   "annotations": {
   }
   }
2. ran into "Failed at step EXEC spawning /echo: No such file or directory" error while trying to launch the aci by Rocket
   I'm sure there is an echo file under rootfs directory in the hello-app.aci and the echo is executable
   $ sudo ~/rocket-v0.1.1/rkt --debug=true run hello-app.aci
   2014/12/19 06:54:32 Unpacking stage1 rootfs
   2014/12/19 06:54:32 Writing stage1 init
   2014/12/19 06:54:32 Wrote filesystem to /var/lib/rkt/containers/d5fb6b7b-4297-477c-b6c6-52e218a9d7a8
   2014/12/19 06:54:32 Loading image sha256-d3d8cac1d9607a8a58a00ab5f2d617a0b3265cb639197b661d5492101bde5572
   2014/12/19 06:54:33 Writing container manifest
   2014/12/19 06:54:33 Pivoting to filesystem /var/lib/rkt/containers/d5fb6b7b-4297-477c-b6c6-52e218a9d7a8
   2014/12/19 06:54:33 Execing stage1/init
   Spawning container stage1 on /var/lib/rkt/containers/d5fb6b7b-4297-477c-b6c6-52e218a9d7a8/stage1.
   Press ^] three times within 1s to kill container.
   Timezone UTC does not exist in container, not updating container timezone.
   systemd 215 running in system mode. (-PAM -AUDIT -SELINUX +IMA -SYSVINIT +LIBCRYPTSETUP -GCRYPT -ACL -XZ +SECCOMP -APPARMOR)
   Detected virtualization 'systemd-nspawn'.
   Detected architecture 'x86-64'.

Welcome to Linux!

Initializing machine ID from container UUID.
[ OK ] Created slice -.slice.
[ OK ] Created slice system.slice.
Starting Graceful exit watcher...
[ OK ] Started Graceful exit watcher.
Starting coreos.com/hello...
[ OK ] Started coreos.com/hello.
[ OK ] Reached target Rocket apps target.
Failed at step EXEC spawning /echo: No such file or directory
sha256-d3d8cac1d9607a8a58a00ab5f2d617a0b3265cb639197b661d5492101bde5572.service: main process exited, code=exited, status=203/EXEC
Service exit-watcher.service is not needed anymore. Stopping.
Unit sha256-d3d8cac1d9607a8a58a00ab5f2d617a0b3265cb639197b661d5492101bde5572.service entered failed state.
Triggering OnFailure= dependencies of sha256-d3d8cac1d9607a8a58a00ab5f2d617a0b3265cb639197b661d5492101bde5572.service.
Shutting down.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounting /proc/sys/kernel/random/boot_id.
All filesystems unmounted.
Halting system.

more information:

1. On the same environment, I'm able to launch the etcd-v0.5.0-alpha.4-linux-amd64.aci
   ./rkt run https://github.com/coreos/etcd/releases/download/v0.5.0-alpha.4/etcd-v0.5.0-alpha.4-linux-amd64.aci
2. I tryied to tar xvf etcd-v0.5.0-alpha.4-linux-amd64.aci, put the echo file under rootfs, modify the manifest from "exec":["/etcd"] to "exec":["/echo hello rocket"], then re-build an image.
   The image is like this:
   $ tar tvf myaci.aci
   drwxr-xr-x 500/500 0 2014-12-19 06:24 rootfs
   drwxr-xr-x 500/500 0 2014-12-11 20:54 rootfs/Documentation
   -rw-r--r-- 500/500 2266 2014-11-26 18:54 rootfs/Documentation/0_4_migration_tool.md
   -rw-r--r-- 500/500 4438 2014-11-26 18:54 rootfs/Documentation/admin_guide.md
   -rw-r--r-- 500/500 25262 2014-11-26 18:54 rootfs/Documentation/api.md
   -rw-r--r-- 500/500 11013 2014-11-26 18:54 rootfs/Documentation/clustering.md
   -rw-r--r-- 500/500 4414 2014-11-26 18:54 rootfs/Documentation/configuration.md
   -rw-r--r-- 500/500 753 2014-11-26 18:54 rootfs/Documentation/glossary.md
   -rw-r--r-- 500/500 2363 2014-11-26 18:54 rootfs/Documentation/other_apis.md
   -rw-r--r-- 500/500 2751 2014-11-26 18:54 rootfs/Documentation/proxy.md
   -rw-r--r-- 500/500 6829 2014-11-26 18:54 rootfs/Documentation/runtime-configuration.md
   -rw-r--r-- 500/500 5186 2014-11-26 18:54 rootfs/README-etcdctl.md
   -rw-r--r-- 500/500 4672 2014-11-26 18:54 rootfs/README.md
   drwx------ 500/500 0 2014-12-19 06:00 rootfs/default.etcd
   -rwxr-xr-x 500/500 31296 2014-12-19 06:24 rootfs/echo
   drwxr-xr-x 500/500 0 2014-12-11 20:54 rootfs/etc
   -rw-r--r-- 500/500 79 2014-12-11 20:57 rootfs/etc/hosts
   -rwxr-xr-x 500/500 9029720 2014-11-26 18:54 rootfs/etcd
   -rwxr-xr-x 500/500 7161080 2014-11-26 18:54 rootfs/etcd-migrate
   -rwxr-xr-x 500/500 8936400 2014-11-26 18:54 rootfs/etcdctl
   -rw-r--r-- root/root 390 2014-12-19 06:27 manifest

then I tried to launch the new image,
$ sudo ~/rocket-v0.1.1/rkt --debug=true run myaci.aci
but I still got the "Failed at step EXEC spawning /echo: No such file or directory" error.

It's really weird that Rocket is able to find and exec etcd but is not able to find echo under the same rootfs directory....

[jonboulle]

The most common cause of this is that the executable in question is dynamically linked and the libraries on which it depends are not available on the rootfs. Is echo statically compiled?

#286

[chadcao]

echo is not re-complied, just copied from /bin
but I tried another statically complied go program (as demostrated by Kelsey Hightower)
package main

import (
"log"
"net/http"
)

func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
log.Printf("request from %v\n", r.RemoteAddr)
w.Write([]byte("hello\n"))
})
log.Fatal(http.ListenAndServe(":5000", nil))
}

complied with "CGO_ENABLED=0 GOOS=linux go build -a -tags netgo -ldflags '-w' ."

when I used this one instead of echo, I got the same error

[chadcao]

Also, if the issue is caused by missing dependent libraries, the error message should be clearer than "Failed at step EXEC spawning /echo: No such file or directory"

[jonboulle]

Yes, please look at the referenced ticket - #286 . We intend to make the error clearer but unfortunately it is nontrivial at present.

From this discussion, could you try building the hello example with the following?

go build -ldflags '-linkmode external -extldflags "-static"' hello.go

[jonboulle]

Also you can verify whether the compilation is successful by just running ldd against the produced binary.

[chadcao]

duplicated issue as #286