New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rkt run hello.aci exits straight after start #270

Closed
milosgajdos83 opened this Issue Dec 11, 2014 · 25 comments

Comments

Projects
None yet
6 participants
@milosgajdos83

milosgajdos83 commented Dec 11, 2014

Hey, I've been following getting started example and when I tried to run hello.aci image, it crashed straight away with the following output:

root@rocket:/opt/golang/src/github.com/coreos/rocket/bin# ./rkt run hello.aci
/etc/localtime is not a symlink, not updating container timezone.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounting /proc/sys/kernel/random/boot_id.
All filesystems unmounted.
Halting system.

I'm running this on Ubuntu Trusty - not sure how that matters.
The rocket manifest is exactly the same as in the example:

{
    "acKind": "ImageManifest",
    "acVersion": "0.1.0",
    "name": "coreos.com/hello-1.0.0",
    "labels": [
        {
            "name": "version",
            "val": "1.0.0"
        },
        {
            "name": "arch",
            "val": "amd64"
        },
        {
            "name": "os",
            "val": "linux"
        }
    ],
    "app": {
        "exec": [
            "/bin/hello"
        ],
        "ports": [
        {
            "name": "www",
            "protocol": "tcp",
            "port": 5000
        }
        ]
    },
    "annotations": {
        "authors": "Kelsey Hightower <kelsey.hightower@gmail.com>"
    }
}

After the rocket crash, syslog spits out:

Dec 11 16:50:00 vagrant-ubuntu-trusty-64 kernel: [21642.566934] cgroup: new mount options do not match the existing superblock, will be ignored

When I run the hello app outside of rocket ie just on a command line, all works fine - ie. it's not related to the binary itself.

Similarly, the example

root@rocket:/opt/golang/src/github.com/coreos/rocket/bin# ./rkt run https://github.com/coreos/etcd/releases/download/v0.5.0-alpha.4/etcd-v0.5.0-alpha.4-linux-amd64.aci
run: error setting up stage0: error setting up image sha256-f9215c18b86f406c7cec4c7b45fd8752b5bfd1a492507d647821c2ce593fbf31: error opening app manifest: open /var/lib/rkt/containers/a17324cc-e2d5-4699-bb84-bbd208159fb3/stage1/opt/stage2/sha256-f9215c18b86f406c7cec4c7b45fd8752b5bfd1a492507d647821c2ce593fbf31/manifest: no such file or directory

But I'm guessing this is because the actual manifest file in the example image is not called "manifest" which rkt expects but rather "app" ? 👍

root@rocket:/opt/golang/src/github.com/coreos/rocket/bin# cat /var/lib/rkt/containers/a17324cc-e2d5-4699-bb84-bbd208159fb3/stage1/opt/stage2/sha256-f9215c18b86f406c7cec4c7b45fd8752b5bfd1a492507d647821c2ce593fbf31/app
{"acVersion":"1.0.0","acKind":"AppManifest","name":"coreos.com/etcd","version":"v0.5.0-alpha.4","os":"linux","arch":"amd64","exec":["/etcd"],"eventHandlers":null,"user":"0","group":"0","environment":{},"mountPoints":null,"ports":null,"isolators":null,"annotations":null}
@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 12, 2014

Contributor

@milosgajdos83 Since you're building rocket from source, it looks like you caught things in a bit of an inconsistent state. Everything should be back to normal now - would you mind walking through the guide again (preferably using the v0.1.1 release - but building from latest rocket+appc master should also work) and let us know how you go?

Contributor

jonboulle commented Dec 12, 2014

@milosgajdos83 Since you're building rocket from source, it looks like you caught things in a bit of an inconsistent state. Everything should be back to normal now - would you mind walking through the guide again (preferably using the v0.1.1 release - but building from latest rocket+appc master should also work) and let us know how you go?

@milosgajdos83

This comment has been minimized.

Show comment
Hide comment
@milosgajdos83

milosgajdos83 Dec 14, 2014

Hi,

I'm afraid latest build does not solve any of the issues I've reported:

root@rocket:/opt/rocket-v0.1.1# pwd
/opt/rocket-v0.1.1
root@rocket:/opt/rocket-v0.1.1# ./rkt run /opt/golang/src/rock/hello.aci
/etc/localtime is not a symlink, not updating container timezone.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounting /proc/sys/kernel/random/boot_id.
All filesystems unmounted.
Halting system.
root@rocket:/opt/rocket-v0.1.1#

I am assuming aci is valid based on the below:

root@rocket:/opt/golang/src/github.com/appc/spec# bin/actool validate /opt/golang/src/rock/hello.aci
root@rocket:/opt/golang/src/github.com/appc/spec#

But then again, I'm not getting any "feedback" from the tool - I'm assuming actool validate should return something like: app.aci: valid app container image kinda thing mentioned all over docs, but it does not. At least the actool's exit code is 0, so I'm assuming all is good with the image :-)

However as for running the example hello container, it's still a nogo for me.
The image looks OK to me:

root@rocket:/opt/golang/src/rock# tar tvf hello.aci
drwxr-xr-x 1000/1000         0 2014-12-14 22:38 rootfs
drwxr-xr-x 1000/1000         0 2014-12-14 23:08 rootfs/bin
-rwxr-xr-x 1000/1000   4449119 2014-12-14 22:38 rootfs/bin/hello
-rw-r--r-- root/root       502 2014-12-14 23:11 manifest
root@rocket:/opt/golang/src/rock#

Running rkt with debug on suggests that /bin/hello could not be found ?/

root@rocket:/opt/rocket-v0.1.1# ./rkt --debug run /opt/golang/src/rock/hello.aci
2014/12/14 23:09:16 Unpacking stage1 rootfs
2014/12/14 23:09:16 Writing stage1 init
2014/12/14 23:09:16 Wrote filesystem to /var/lib/rkt/containers/d12e2518-75af-4d8b-8e6b-7331e6f6de07
2014/12/14 23:09:16 Loading image sha256-83c8f7d659c110ee66c875579819d3f60928a06ed7aa945ba2299ef48998618a
2014/12/14 23:09:16 Writing container manifest
2014/12/14 23:09:17 Pivoting to filesystem /var/lib/rkt/containers/d12e2518-75af-4d8b-8e6b-7331e6f6de07
2014/12/14 23:09:17 Execing stage1/init
Spawning container stage1 on /var/lib/rkt/containers/d12e2518-75af-4d8b-8e6b-7331e6f6de07/stage1.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
systemd 215 running in system mode. (-PAM -AUDIT -SELINUX +IMA -SYSVINIT +LIBCRYPTSETUP -GCRYPT -ACL -XZ +SECCOMP -APPARMOR)
Detected virtualization 'systemd-nspawn'.
Detected architecture 'x86-64'.

Welcome to Linux!

Initializing machine ID from container UUID.
[  OK  ] Created slice -.slice.
[  OK  ] Created slice system.slice.
         Starting Graceful exit watcher...
[  OK  ] Started Graceful exit watcher.
         Starting coreos.com/hello...
[  OK  ] Started coreos.com/hello.
[  OK  ] Reached target Rocket apps target.
Failed at step EXEC spawning /bin/hello: No such file or directory
sha256-83c8f7d659c110ee66c875579819d3f60928a06ed7aa945ba2299ef48998618a.service: main process exited, code=exited, status=203/EXEC
Service exit-watcher.service is not needed anymore. Stopping.
Unit sha256-83c8f7d659c110ee66c875579819d3f60928a06ed7aa945ba2299ef48998618a.service entered failed state.
Triggering OnFailure= dependencies of sha256-83c8f7d659c110ee66c875579819d3f60928a06ed7aa945ba2299ef48998618a.service.
Shutting down.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounting /proc/sys/kernel/random/boot_id.
All filesystems unmounted.
Halting system.

Container stage1 has been shut down.
root@rocket:/opt/rocket-v0.1.1#

So I digged in and noticed that /bin/hello is NOT being copied over from rootfs subdirectory as per documentation but rather stays in it:

root@rocket:/var/lib/rkt/containers# cd 27052ad0-0bf0-458b-8839-b28fa188c94e/
root@rocket:/var/lib/rkt/containers/27052ad0-0bf0-458b-8839-b28fa188c94e# ls -ltr
total 12
-rwx------  1 root root  376 Dec 14 23:14 container
drwxr-xr-- 11 root root 4096 Dec 14 23:14 stage1
-rw-r-----  1 root root    5 Dec 14 23:14 pid
root@rocket:/var/lib/rkt/containers/27052ad0-0bf0-458b-8839-b28fa188c94e# find . -iname "hello"
./stage1/opt/stage2/sha256-df2005a9cff224ac4073a0d0ce71d0964e72d3149c0ae8fb6569ec7b60099aab/rootfs/bin/hello

I am assuming the behavior which I'm seeing with the example hello is a BUG ? Looks like when you have a subdirectory of rootfs of some sort, it does not copy over to the created container ?

I tested my theory on build another test image from a different layout - i.e. I left out bin bit and placed the hello binary directly into rootfs - obviously I have modified container manifest accordingly:

root@rocket:/opt/golang/src/rock# tree hello-layout/
hello-layout/
├── manifest
└── rootfs
    └── hello
root@rocket:/opt/golang/src/rock# grep \"hello\" hello-layout/manifest
            "hello"
root@rocket:/opt/golang/src/rock#

I built the image and ran the rocket:

root@rocket:/opt/rocket-v0.1.1# ./rkt --debug run /opt/golang/src/github.com/appc/spec/hello-test.aci
2014/12/14 23:32:33 Unpacking stage1 rootfs
2014/12/14 23:32:33 Writing stage1 init
2014/12/14 23:32:33 Wrote filesystem to /var/lib/rkt/containers/b85efddb-df1c-416f-856d-5497af232ed7
2014/12/14 23:32:33 Loading image sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7
2014/12/14 23:32:33 Writing container manifest
2014/12/14 23:32:33 Pivoting to filesystem /var/lib/rkt/containers/b85efddb-df1c-416f-856d-5497af232ed7
2014/12/14 23:32:33 Execing stage1/init
Spawning container stage1 on /var/lib/rkt/containers/b85efddb-df1c-416f-856d-5497af232ed7/stage1.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
systemd 215 running in system mode. (-PAM -AUDIT -SELINUX +IMA -SYSVINIT +LIBCRYPTSETUP -GCRYPT -ACL -XZ +SECCOMP -APPARMOR)
Detected virtualization 'systemd-nspawn'.
Detected architecture 'x86-64'.

Welcome to Linux!

Initializing machine ID from container UUID.
[/usr/lib64/systemd/system/sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7.service:11] Executable path is not absolute, ignoring: hello
sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7.service lacks ExecStart setting. Refusing.
Cannot add dependency job for unit sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7.service, ignoring: Operation refused, unit may not be isolated.
[  OK  ] Reached target Rocket apps target.
^C^C^C^]^]
Container stage1 terminated by signal KILL.

BOOM! All worked fine. So it seems that there is some problem with copying subdirectories of some sort. I did not dig deeper. But this explains why the etcd example works like charm - i.e. it has etcd binary directly in rootfs:

root@rocket:/opt/rocket-v0.1.1# tar tvf /var/lib/rkt/cas/blob/sha256/66/sha256-6635e9cbe18c6f51e8c70c143948df111b5626db39198182fbeb9277beb606db
drwxrwxr-x 1000/1000         0 2014-12-11 20:54 rootfs
drwxr-xr-x 1000/1000         0 2014-12-11 20:54 rootfs/Documentation
-rw-r--r-- 1000/1000      2266 2014-11-26 18:54 rootfs/Documentation/0_4_migration_tool.md
-rw-r--r-- 1000/1000      4438 2014-11-26 18:54 rootfs/Documentation/admin_guide.md
-rw-r--r-- 1000/1000     25262 2014-11-26 18:54 rootfs/Documentation/api.md
-rw-r--r-- 1000/1000     11013 2014-11-26 18:54 rootfs/Documentation/clustering.md
-rw-r--r-- 1000/1000      4414 2014-11-26 18:54 rootfs/Documentation/configuration.md
-rw-r--r-- 1000/1000       753 2014-11-26 18:54 rootfs/Documentation/glossary.md
-rw-r--r-- 1000/1000      2363 2014-11-26 18:54 rootfs/Documentation/other_apis.md
-rw-r--r-- 1000/1000      2751 2014-11-26 18:54 rootfs/Documentation/proxy.md
-rw-r--r-- 1000/1000      6829 2014-11-26 18:54 rootfs/Documentation/runtime-configuration.md
-rw-r--r-- 1000/1000      5186 2014-11-26 18:54 rootfs/README-etcdctl.md
-rw-r--r-- 1000/1000      4672 2014-11-26 18:54 rootfs/README.md
drwxrwxr-x 1000/1000         0 2014-12-11 20:54 rootfs/etc
-rw-rw-r-- 1000/1000        79 2014-12-11 20:57 rootfs/etc/hosts
-rwxr-xr-x 1000/1000   9029720 2014-11-26 18:54 rootfs/etcd
-rwxr-xr-x 1000/1000   7161080 2014-11-26 18:54 rootfs/etcd-migrate
-rwxr-xr-x 1000/1000   8936400 2014-11-26 18:54 rootfs/etcdctl
-rw-r--r-- root/root       377 2014-12-11 20:57 manifest
root@rocket:/opt/rocket-v0.1.1#

milosgajdos83 commented Dec 14, 2014

Hi,

I'm afraid latest build does not solve any of the issues I've reported:

root@rocket:/opt/rocket-v0.1.1# pwd
/opt/rocket-v0.1.1
root@rocket:/opt/rocket-v0.1.1# ./rkt run /opt/golang/src/rock/hello.aci
/etc/localtime is not a symlink, not updating container timezone.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounting /proc/sys/kernel/random/boot_id.
All filesystems unmounted.
Halting system.
root@rocket:/opt/rocket-v0.1.1#

I am assuming aci is valid based on the below:

root@rocket:/opt/golang/src/github.com/appc/spec# bin/actool validate /opt/golang/src/rock/hello.aci
root@rocket:/opt/golang/src/github.com/appc/spec#

But then again, I'm not getting any "feedback" from the tool - I'm assuming actool validate should return something like: app.aci: valid app container image kinda thing mentioned all over docs, but it does not. At least the actool's exit code is 0, so I'm assuming all is good with the image :-)

However as for running the example hello container, it's still a nogo for me.
The image looks OK to me:

root@rocket:/opt/golang/src/rock# tar tvf hello.aci
drwxr-xr-x 1000/1000         0 2014-12-14 22:38 rootfs
drwxr-xr-x 1000/1000         0 2014-12-14 23:08 rootfs/bin
-rwxr-xr-x 1000/1000   4449119 2014-12-14 22:38 rootfs/bin/hello
-rw-r--r-- root/root       502 2014-12-14 23:11 manifest
root@rocket:/opt/golang/src/rock#

Running rkt with debug on suggests that /bin/hello could not be found ?/

root@rocket:/opt/rocket-v0.1.1# ./rkt --debug run /opt/golang/src/rock/hello.aci
2014/12/14 23:09:16 Unpacking stage1 rootfs
2014/12/14 23:09:16 Writing stage1 init
2014/12/14 23:09:16 Wrote filesystem to /var/lib/rkt/containers/d12e2518-75af-4d8b-8e6b-7331e6f6de07
2014/12/14 23:09:16 Loading image sha256-83c8f7d659c110ee66c875579819d3f60928a06ed7aa945ba2299ef48998618a
2014/12/14 23:09:16 Writing container manifest
2014/12/14 23:09:17 Pivoting to filesystem /var/lib/rkt/containers/d12e2518-75af-4d8b-8e6b-7331e6f6de07
2014/12/14 23:09:17 Execing stage1/init
Spawning container stage1 on /var/lib/rkt/containers/d12e2518-75af-4d8b-8e6b-7331e6f6de07/stage1.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
systemd 215 running in system mode. (-PAM -AUDIT -SELINUX +IMA -SYSVINIT +LIBCRYPTSETUP -GCRYPT -ACL -XZ +SECCOMP -APPARMOR)
Detected virtualization 'systemd-nspawn'.
Detected architecture 'x86-64'.

Welcome to Linux!

Initializing machine ID from container UUID.
[  OK  ] Created slice -.slice.
[  OK  ] Created slice system.slice.
         Starting Graceful exit watcher...
[  OK  ] Started Graceful exit watcher.
         Starting coreos.com/hello...
[  OK  ] Started coreos.com/hello.
[  OK  ] Reached target Rocket apps target.
Failed at step EXEC spawning /bin/hello: No such file or directory
sha256-83c8f7d659c110ee66c875579819d3f60928a06ed7aa945ba2299ef48998618a.service: main process exited, code=exited, status=203/EXEC
Service exit-watcher.service is not needed anymore. Stopping.
Unit sha256-83c8f7d659c110ee66c875579819d3f60928a06ed7aa945ba2299ef48998618a.service entered failed state.
Triggering OnFailure= dependencies of sha256-83c8f7d659c110ee66c875579819d3f60928a06ed7aa945ba2299ef48998618a.service.
Shutting down.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounting /proc/sys/kernel/random/boot_id.
All filesystems unmounted.
Halting system.

Container stage1 has been shut down.
root@rocket:/opt/rocket-v0.1.1#

So I digged in and noticed that /bin/hello is NOT being copied over from rootfs subdirectory as per documentation but rather stays in it:

root@rocket:/var/lib/rkt/containers# cd 27052ad0-0bf0-458b-8839-b28fa188c94e/
root@rocket:/var/lib/rkt/containers/27052ad0-0bf0-458b-8839-b28fa188c94e# ls -ltr
total 12
-rwx------  1 root root  376 Dec 14 23:14 container
drwxr-xr-- 11 root root 4096 Dec 14 23:14 stage1
-rw-r-----  1 root root    5 Dec 14 23:14 pid
root@rocket:/var/lib/rkt/containers/27052ad0-0bf0-458b-8839-b28fa188c94e# find . -iname "hello"
./stage1/opt/stage2/sha256-df2005a9cff224ac4073a0d0ce71d0964e72d3149c0ae8fb6569ec7b60099aab/rootfs/bin/hello

I am assuming the behavior which I'm seeing with the example hello is a BUG ? Looks like when you have a subdirectory of rootfs of some sort, it does not copy over to the created container ?

I tested my theory on build another test image from a different layout - i.e. I left out bin bit and placed the hello binary directly into rootfs - obviously I have modified container manifest accordingly:

root@rocket:/opt/golang/src/rock# tree hello-layout/
hello-layout/
├── manifest
└── rootfs
    └── hello
root@rocket:/opt/golang/src/rock# grep \"hello\" hello-layout/manifest
            "hello"
root@rocket:/opt/golang/src/rock#

I built the image and ran the rocket:

root@rocket:/opt/rocket-v0.1.1# ./rkt --debug run /opt/golang/src/github.com/appc/spec/hello-test.aci
2014/12/14 23:32:33 Unpacking stage1 rootfs
2014/12/14 23:32:33 Writing stage1 init
2014/12/14 23:32:33 Wrote filesystem to /var/lib/rkt/containers/b85efddb-df1c-416f-856d-5497af232ed7
2014/12/14 23:32:33 Loading image sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7
2014/12/14 23:32:33 Writing container manifest
2014/12/14 23:32:33 Pivoting to filesystem /var/lib/rkt/containers/b85efddb-df1c-416f-856d-5497af232ed7
2014/12/14 23:32:33 Execing stage1/init
Spawning container stage1 on /var/lib/rkt/containers/b85efddb-df1c-416f-856d-5497af232ed7/stage1.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
systemd 215 running in system mode. (-PAM -AUDIT -SELINUX +IMA -SYSVINIT +LIBCRYPTSETUP -GCRYPT -ACL -XZ +SECCOMP -APPARMOR)
Detected virtualization 'systemd-nspawn'.
Detected architecture 'x86-64'.

Welcome to Linux!

Initializing machine ID from container UUID.
[/usr/lib64/systemd/system/sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7.service:11] Executable path is not absolute, ignoring: hello
sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7.service lacks ExecStart setting. Refusing.
Cannot add dependency job for unit sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7.service, ignoring: Operation refused, unit may not be isolated.
[  OK  ] Reached target Rocket apps target.
^C^C^C^]^]
Container stage1 terminated by signal KILL.

BOOM! All worked fine. So it seems that there is some problem with copying subdirectories of some sort. I did not dig deeper. But this explains why the etcd example works like charm - i.e. it has etcd binary directly in rootfs:

root@rocket:/opt/rocket-v0.1.1# tar tvf /var/lib/rkt/cas/blob/sha256/66/sha256-6635e9cbe18c6f51e8c70c143948df111b5626db39198182fbeb9277beb606db
drwxrwxr-x 1000/1000         0 2014-12-11 20:54 rootfs
drwxr-xr-x 1000/1000         0 2014-12-11 20:54 rootfs/Documentation
-rw-r--r-- 1000/1000      2266 2014-11-26 18:54 rootfs/Documentation/0_4_migration_tool.md
-rw-r--r-- 1000/1000      4438 2014-11-26 18:54 rootfs/Documentation/admin_guide.md
-rw-r--r-- 1000/1000     25262 2014-11-26 18:54 rootfs/Documentation/api.md
-rw-r--r-- 1000/1000     11013 2014-11-26 18:54 rootfs/Documentation/clustering.md
-rw-r--r-- 1000/1000      4414 2014-11-26 18:54 rootfs/Documentation/configuration.md
-rw-r--r-- 1000/1000       753 2014-11-26 18:54 rootfs/Documentation/glossary.md
-rw-r--r-- 1000/1000      2363 2014-11-26 18:54 rootfs/Documentation/other_apis.md
-rw-r--r-- 1000/1000      2751 2014-11-26 18:54 rootfs/Documentation/proxy.md
-rw-r--r-- 1000/1000      6829 2014-11-26 18:54 rootfs/Documentation/runtime-configuration.md
-rw-r--r-- 1000/1000      5186 2014-11-26 18:54 rootfs/README-etcdctl.md
-rw-r--r-- 1000/1000      4672 2014-11-26 18:54 rootfs/README.md
drwxrwxr-x 1000/1000         0 2014-12-11 20:54 rootfs/etc
-rw-rw-r-- 1000/1000        79 2014-12-11 20:57 rootfs/etc/hosts
-rwxr-xr-x 1000/1000   9029720 2014-11-26 18:54 rootfs/etcd
-rwxr-xr-x 1000/1000   7161080 2014-11-26 18:54 rootfs/etcd-migrate
-rwxr-xr-x 1000/1000   8936400 2014-11-26 18:54 rootfs/etcdctl
-rw-r--r-- root/root       377 2014-12-11 20:57 manifest
root@rocket:/opt/rocket-v0.1.1#
@vaibhavkhanduja

This comment has been minimized.

Show comment
Hide comment
@vaibhavkhanduja

vaibhavkhanduja Dec 15, 2014

Can you run with debug option? 
run --debug

 On Sunday, December 14, 2014 2:49 PM, Milos Gajdos <notifications@github.com> wrote:

Hi,I'm afraid latest build does not solve any of the issues I've reported:root@rocket:/opt/rocket-v0.1.1# pwd
/opt/rocket-v0.1.1
root@rocket:/opt/rocket-v0.1.1# ./rkt run /opt/golang/src/rock/hello.aci
/etc/localtime is not a symlink, not updating container timezone.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounting /proc/sys/kernel/random/boot_id.
All filesystems unmounted.
Halting system.
root@rocket:/opt/rocket-v0.1.1#
Does this have to do anything with me running this on non-systemd OS ? I'm trying to run this on Trusty like I mentioned above. I will try to test this on CoreOS at some point today, but like I said as for Trusty this still seems broken.I am assuming aci is valid based on the below:root@rocket:/opt/golang/src/github.com/appc/spec# bin/actool validate /opt/golang/src/rock/hello.aci
root@rocket:/opt/golang/src/github.com/appc/spec#
But then again, I'm not getting any feedback from the tool (as oppose to app.aci: valid app container image kinda thing mentioned all over docs, but at least the actool's exit code is 0, so I'm assuming all is good with the image ?—
Reply to this email directly or view it on GitHub.

vaibhavkhanduja commented Dec 15, 2014

Can you run with debug option? 
run --debug

 On Sunday, December 14, 2014 2:49 PM, Milos Gajdos <notifications@github.com> wrote:

Hi,I'm afraid latest build does not solve any of the issues I've reported:root@rocket:/opt/rocket-v0.1.1# pwd
/opt/rocket-v0.1.1
root@rocket:/opt/rocket-v0.1.1# ./rkt run /opt/golang/src/rock/hello.aci
/etc/localtime is not a symlink, not updating container timezone.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounting /proc/sys/kernel/random/boot_id.
All filesystems unmounted.
Halting system.
root@rocket:/opt/rocket-v0.1.1#
Does this have to do anything with me running this on non-systemd OS ? I'm trying to run this on Trusty like I mentioned above. I will try to test this on CoreOS at some point today, but like I said as for Trusty this still seems broken.I am assuming aci is valid based on the below:root@rocket:/opt/golang/src/github.com/appc/spec# bin/actool validate /opt/golang/src/rock/hello.aci
root@rocket:/opt/golang/src/github.com/appc/spec#
But then again, I'm not getting any feedback from the tool (as oppose to app.aci: valid app container image kinda thing mentioned all over docs, but at least the actool's exit code is 0, so I'm assuming all is good with the image ?—
Reply to this email directly or view it on GitHub.

@milosgajdos83

This comment has been minimized.

Show comment
Hide comment
@milosgajdos83

milosgajdos83 Dec 15, 2014

If you read my last reply on Github as oppose to via email, you'd see all the details - including -debug output indeed.

milosgajdos83 commented Dec 15, 2014

If you read my last reply on Github as oppose to via email, you'd see all the details - including -debug output indeed.

@milosgajdos83

This comment has been minimized.

Show comment
Hide comment
@milosgajdos83

milosgajdos83 Dec 15, 2014

What I'm talking about here is a STATICALLY LINKED Go binary - there are no depedencies, no script - its all compiled into one binary. You can find the link to the example here

milosgajdos83 commented Dec 15, 2014

What I'm talking about here is a STATICALLY LINKED Go binary - there are no depedencies, no script - its all compiled into one binary. You can find the link to the example here

@vcaputo

This comment has been minimized.

Show comment
Hide comment
@vcaputo

vcaputo Dec 15, 2014

Contributor

@milosgajdos83 correct me if I'm wrong, but judging from what you've posted above this appears to be an issue with actool build producing an aci containing a manifest with exec paths inconsistent with the aci's filesystem?

You were able to fix your test by moving the hello executable, presumably you also could have corrected it by editing the manifest so the exec path referred to the appropriate place?

I'm not sure if we're intentionally allowing the exec paths in the app manifest to point at potentially absent-at-build/validate-time executables, but it seems prudent that both build and validate should at least emit a warning about it.

We probably just need to make actool build and actool validate more robust around these details.

Contributor

vcaputo commented Dec 15, 2014

@milosgajdos83 correct me if I'm wrong, but judging from what you've posted above this appears to be an issue with actool build producing an aci containing a manifest with exec paths inconsistent with the aci's filesystem?

You were able to fix your test by moving the hello executable, presumably you also could have corrected it by editing the manifest so the exec path referred to the appropriate place?

I'm not sure if we're intentionally allowing the exec paths in the app manifest to point at potentially absent-at-build/validate-time executables, but it seems prudent that both build and validate should at least emit a warning about it.

We probably just need to make actool build and actool validate more robust around these details.

@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 15, 2014

Contributor

But then again, I'm not getting any "feedback" from the tool - I'm assuming actool validate should return something like: app.aci: valid app container image kinda thing mentioned all over docs, but it does not. At least the actool's exit code is 0, so I'm assuming all is good with the image :-)

This is a bug in the docs, we moved to quiet behaviour as the default - appc/spec#41

Contributor

jonboulle commented Dec 15, 2014

But then again, I'm not getting any "feedback" from the tool - I'm assuming actool validate should return something like: app.aci: valid app container image kinda thing mentioned all over docs, but it does not. At least the actool's exit code is 0, so I'm assuming all is good with the image :-)

This is a bug in the docs, we moved to quiet behaviour as the default - appc/spec#41

@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 15, 2014

Contributor

So I digged in and noticed that /bin/hello is NOT being copied over from rootfs subdirectory as per documentation but rather stays in it:

I don't understand what you mean here, can you explain where the documentation indicates that /bin/hello would be copied from the rootfs directory elsewhere?

Contributor

jonboulle commented Dec 15, 2014

So I digged in and noticed that /bin/hello is NOT being copied over from rootfs subdirectory as per documentation but rather stays in it:

I don't understand what you mean here, can you explain where the documentation indicates that /bin/hello would be copied from the rootfs directory elsewhere?

@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 15, 2014

Contributor

My apologies but I had also missed updating a couple of things in the getting started guide, since resolved here

I will try again to reproduce this on trusty now.

Contributor

jonboulle commented Dec 15, 2014

My apologies but I had also missed updating a couple of things in the getting started guide, since resolved here

I will try again to reproduce this on trusty now.

@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 15, 2014

Contributor

I still cannot reproduce:

root@vagrant-ubuntu-trusty-64:/home/vagrant# tar zvtf hello.aci 
drwxrwxr-x 1000/1000         0 2014-12-15 21:00 rootfs
drwxrwxr-x 1000/1000         0 2014-12-15 21:00 rootfs/bin
-rwxrwxr-x 1000/1000   4383380 2014-12-15 21:00 rootfs/bin/hello
-rw-r--r-- root/root       502 2014-12-15 21:00 manifest
root@vagrant-ubuntu-trusty-64:/home/vagrant# rocket-v0.1.1/rkt -debug run hello.aci 
2014/12/15 21:23:09 Unpacking stage1 rootfs
2014/12/15 21:23:10 Writing stage1 init
2014/12/15 21:23:10 Wrote filesystem to /var/lib/rkt/containers/0c74f8ed-6cf8-4374-a8ec-01d3a07e9b1a
2014/12/15 21:23:10 Loading image sha256-dc349b4972c0712352415deae8e7151fffc8d6ce2b5f44dfc4413e2302aab9c5
2014/12/15 21:23:10 Writing container manifest
2014/12/15 21:23:10 Pivoting to filesystem /var/lib/rkt/containers/0c74f8ed-6cf8-4374-a8ec-01d3a07e9b1a
2014/12/15 21:23:10 Execing stage1/init
Spawning container stage1 on /var/lib/rkt/containers/0c74f8ed-6cf8-4374-a8ec-01d3a07e9b1a/stage1.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
systemd 215 running in system mode. (-PAM -AUDIT -SELINUX +IMA -SYSVINIT +LIBCRYPTSETUP -GCRYPT -ACL -XZ +SECCOMP -APPARMOR)
Detected virtualization 'systemd-nspawn'.
Detected architecture 'x86-64'.

Welcome to Linux!

Initializing machine ID from container UUID.
[  OK  ] Created slice -.slice.
[  OK  ] Created slice system.slice.
         Starting Graceful exit watcher...
[  OK  ] Started Graceful exit watcher.
         Starting coreos.com/hello...
[  OK  ] Started coreos.com/hello.
[  OK  ] Reached target Rocket apps target.

^]^]
Container stage1 terminated by signal KILL.
root@vagrant-ubuntu-trusty-64:/home/vagrant# find /var/lib/rkt/containers/0c74f8ed-6cf8-4374-a8ec-01d3a07e9b1a -name hello
/var/lib/rkt/containers/0c74f8ed-6cf8-4374-a8ec-01d3a07e9b1a/stage1/opt/stage2/sha256-dc349b4972c0712352415deae8e7151fffc8d6ce2b5f44dfc4413e2302aab9c5/rootfs/bin/hello
root@vagrant-ubuntu-trusty-64:/home/vagrant# 
Contributor

jonboulle commented Dec 15, 2014

I still cannot reproduce:

root@vagrant-ubuntu-trusty-64:/home/vagrant# tar zvtf hello.aci 
drwxrwxr-x 1000/1000         0 2014-12-15 21:00 rootfs
drwxrwxr-x 1000/1000         0 2014-12-15 21:00 rootfs/bin
-rwxrwxr-x 1000/1000   4383380 2014-12-15 21:00 rootfs/bin/hello
-rw-r--r-- root/root       502 2014-12-15 21:00 manifest
root@vagrant-ubuntu-trusty-64:/home/vagrant# rocket-v0.1.1/rkt -debug run hello.aci 
2014/12/15 21:23:09 Unpacking stage1 rootfs
2014/12/15 21:23:10 Writing stage1 init
2014/12/15 21:23:10 Wrote filesystem to /var/lib/rkt/containers/0c74f8ed-6cf8-4374-a8ec-01d3a07e9b1a
2014/12/15 21:23:10 Loading image sha256-dc349b4972c0712352415deae8e7151fffc8d6ce2b5f44dfc4413e2302aab9c5
2014/12/15 21:23:10 Writing container manifest
2014/12/15 21:23:10 Pivoting to filesystem /var/lib/rkt/containers/0c74f8ed-6cf8-4374-a8ec-01d3a07e9b1a
2014/12/15 21:23:10 Execing stage1/init
Spawning container stage1 on /var/lib/rkt/containers/0c74f8ed-6cf8-4374-a8ec-01d3a07e9b1a/stage1.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
systemd 215 running in system mode. (-PAM -AUDIT -SELINUX +IMA -SYSVINIT +LIBCRYPTSETUP -GCRYPT -ACL -XZ +SECCOMP -APPARMOR)
Detected virtualization 'systemd-nspawn'.
Detected architecture 'x86-64'.

Welcome to Linux!

Initializing machine ID from container UUID.
[  OK  ] Created slice -.slice.
[  OK  ] Created slice system.slice.
         Starting Graceful exit watcher...
[  OK  ] Started Graceful exit watcher.
         Starting coreos.com/hello...
[  OK  ] Started coreos.com/hello.
[  OK  ] Reached target Rocket apps target.

^]^]
Container stage1 terminated by signal KILL.
root@vagrant-ubuntu-trusty-64:/home/vagrant# find /var/lib/rkt/containers/0c74f8ed-6cf8-4374-a8ec-01d3a07e9b1a -name hello
/var/lib/rkt/containers/0c74f8ed-6cf8-4374-a8ec-01d3a07e9b1a/stage1/opt/stage2/sha256-dc349b4972c0712352415deae8e7151fffc8d6ce2b5f44dfc4413e2302aab9c5/rootfs/bin/hello
root@vagrant-ubuntu-trusty-64:/home/vagrant# 
@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 15, 2014

Contributor

BOOM! All worked fine. So it seems that there is some problem with copying subdirectories of some sort. I did not dig deeper.

Ah, I missed this on my first read through. Note that this example is not actually working fine at all:

[/usr/lib64/systemd/system/sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7.service:11] Executable path is not absolute, ignoring: hello
sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7.service lacks ExecStart setting. Refusing.

Presumably at this stage you had "hello" in your image manifest (instead of "/hello"). So this means it did not really verify that non-nested paths worked.

Contributor

jonboulle commented Dec 15, 2014

BOOM! All worked fine. So it seems that there is some problem with copying subdirectories of some sort. I did not dig deeper.

Ah, I missed this on my first read through. Note that this example is not actually working fine at all:

[/usr/lib64/systemd/system/sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7.service:11] Executable path is not absolute, ignoring: hello
sha256-1b340c11cd1a6c688f4d299da34f82ce4efb819340f368442badb31607dd0fb7.service lacks ExecStart setting. Refusing.

Presumably at this stage you had "hello" in your image manifest (instead of "/hello"). So this means it did not really verify that non-nested paths worked.

@nadrimajstor

This comment has been minimized.

Show comment
Hide comment
@nadrimajstor

nadrimajstor Dec 15, 2014

I have the same issue on trusty.
Stating absolute path "/bin/hello" in the manifest leads to
Failed at step EXEC spawning /bin/hello: No such file or directory
and immediate exit from rkt.

Stating relative path leads to
[/usr/lib64/systemd/system/sha256-d9b4368a95ed2e128762408893f23ef9f8d8cefd2bd55a46d9753cf38c1d44ba.service:11] Executable path is not absolute, ignoring: bin/hello sha256-d9b4368a95ed2e128762408893f23ef9f8d8cefd2bd55a46d9753cf38c1d44ba.service lacks ExecStart setting. Refusing. Cannot add dependency job for unit sha256-d9b4368a95ed2e128762408893f23ef9f8d8cefd2bd55a46d9753cf38c1d44ba.service, ignoring: Operation refused, unit may not be isolated.
and rkt waits for triple ^] to exit.

In both cases kernel emits cgroup: new mount options do not match the existing superblock, will be ignored

nadrimajstor commented Dec 15, 2014

I have the same issue on trusty.
Stating absolute path "/bin/hello" in the manifest leads to
Failed at step EXEC spawning /bin/hello: No such file or directory
and immediate exit from rkt.

Stating relative path leads to
[/usr/lib64/systemd/system/sha256-d9b4368a95ed2e128762408893f23ef9f8d8cefd2bd55a46d9753cf38c1d44ba.service:11] Executable path is not absolute, ignoring: bin/hello sha256-d9b4368a95ed2e128762408893f23ef9f8d8cefd2bd55a46d9753cf38c1d44ba.service lacks ExecStart setting. Refusing. Cannot add dependency job for unit sha256-d9b4368a95ed2e128762408893f23ef9f8d8cefd2bd55a46d9753cf38c1d44ba.service, ignoring: Operation refused, unit may not be isolated.
and rkt waits for triple ^] to exit.

In both cases kernel emits cgroup: new mount options do not match the existing superblock, will be ignored

@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 15, 2014

Contributor

@nadrimajstor can you provide the output of tar tzvf hello.aci and find /var/lib/rkt/containers -iname hello.aci ?

Contributor

jonboulle commented Dec 15, 2014

@nadrimajstor can you provide the output of tar tzvf hello.aci and find /var/lib/rkt/containers -iname hello.aci ?

@nadrimajstor

This comment has been minimized.

Show comment
Hide comment
@nadrimajstor

nadrimajstor Dec 15, 2014

ivan@ikahome:~/rocket$ tar tzvf hello.aci
drwxrwxr-x 1000/1000         0 2014-12-15 22:14 rootfs
drwxrwxr-x 1000/1000         0 2014-12-15 22:10 rootfs/bin
-rwxrwxr-x 1000/1000   4449072 2014-12-15 22:10 rootfs/bin/hello
-rw-r--r-- root/root       502 2014-12-15 22:16 manifest
ivan@ikahome:~/rocket$ sudo find /var/lib/rkt/containers -iname hello.aci
ivan@ikahome:~/rocket$ 

nadrimajstor commented Dec 15, 2014

ivan@ikahome:~/rocket$ tar tzvf hello.aci
drwxrwxr-x 1000/1000         0 2014-12-15 22:14 rootfs
drwxrwxr-x 1000/1000         0 2014-12-15 22:10 rootfs/bin
-rwxrwxr-x 1000/1000   4449072 2014-12-15 22:10 rootfs/bin/hello
-rw-r--r-- root/root       502 2014-12-15 22:16 manifest
ivan@ikahome:~/rocket$ sudo find /var/lib/rkt/containers -iname hello.aci
ivan@ikahome:~/rocket$ 
@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 15, 2014

Contributor

@nadrimajstor as @philips pointed out in #286 this could also very possibly be because the file is not completely statically linked - can you run ldd against rootfs/bin/hello?

Contributor

jonboulle commented Dec 15, 2014

@nadrimajstor as @philips pointed out in #286 this could also very possibly be because the file is not completely statically linked - can you run ldd against rootfs/bin/hello?

@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 15, 2014

Contributor

@milosgajdos83 I understand that this ended up being because of #286 so I am going to close this out. Sorry for the runaround!

Contributor

jonboulle commented Dec 15, 2014

@milosgajdos83 I understand that this ended up being because of #286 so I am going to close this out. Sorry for the runaround!

@jonboulle jonboulle closed this Dec 15, 2014

@nadrimajstor

This comment has been minimized.

Show comment
Hide comment
@nadrimajstor

nadrimajstor Dec 15, 2014

Correct. I have:

ivan@ikahome:~/rocket$ ldd hello-layout/rootfs/bin/hello 
    linux-vdso.so.1 =>  (0x00007ffff47fe000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f93261eb000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9325e25000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f932643a000)

Please do provide correct instructions on how to build completely statically linked binary file hello.
And @jonboulle thank you for your time and patience.

nadrimajstor commented Dec 15, 2014

Correct. I have:

ivan@ikahome:~/rocket$ ldd hello-layout/rootfs/bin/hello 
    linux-vdso.so.1 =>  (0x00007ffff47fe000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f93261eb000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9325e25000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f932643a000)

Please do provide correct instructions on how to build completely statically linked binary file hello.
And @jonboulle thank you for your time and patience.

@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 15, 2014

Contributor

@nadrimajstor check out the doc here - you will need to run:

$ CGO_ENABLED=0 GOOS=linux go build -a -tags netgo -ldflags '-w' .
Contributor

jonboulle commented Dec 15, 2014

@nadrimajstor check out the doc here - you will need to run:

$ CGO_ENABLED=0 GOOS=linux go build -a -tags netgo -ldflags '-w' .
@nadrimajstor

This comment has been minimized.

Show comment
Hide comment
@nadrimajstor

nadrimajstor Dec 15, 2014

I'm sorry if my knowledge of English is not sufficient. With "Please do provide correct instructions" I've meant: Followed exactly as in the docs and ended up with dynamic libc.

nadrimajstor commented Dec 15, 2014

I'm sorry if my knowledge of English is not sufficient. With "Please do provide correct instructions" I've meant: Followed exactly as in the docs and ended up with dynamic libc.

@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 15, 2014

Contributor

@kelseyhightower any ideas ^ ?

Contributor

jonboulle commented Dec 15, 2014

@kelseyhightower any ideas ^ ?

@kelseyhightower

This comment has been minimized.

Show comment
Hide comment
@kelseyhightower

kelseyhightower Dec 15, 2014

Contributor

@jonboulle Reviewing now.

Contributor

kelseyhightower commented Dec 15, 2014

@jonboulle Reviewing now.

@kelseyhightower

This comment has been minimized.

Show comment
Hide comment
@kelseyhightower

kelseyhightower Dec 15, 2014

Contributor

@nadrimajstor Can we chat in IRC? #coreos on freenode.

Contributor

kelseyhightower commented Dec 15, 2014

@nadrimajstor Can we chat in IRC? #coreos on freenode.

@nadrimajstor

This comment has been minimized.

Show comment
Hide comment
@nadrimajstor

nadrimajstor Dec 16, 2014

After a (more than) few tries and fails, this

go build -ldflags '-linkmode external -extldflags "-static"' hello.go

produced a statically-linked hello binary that (for the purpose of demo) works fine.
Why the -tags netgo is ignored and pthread and libc get dynamically linked is beyond me. 😕

edit: This applies to go 1.4 on trusty.

nadrimajstor commented Dec 16, 2014

After a (more than) few tries and fails, this

go build -ldflags '-linkmode external -extldflags "-static"' hello.go

produced a statically-linked hello binary that (for the purpose of demo) works fine.
Why the -tags netgo is ignored and pthread and libc get dynamically linked is beyond me. 😕

edit: This applies to go 1.4 on trusty.

@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Dec 16, 2014

Contributor

@kelseyhightower so should we just add -extldflags "-static" to the example? I'm a bit confused now which arguments are required.

Contributor

jonboulle commented Dec 16, 2014

@kelseyhightower so should we just add -extldflags "-static" to the example? I'm a bit confused now which arguments are required.

@milosgajdos83

This comment has been minimized.

Show comment
Hide comment
@milosgajdos83

milosgajdos83 Dec 17, 2014

CGO_ENABLED=0 go build -a does the trick for me - as well as . But this only works in go 1.3. Apparently -a flag no longer takes effect on the standard library in 1.4 so atm this can be tricky or impossible on the latest release.

milosgajdos83 commented Dec 17, 2014

CGO_ENABLED=0 go build -a does the trick for me - as well as . But this only works in go 1.3. Apparently -a flag no longer takes effect on the standard library in 1.4 so atm this can be tricky or impossible on the latest release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment