New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[os-images] OpenISCSI service crashes on startup with "no such container" message #2405

Closed
jiphex opened this Issue Jul 10, 2018 · 7 comments

Comments

Projects
None yet
4 participants
@jiphex

jiphex commented Jul 10, 2018

RancherOS Version: v1.4.0 latest

Where are you running RancherOS? VMWare vSphere (via Rancher v2)

I've enabled OpenISCSI using the commands specified in rancher/os-services#110, but it appears that OpenISCSI doesn't start. The output from ros s logs open-iscsi is here, with the final error below:

open-iscsi_1 | installing wonka bin links in console
open-iscsi_1 | + /setup_wonka.sh iscsi-tools
open-iscsi_1 | installing wonka bin links in iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
@jiphex

This comment has been minimized.

jiphex commented Jul 10, 2018

Closing this, it works on v1.4.0 I think. It's basically now a dup of #1967

@jiphex jiphex closed this Jul 10, 2018

@jiphex jiphex reopened this Jul 10, 2018

@jiphex

This comment has been minimized.

jiphex commented Jul 10, 2018

After some more testing, this is still broken on v1.4.0/latest. I'll do some more debugging.

@jiphex jiphex changed the title from OpenISCSI service crashes on startup with "no such container" message to [os-images] OpenISCSI service crashes on startup with "no such container" message Jul 10, 2018

@niusmallnan

This comment has been minimized.

Member

niusmallnan commented Jul 11, 2018

Here are my steps, it works on v1.4.0:

# ros service enable open-iscsi

# ros service up open-iscsi

# ros s logs open-iscsi
...
...
open-iscsi_1 | + modprobe iscsi_tcp
open-iscsi_1 | + /dist/arch/setup_wonka.sh console
open-iscsi_1 | installing wonka bin links in console
open-iscsi_1 | + touch /lib/modules/4.14.32-rancher2/.open-iscsi-done
open-iscsi_1 | + echo open-iscsi for 4.14.32-rancher2 installed. Delete /lib/modules/4.14.32-rancher2/.open-iscsi-done to reinstall
open-iscsi_1 | open-iscsi for 4.14.32-rancher2 installed. Delete /lib/modules/4.14.32-rancher2/.open-iscsi-done to reinstall

I personally recommend you switch to ubuntu console, then install open-iscsi by apt.

@mciupak

This comment has been minimized.

mciupak commented Jul 24, 2018

I do confirm that it does not work. It installs properly but after reboot there is following issue:

# ros s logs open-iscsi
...
...
open-iscsi_1 | + uname -r
open-iscsi_1 | open-iscsi for 4.14.32-rancher2
open-iscsi_1 | + KERNEL_VERSION=4.14.32-rancher2
open-iscsi_1 | + echo open-iscsi for 4.14.32-rancher2
open-iscsi_1 | + DIR=/lib/modules/4.14.32-rancher2/build
open-iscsi_1 | + STAMP=/lib/modules/4.14.32-rancher2/.open-iscsi-done
open-iscsi_1 | + [ -e /lib/modules/4.14.32-rancher2/.open-iscsi-done ]
open-iscsi_1 | + modprobe iscsi_tcp
open-iscsi_1 | + system-docker run --rm iscsi-tools cat /setup_wonka.sh
open-iscsi_1 | installing wonka bin links in console
open-iscsi_1 | + chmod 755 /setup_wonka.sh
open-iscsi_1 | + /setup_wonka.sh console
open-iscsi_1 | + /setup_wonka.sh iscsi-tools
open-iscsi_1 | installing wonka bin links in iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
open-iscsi_1 | Error response from daemon: No such container: iscsi-tools
# uname -a
Linux hub 4.14.32-rancher2 #1 SMP Fri May 11 11:30:31 UTC 2018 x86_64 GNU/Linux
# ros os version
v1.4.0
# system-docker container ls -a
CONTAINER ID        IMAGE                                 COMMAND                  CREATED                  STATUS                                PORTS               NAMES
37cdb118e257        rancher/os-console:v1.4.0             "/usr/bin/ros entr..."   Less than a second ago   Up Less than a second                                     console
c41397fa2c88        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   Less than a second ago   Created                                                   all-volumes
a6e5d19d5f03        rancher/os-docker:18.03.1             "ros user-docker"        7 minutes ago            Up Less than a second                                     docker
b1045ffb1879        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   7 minutes ago            Exited (0) Less than a second ago                         preload-user-images
64d63b333ccc        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   7 minutes ago            Exited (0) Less than a second ago                         cloud-init-execute
31a4902b0994        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   7 minutes ago            Up Less than a second                                     ntp
7d7248e9c032        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   7 minutes ago            Up Less than a second                                     network
53ce96b84396        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   7 minutes ago            Up Less than a second                                     udev
06729b0e69d1        rancher/container-crontab:v0.4.0      "container-crontab"      7 minutes ago            Up Less than a second                                     system-cron
9c037d59d691        rancher/os-acpid:v1.4.0               "/usr/bin/ros entr..."   7 minutes ago            Up Less than a second                                     acpid
05abc1fbeed4        rancher/os-syslog:v1.4.0              "/usr/bin/entrypoi..."   7 minutes ago            Up Less than a second                                     syslog
fb766edfcd80        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   7 minutes ago            Exited (0) Less than a second ago                         udev-cold
2f126d5933c6        rancher/os-logrotate:v1.4.0           "/usr/bin/entrypoi..."   7 minutes ago            Exited (0) About an hour ago                              logrotate
318fea719a3f        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   7 minutes ago            Created                                                   media-volumes
fe22ce54fb51        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   7 minutes ago            Created                                                   user-volumes
117fd25fa942        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   7 minutes ago            Created                                                   container-data-volumes
c4990ec064dd        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   7 minutes ago            Created                                                   system-volumes
e84aecfc8c59        rancher/os-base:v1.4.0                "/usr/bin/ros entr..."   7 minutes ago            Created                                                   command-volumes
b0f0ab214a85        rancher/os-console:v1.4.0             "/usr/bin/ros entr..."   14 minutes ago           Exited (255) Less than a second ago                       reboot
0b208e676987        rancher/os-iscsi:v1.4.0-rc1           "/usr/bin/ros entr..."   18 minutes ago           Exited (1) Less than a second ago                         open-iscsi
30cf6ee3490a        rancher/os-headers:4.14.32-rancher2   "/headers.sh"            2 hours ago              Exited (0) Less than a second ago                         kernel-headers-system-docker
# system-docker image ls -a
REPOSITORY                  TAG                 IMAGE ID            CREATED             SIZE
iscsi-tools                 latest              78c3c65a9bf6        19 minutes ago      167MB
<none>                      <none>              8f78304bd0dd        19 minutes ago      167MB
<none>                      <none>              67d58a83afe5        19 minutes ago      167MB
<none>                      <none>              981a9ce7b59c        19 minutes ago      167MB
<none>                      <none>              6afd01cb1535        19 minutes ago      167MB
<none>                      <none>              ea2bb1c3a952        19 minutes ago      163MB
<none>                      <none>              8022e7dfccca        19 minutes ago      160MB
<none>                      <none>              c6ab87f883e5        19 minutes ago      163MB
<none>                      <none>              7f7c36307e76        19 minutes ago      163MB
<none>                      <none>              30cb5d0e11dd        19 minutes ago      160MB
ubuntu                      16.04               e13f3d529b1a        7 days ago          115MB
rancher/os-iscsi            v1.4.0-rc1          a9dcddc6f599        6 weeks ago         1.46GB
rancher/os-syslog           v1.4.0              dc7101232763        7 weeks ago         39.8MB
rancher/os-logrotate        v1.4.0              87ce723acdc5        7 weeks ago         39.8MB
rancher/os-console          v1.4.0              98809951ed29        7 weeks ago         39.8MB
rancher/os-acpid            v1.4.0              19d5a24ed6bd        7 weeks ago         39.8MB
rancher/os-bootstrap        v1.4.0              74eaa18b4dae        7 weeks ago         39.8MB
rancher/os-base             v1.4.0              b5fedf79588c        7 weeks ago         39.8MB
rancher/os-docker           18.03.1             291710cafacd        2 months ago        131MB
rancher/os-headers          4.14.32-rancher2    c6e13239dde6        2 months ago        24.5MB
rancher/container-crontab   v0.4.0              7a2485d285d9        6 months ago        12.9MB

Any ideas?

@niusmallnan

This comment has been minimized.

Member

niusmallnan commented Jul 26, 2018

There is a problem here: /setup_wonka.sh iscsi-tools.
There is no need to setup wonka script to iscsi-tools, I have removed it.

@rootwuj

This comment has been minimized.

rootwuj commented Aug 28, 2018

This issue has been fixed in RancherOS 1.4.1-rc2, no error after reboot.

@rootwuj rootwuj closed this Aug 28, 2018

@niusmallnan niusmallnan added BACKPORTED and removed BACKPORT labels Sep 16, 2018

@niusmallnan

This comment has been minimized.

Member

niusmallnan commented Sep 19, 2018

Backported to v1.4.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment