Error running in Centos 7 and Debian 8

[dploeger]

What you expected to happen?

Containers start well on Centos 7 and Debian 8 just like they do on Debian 9, Ubuntu 16.04 and 18.04.

What happened?

The container isn't started by systemd. This error message pops up in journal:

Jul 05 07:25:54 default-centos-7.vagrantup.com systemd[1]: Started Daemon for hana-testhana1.
Jul 05 07:25:55 default-centos-7.vagrantup.com docker-hana-testhana1[8468]: /usr/bin/docker: invalid reference format.
Jul 05 07:25:55 default-centos-7.vagrantup.com docker-hana-testhana1[8468]: See '/usr/bin/docker run --help'.

Copying and pasting the ExecStart command directly to the terminal works. So I believe, this is some weird escaping issue with systemd on Centos 7 and Debian 8. I propose writing a startup script and not simply pasting the docker run command into ExecStart.

How to reproduce it?

Configure the module like this:

  class {
  'hanaexpress':
    store_username => 'some-docker-store-user',
    store_password => 'its-password'
}
  docker::run {
    "hana-testhana1":
      hostname => "hana-testhana1",
      image => "store/saplabs/hanaexpress:2.00.030.00.20180403.2",
      ports => [
        "39017:39017",
        "39041:39041"
      ],
      volumes => [
        "/data/testhana1:/hana/mounts"
      ],
      extra_parameters => [
        "--ulimit nofile=1048576:1048576",
        "--sysctl kernel.shmmax=1073741824",
        "--sysctl net.ipv4.ip_local_port_range=\"40000 60999\"",
        "--sysctl kernel.shmmni=524288",
        "--sysctl kernel.shmall=8388608",
      ],
      command => "--passwords-url file:///hana/mounts/password.json --agree-to-sap-license"
  }

This is based on our SAP HANA Express puppet module. (We're using the SAP HANA Express docker image here)

Anything else we need to know?

This is the systemd service unit generated by the module:

# This file is managed by Puppet and local changes
# may be overwritten

[Unit]
Description=Daemon for hana-testhana1
After=docker.service
Wants=
Requires=docker.service

[Service]
Restart=on-failure
StartLimitInterval=20
StartLimitBurst=5
TimeoutStartSec=0
RestartSec=5
Environment="HOME=/root"
SyslogIdentifier=docker-hana-testhana1
ExecStartPre=-/usr/bin/docker kill hana-testhana1
ExecStartPre=-/usr/bin/docker rm  hana-testhana1

ExecStart=/usr/bin/docker run \
        -h 'hana-testhana1' --net bridge -m 0b -p 39017:39017 \
 -p 39041:39041 \
 -v /data/testhana1:/hana/mounts \
 --ulimit nofile=1048576:1048576 --sysctl kernel.shmmax=1073741824 --sysctl net.ipv4.ip_local_port_range="40000 60999" --sysctl kernel.shmmni=524288 --sysctl kernel.shmall=8388608 \
        --name hana-testhana1 \
        store/saplabs/hanaexpress:2.00.030.00.20180403.2 \
         --passwords-url file:///hana/mounts/password.json --agree-to-sap-license
ExecStop=-/usr/bin/docker stop --time=0 hana-testhana1
ExecStop=-/usr/bin/docker rm  hana-testhana1

[Install]
WantedBy=multi-user.target

Versions:

[root@default-centos-7 ~]# puppet --version
docker ve5.5.2
[root@default-centos-7 ~]# docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:20:16 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:23:58 2018
  OS/Arch:      linux/amd64
  Experimental: false
[root@default-centos-7 ~]# facter os
{
  architecture => "x86_64",
  family => "RedHat",
  hardware => "x86_64",
  name => "CentOS",
  release => {
    full => "7.4.1708",
    major => "7",
    minor => "4"
  },
  selinux => {
    config_mode => "permissive",
    config_policy => "targeted",
    current_mode => "permissive",
    enabled => true,
    enforced => false,
    policy_version => "28"
  }
}
[root@default-centos-7 ~]# puppet module list
/etc/puppetlabs/code/environments/production/modules (no modules installed)
/etc/puppetlabs/code/modules (no modules installed)
/opt/puppetlabs/puppet/modules (no modules installed)
[root@default-centos-7 ~]# puppet module list --modulepath=/tmp/kitchen/modules
/tmp/kitchen/modules
├── dodevops-hanaexpress (v0.1.0)
├── herculesteam-augeasproviders_core (v2.1.4)
├── herculesteam-augeasproviders_sysctl (v2.2.0)
├── puppetlabs-apt (v4.5.1)
├── puppetlabs-docker (v1.1.0)
├── puppetlabs-stdlib (v4.25.1)
└── puppetlabs-translate (v1.0.0)

[davejrt]

Hi @dploeger I can't replicate that issue.

With the following in a manifest

node default {
 class {'docker':}

  docker::run {'nginx':
    ports => ['80:80'],
    image => 'nginx:alpine',
    pull_on_start => true,
  }

}

I get the following output:

root@node-01:~# facter os
{
  architecture => "amd64",
  distro => {
    codename => "jessie",
    description => "Debian GNU/Linux 8.6 (jessie)",
    id => "Debian",
    release => {
      full => "8.6",
      major => "8",
      minor => "6"
    }
  },
  family => "Debian",
  hardware => "x86_64",
  name => "Debian",
  release => {
    full => "8.6",
    major => "8",
    minor => "6"
  },
  selinux => {
    enabled => false
  }
}
root@node-01:~# journal | grep nginx
-bash: journal: command not found
root@node-01:~# journalctl | grep nginx
Jul 05 23:12:18 node-01 systemd[1]: Configuration file /etc/systemd/system/docker-nginx.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Jul 05 23:12:18 node-01 systemd[1]: Configuration file /etc/systemd/system/docker-nginx.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Jul 05 23:12:18 node-01 systemd[1]: Starting Daemon for nginx...
Jul 05 23:12:18 node-01 docker-nginx[9587]: Error response from daemon: Cannot kill container: nginx: No such container: nginx
Jul 05 23:12:18 node-01 docker-nginx[9591]: Error: No such container: nginx
Jul 05 23:12:20 node-01 docker-nginx[9597]: alpine: Pulling from library/nginx
Jul 05 23:12:20 node-01 docker-nginx[9597]: ff3a5c916c92: Pulling fs layer
Jul 05 23:12:20 node-01 docker-nginx[9597]: c09b62762e87: Pulling fs layer
Jul 05 23:12:20 node-01 docker-nginx[9597]: f69f407b9820: Pulling fs layer
Jul 05 23:12:20 node-01 docker-nginx[9597]: f51f7ad16a88: Pulling fs layer
Jul 05 23:12:20 node-01 docker-nginx[9597]: f51f7ad16a88: Waiting
Jul 05 23:12:22 node-01 docker-nginx[9597]: f69f407b9820: Verifying Checksum
Jul 05 23:12:22 node-01 docker-nginx[9597]: f69f407b9820: Download complete
Jul 05 23:12:22 node-01 docker-nginx[9597]: ff3a5c916c92: Verifying Checksum
Jul 05 23:12:22 node-01 docker-nginx[9597]: ff3a5c916c92: Download complete
Jul 05 23:12:22 node-01 docker-nginx[9597]: ff3a5c916c92: Pull complete
Jul 05 23:12:23 node-01 docker-nginx[9597]: f51f7ad16a88: Verifying Checksum
Jul 05 23:12:23 node-01 docker-nginx[9597]: f51f7ad16a88: Download complete
Jul 05 23:12:23 node-01 docker-nginx[9597]: c09b62762e87: Verifying Checksum
Jul 05 23:12:23 node-01 docker-nginx[9597]: c09b62762e87: Download complete
Jul 05 23:12:23 node-01 docker-nginx[9597]: c09b62762e87: Pull complete
Jul 05 23:12:23 node-01 docker-nginx[9597]: f69f407b9820: Pull complete
Jul 05 23:12:23 node-01 docker-nginx[9597]: f51f7ad16a88: Pull complete
Jul 05 23:12:23 node-01 docker-nginx[9597]: Digest: sha256:23b4db9adcbd3d46e8e792ec30c2a4d69e34fe35dfff9147776133a9b105b46c
Jul 05 23:12:23 node-01 docker-nginx[9597]: Status: Downloaded newer image for nginx:alpine
Jul 05 23:12:23 node-01 systemd[1]: Started Daemon for nginx.
Jul 05 23:12:23 node-01 systemd[1]: Configuration file /etc/systemd/system/docker-nginx.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Jul 05 23:12:23 node-01 systemd[1]: Configuration file /etc/systemd/system/docker-nginx.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Jul 05 23:12:23 node-01 systemd[1]: Configuration file /etc/systemd/system/docker-nginx.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Jul 05 23:12:23 node-01 systemd[1]: Configuration file /etc/systemd/system/docker-nginx.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
root@node-01:~#```

[dploeger]

@davejrt I really believe, it's somewhat in our parameters. Could you perhaps try out integration test suite to replicate? You only need a username and password for the Docker store and add the following to .kitchen.yml once you cloned our repository:

platforms:
    - name: ubuntu-16.04
      driver:
        box: bento/ubuntu-16.04
    - name: debian-8
      driver:
        box: bento/debian-8

And then run

export STORE_USERNAME="yourdockerusername"
read -s STORE_PASSWORD
#enter your store password
export STORE_PASSWORD 
kitchen test default:debian-8

[davejrt]

Can you show me your systemd file? If this is an issue with your params as an edge case then perhaps it's worth forking the module or wrapping it with your own script.

[dploeger]

Sure, here it is:

[Unit]
Description=Daemon for hana-testhana1
After=docker.service
Wants=
Requires=docker.service

[Service]
Restart=on-failure
StartLimitInterval=20
StartLimitBurst=5
TimeoutStartSec=0
RestartSec=5
Environment="HOME=/root"
SyslogIdentifier=docker-hana-testhana1
ExecStartPre=-/usr/bin/docker kill hana-testhana1
ExecStartPre=-/usr/bin/docker rm  hana-testhana1

ExecStart=/usr/bin/docker run \
        -h 'hana-testhana1' --net bridge -m 0b -p 39017:39017 \
 -p 39041:39041 \
 -v /data/testhana1:/hana/mounts \
 --ulimit nofile=1048576:1048576 --sysctl kernel.shmmax=1073741824 --sysctl net.ipv4.ip_local_port_range="40000 60999" --sysctl kernel.shmmni=524288 --sysctl kernel.shmall=8388608 \
        --name hana-testhana1 \
        store/saplabs/hanaexpress:2.00.030.00.20180403.2 \
         --passwords-url file:///hana/mounts/password.json --agree-to-sap-license
ExecStop=-/usr/bin/docker stop --time=0 hana-testhana1
ExecStop=-/usr/bin/docker rm  hana-testhana1

[Install]
WantedBy=multi-user.target

I really believe, it's some sort of weird escaping problem with older versions of systemd, because running the ExecStart line in a terminal works flawlessly.

[davejrt]

We'd look at accepting PR to add in a script instead of using a command to fix edge cases like this, or alternatively you could look at using something like systemd-escape to see if that helps. The culprit is the line there you're specify net.ipv4.ip_local_port_range