atomic breaks port mappings for non SPCs

[aveshagarwal]

I am having issues with "atomic run" and also with LABEL RUN where the exposed ports in container are not being published to the host. I have following in Dockerfile:

LABEL RUN="docker run -d -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 IMAGE"

Running the container:

#atomic run etcd
etcd
# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4804270b3b28 etcd "/usr/bin/etcd" About a minute ago Up 4 seconds 2379-2380/tcp, 4001/tcp, 7001/tcp etcd

In the above, the ports are not being published to the host.

However when I try docker run, the same container works as expected:

docker run -d -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 etcd

754c29fcc2a0e2edb16981afbf4ba5b92a641099ab014a58bb7c213141cc07b6
Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
754c29fcc2a0 etcd "/usr/bin/etcd" 20 seconds ago Up 18 seconds 0.0.0.0:2379-2380->2379-2380/tcp, 0.0.0.0:4001->4001/tcp, 0.0.0.0:7001->7001/tcp modest_feynman

Here the ports are published to the host correctly.

I also tried another version of LABEL RUN with optional argument as follows:

LABEL RUN /usr/bin/docker run -d $OPT1 --name $NAME $IMAGE $OPT2 $OPT3

atomic run --opt1="-p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380" etcd

etcd

docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b4f393cfbae4 etcd "/usr/bin/etcd" 50 seconds ago Up 7 seconds 2379-2380/tcp, 4001/tcp, 7001/tcp etcd

But here again it does not work as expected as ports not are being published to the host correctly.

[aveshagarwal]

As per @cgwalters
https://lists.projectatomic.io/projectatomic-archives/atomic/2015-August/msg00002.html

[rhatdan]

@aveshagarwal That makes no sense. You are not running the container in --spc mode which is the only thing hard coded to run --net=host.

When you run the atomic run etcd command what does the atomic tool say it is executing?

Please attach your dockerfile and I will try it here.

[aveshagarwal]

@rhatdan what does not make sense? Yes I am not running with --spc mode.
The dockerfile is here: https://github.com/aveshagarwal/etcd-container/blob/master/Dockerfile
and the repo is here:
https://github.com/aveshagarwal/etcd-container

[aveshagarwal]

@rhatdan any feedback about what might be wrong with the dockerfile or something else?

[bexelbie]

@aveshagarwal when you execute atomic run etcd it should print out the docker command that is being executed. Can you include that here? That will show if any extra things are appearing on your command line.

[rhatdan]

@aveshagarwal I am back from PTO and business trips. Your Dockerfile looks fine. What command are you executing to install your image?

[aveshagarwal]

@bexelbie @rhatdan

on rhel7.2beta:

[root@kube-master etcd-container]#atomic install etcd
/usr/bin/docker run --rm --privileged -v /:/host -e HOST=/host -e NAME=etcd -e IMAGE=etcd etcd /usr/bin/install.sh
Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.
fb886585ae2c88edb93141d0b0720c2db3910ad0f6370c5f6b627a7b695596ae
Full path required for exclude: net:[4026532349].
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd_container_etcd.service to /etc/systemd/system/etcd_container_etcd.service.
[root@kube-master etcd-container]# atomic run etcd
etcd
[root@kube-master etcd-container]#

On atomic 7.1.4:
-bash-4.2# atomic install etcd
/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3
faf2bcf9b889f2d75e6c216026636d50fe7de8982f33ba0fb33a82df77f54129
Full path required for exclude: net:[4026532218].
ln -s '/etc/systemd/system/etcd_container_etcd.service' '/etc/systemd/system/multi-user.target.wants/etcd_container_etcd.service'
-bash-4.2# atomic run etcd
etcd
-bash-4.2# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
faf2bcf9b889 etcd "/usr/bin/etcd" 50 seconds ago Up 3 seconds 2379-2380/tcp, 4001/tcp, 7001/tcp etcd

[rhatdan]

@aveshagarwal They look like they are using two different images at least from the point of view of the labels being executed.

But is there a bug above that I am missing?

[aveshagarwal]

@rhatdan they are using exactly same image

on rhel7.2 beta:
[root@kube-master etcd-container]# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
etcd latest 11328ba51941 About an hour ago 515.1 MB

atomic version on 7.2beta:
[root@kube-master etcd-container]# rpm -qa|grep ^atomic
atomic-1.0-108.el7.x86_64

On atomic 7.1.4:
-bash-4.2# docker images |grep etcd
etcd latest 11328ba51941 About an hour ago 515.1 MB

atomic version on atomic 7.1.4
-bash-4.2# rpm -qa|grep ^atomic
atomic-0-0.42.gitf133684.el7.x86_64

Is it possible that the difference in labels being executed might be due to difference versions of atomic on 7.2beta and atomic 7.1.4?

[rhatdan]

atomic info IMAGE

On both images. Maybe you have a different version of the atomic tool?

[aveshagarwal]

Right, I provided their version numbers in the previous comment.

On 7.2.beta:
[root@kube-master ~]# atomic info etcd
RUN: /usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 /usr/bin/etcd $OPT3
Name: rhel7/etcd
Version: 1.0
Architecture: x86_64
INSTALL: /usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3
Release: 1
Vendor: Red Hat
BZComponent: etcd-docker
UNINSTALL: /usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3

on atomic 7.1.4:
-bash-4.2# atomic info etcd
RUN : /usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 /usr/bin/etcd $OPT3
Name : rhel7/etcd
Version : 1.0
Architecture : x86_64
INSTALL : /usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3
Release : 1
Vendor : Red Hat
BZComponent : etcd-docker
UNINSTALL : /usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3

[rhatdan]

You never showed the output of

-bash-4.2# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
faf2bcf9b889 etcd "/usr/bin/etcd" 50 seconds ago Up 3 seconds 2379-2380/tcp, 4001/tcp, 7001/tcp etcd

On beta release version.

[aveshagarwal]

7.2beta:

[root@kube-master ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fb886585ae2c etcd "/usr/bin/etcd" 2 hours ago Up 2 hours 2379-2380/tcp, 4001/tcp, 7001/tcp etcd

[rhatdan]

And the expected results should look like?
754c29fcc2a0 etcd "/usr/bin/etcd" 20 seconds ago Up 18 seconds 0.0.0.0:2379-2380->2379-2380/tcp, 0.0.0.0:4001->4001/tcp, 0.0.0.0:7001->7001/tcp modest_feynman

[aveshagarwal]

Right

[aveshagarwal]

if you run the same image directly with docker run, port mapping works correctly on both rhel and atomic.

[rhatdan]

It looks like the $OPT are causing problems. I am attempting to build your image locally

[aveshagarwal]

@rhatdan unless something changed recently I had tried without OPT args and the result was same. This problem I have seen even before OPT came into existence as far as I remember.

[aveshagarwal]

https://lists.projectatomic.io/projectatomic-archives/atomic/2015-August/msg00000.html shows I had the same issue without OPT too.

[rhatdan]

Do you have the root directory?

[aveshagarwal]

Yes on both 7.2beta and atomic 7.1.4, why would you ask that?

[rhatdan]

I need one to build my version.
Step 8 : ADD root /
root: no such file or directory

[rhatdan]

I got to run out now, I will look at this in the morning.

[aveshagarwal]

No problem: git clone https://github.com/aveshagarwal/etcd-container

[rhatdan]

The problem is that you are creating the container within the install without actually setting up the port mappings.

In your install.sh you should be doing the following:

chroot ${HOST} /usr/bin/docker create -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name --name ${NAME} ${IMAGE}

atomic run -n etcd etcd

Will start the etcd container if it exists. If it does not exists it uses the run label. Since you were creating the container during install, if you do atomic install before atomic run, the run will only start the container if it is stopped.

[aveshagarwal]

@rhatdan thanks, I tested it and it worked.

I think i thought that normal flow was always atomic install followed by atomic run. It seems that once a container is created with atomic install, atomic run just starts the container and ignores the arguments passed in LABEL RUN. I think it would be better if it is documented more clearly.

Also my understanding was that atomic run should not be used without running atomic install first.
That means that atomic run wont start if it can not find the container if atomic install is not run previously. But that does not seem to be true too.

Also If it is part of expected workflow that atomic run can be run without running atomic install first, it should be documented that LABEL RUN params and docker create params in the install script part of atomic install should be same to have a consistent behavior for 2 workflows (atomic install followed by atomic run and and just running atomic run directly) in my understanding.

I think I am ok if this issue can be closed now.

[rhatdan]

I think i thought that normal flow was always atomic install followed by atomic run. It seems that once a container is created with atomic install, atomic run just starts the container and ignores the arguments passed in LABEL RUN. I think it would be better if it is documented more clearly.

LABEL RUN is independent of LABEL INSTALL. The confusing thing about atomic run was that it was designed for the use case of the rhel tools container, where you would be running multiple commands in the same container.

atomic run rheltools /bin/sh
atomic run rheltools man atomic
atomic run rheltools sosreport

So after the initial run, the container gets created and all future runs become docker exec into
the existing container.

When people try to do single run containers, it gets confusing. I am not sure how we fix this problem
other then better documentation.

atomic install is just in charge of setting up the container to run, it could use the atomic run command within its install, IE have the systemd unit files execute atomic run, or it could just do docker directly or k8s or run containers via the docker daemon to auto restart. Really want ultimate flexibility here.

Also my understanding was that atomic run should not be used without running atomic install first.
That means that atomic run wont start if it can not find the container if atomic install is not run previously. But that does not seem to be true too.

As stated above atomic run can be executed independently of atomic install
Also If it is part of expected workflow that atomic run can be run without running atomic install first, it should be documented that LABEL RUN params and docker create params in the install script part of atomic install should be same to have a consistent behavior for 2 workflows (atomic install followed by atomic run and and just running atomic run directly) in my understanding.

I think I am ok if this issue can be closed now.