awx_web and awx_task not running

[krentiken]

ISSUE TYPE

• Bug Report

SUMMARY

Error response from daemon: Cannot start container awx_web and awx_task

ENVIRONMENT

• AWX version: 4.0.0
• AWX install method: docker on linux
• Ansible version: 2.7.5
• Operating System: RHEL 7.6
• Web Browser: Google Chrome 73

STEPS TO REPRODUCE

1. container start fails:

# docker start awx_web
Error response from daemon: Cannot restart container awx_web: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/awxcompose/environment.sh\\\" to rootfs \\\"/var/lib/docker/overlay2/f8651bc3f821fe0179904551b4f37a4b92c6a9dfa4e1ee15e84ead0a0201c79d/merged\\\" at \\\"/var/lib/docker/overlay2/f8651bc3f821fe0179904551b4f37a4b92c6a9dfa4e1ee15e84ead0a0201c79d/merged/etc/tower/conf.d/environment.sh\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

# docker start awx_task
Error response from daemon: Cannot restart container awx_task: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/awxcompose/environment.sh\\\" to rootfs \\\"/var/lib/docker/overlay2/cc91a0cf1cf9bcb91e10d6a3d7a0409173000a65b7b62dd834a52d22a34fd65f/merged\\\" at \\\"/var/lib/docker/overlay2/cc91a0cf1cf9bcb91e10d6a3d7a0409173000a65b7b62dd834a52d22a34fd65f/merged/etc/tower/conf.d/environment.sh\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

EXPECTED RESULTS

starting docker containers awx_web and awx_task

ADDITIONAL INFORMATION

I managed to find out that sistemd tmpfiles deletes files older than 10 days.
In the directory /tmp has been removed "awxcompose".
At the time of launch of the containers awx_web and awx_task in /tmp directory was created ./awx compose/environment.sh, although the file must be created "environment.sh"

/tmp/awxcompose/
total 8
-rw-------. 1 root root 553 Apr 2 17:10 credentials.py
drwxr-xr-x. 2 root root 6 Apr 15 17:30 environment.sh
-rw-------. 1 root root 9 Apr 2 17:10 SECRET_KEY

Why create files in the /tmp directory? Why is the "environment.sh" directory created when starting docker containers?

[herdsonb01]

Same issue with similar error messages (further below). When I go to /tmp/awxcompose 'environment.sh', 'SECRET_KEY' and are folders (below)?!?
On Ubuntu 18.04.29.
Only installed it about 2 hours ago. Worked fine directly after install....
It seems this part of the error is correct: Are you trying to mount a directory onto a file (or vice-versa)?

EDIT: On reboot it's turning the files in /tmp/awxcompose into folders. Tested turning these folders into files and the containers would start (not working of course but they start without obvious error). If you have nothing in /tmp/awxcompose it will create the files as folders on container start.
Something to do with /roles/local_docker/templates?

admin@hsc-goyry01:/$ cd /tmp/awxcompose/
admin@hsc-goyry01:/tmp/awxcompose$ ls
credentials.py environment.sh SECRET_KEY
admin@hsc-goyry01:/tmp/awxcompose$ ls -la
total 20
drwxr-xr-x 5 root root 4096 Apr 18 00:43 .
drwxrwxrwt 11 root root 4096 Apr 18 01:34 ..
drwxr-xr-x 2 root root 4096 Apr 18 00:43 credentials.py
drwxr-xr-x 2 root root 4096 Apr 18 00:43 environment.sh
drwxr-xr-x 2 root root 4096 Apr 18 00:43 SECRET_KEY
admin@hsc-goyry01:/tmp/awxcompose$ cd environment.sh/
admin@hsc-goyry01:/tmp/awxcompose/environment.sh$ ls
admin@hsc-goyry01:/tmp/awxcompose/environment.sh$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Apr 18 00:43 .
drwxr-xr-x 5 root root 4096 Apr 18 00:43 ..
admin@hsc-goyry01:/tmp/awxcompose/environment.sh$ cd ..
admin@hsc-goyry01:/tmp/awxcompose$ cd SECRET_KEY/
admin@hsc-goyry01:/tmp/awxcompose/SECRET_KEY$ cd ..
admin@hsc-goyry01:/tmp/awxcompose$ cd credentials.py/
admin@hsc-goyry01:/tmp/awxcompose/credentials.py$ ls
admin@hsc-goyry01:/tmp/awxcompose/credentials.py$

docker start awx_web errors:

Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused "rootfs_linux.go:58: mounting \"/tmp/awxcompose/SECRET_KEY\" to rootfs \"/var/lib/docker/overlay2/62583d1a85309086b0e867012d16d0fbbf145fb55e3db179eba32ba3821df030/merged\" at \"/var/lib/docker/overlay2/62583d1a85309086b0e867012d16d0fbbf145fb55e3db179eba32ba3821df030/merged/etc/tower/SECRET_KEY\" caused \"not a directory\""": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
Error: failed to start containers: awx_web

docker start awx_task errors:

Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused "rootfs_linux.go:58: mounting \"/tmp/awxcompose/SECRET_KEY\" to rootfs \"/var/lib/docker/overlay2/0890bc6d7fcffcba440cabc4057782f66829dbde448414b4984536857c80a401/merged\" at \"/var/lib/docker/overlay2/0890bc6d7fcffcba440cabc4057782f66829dbde448414b4984536857c80a401/merged/etc/tower/SECRET_KEY\" caused \"not a directory\""": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
Error: failed to start containers: awx_task

[vin-c]

Same here, did you find a workaround to start the containers ?

[krentiken]

I am back on version 3.0.1. I'm waiting for the developers to answer

[lijok]

1. If docker can't find the mount source paths when starting up a container, it will create those paths as directories
2. Files in /tmp get removed automatically after some time depending on which os you're using. On ubuntu, /tmp is cleared on reboot

So what's happening is that you're not changing the postgres_data_dir and docker_compose_dir variables in your awx/installer/inventory file before installation, the critical files get dropped into /tmp, after a while (or on reboot in case of ubuntu) those files get deleted, you try to start up the containers, docker doesn't find /tmp/awxcompose/environment.sh so creates it as a directory, tries to mount it into a file, fails

Change the postgres_data_dir and docker_compose_dir variables in the inventory file and rerun the installation

[vin-c]

default value for docker_compose_dir was changed between 3.0.1 and 4.0.0 from /var/lib/ to /tmp/ indeed

[vin-c]

confirm that after setting those values, no more issue after a reboot and no more files in /tmp

[ryanpetrello]

@lijok's explanation makes sense to me, it sounds like adjusting your inventory to reflect 3.0.1 -> 4.0.0 should be sufficient?

Please feel free to reopen if you're still having issues with your 4.0.0 install.

[ghost]

It worked for me by changing the postgres_data_dir and docker_compose_dir but I'm curious why this is used as the default. It does not look like the logical place to put persistent data.

[philipneil]

Hey there!

I'm facing this issue too.

What should I be setting the progres_data_dir and docker_compose_dir variables too?

[philipneil]

OK think I got it.. changed docker_compose_dir to /var/lib/awxcompose

AWX starting after reboot now :)

[ghost]

@ryanpetrello It seems like yet more people are running into this same problem... In the spirit of "sensible defaults", could we have this issue reopened and fixed please?

[csdozier]

Just ran into this one, I think the default should not be /tmp

[demogorgonz]

Just ran into this one, I think the default should not be /tmp

Definitely not /tmp

[YashSrivastava20bcs4443]

Error response from daemon: No such container: awx_task