Launch scripts don't work as non-root

[ToxicGLaDOS]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

In each of the launch scripts (launch_awx_rsyslog.sh, launch_awx_task.sh, launch_awx_web.sh), there's this block:

if [ `id -u` -ge 500 ]; then
    echo "awx:x:`id -u`:`id -g`:,,,:/var/lib/awx:/bin/bash" >> /tmp/passwd
    cat /tmp/passwd > /etc/passwd
    rm /tmp/passwd
fi

But the logic doesn't make sense to me. If our UID is greater than 500 then we don't have root privileges so we can't edit /etc/passwd. For us this presents as /usr/bin/launch_awx_task.sh: line 4: /etc/passwd: Permission denied in the container logs when running as UID = 1000. @rooftopcellist Could you explain what this block is there to accomplish? It looks like it was added by you in #9289.

AWX version

22.0.0

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

Deploy AWX with a security context that defines runAsUser and runAsGroup. Here's a script that deploys awx fresh and shows the issue.

#!/usr/bin/env bash

cat << EOF > kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/ansible/awx-operator/config/default?ref=2.0.1

images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.0.1

namespace: awx
EOF

kubectl apply -k .

cat << EOF > awx.yaml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
  namespace: awx
spec:
  image_pull_policy: Always
  service_type: nodeport
  security_context_settings:
    runAsGroup: 1000
    runAsUser: 1000
EOF

kubectl apply -f awx.yaml

After the awx-task pod comes up you can see the error in the awx-task, awx-rsyslog and awx-ee containers

> k logs -n awx awx-task-5c6d49b7fb-lglgx -c awx-task
/usr/bin/launch_awx_rsyslog.sh: line 4: /etc/passwd: Permission denied
[wait-for-migrations] Waiting for database migrations...
[wait-for-migrations] Attempt 1 of 30
[wait-for-migrations] Waiting 0.5 seconds before next attempt
[wait-for-migrations] Attempt 2 of 30
...

Expected results

AWX runs properly as a non-root user

Actual results

3 (awx-ee, awx-task, awx-rsyslog) of the 4 containers in the pod fail to start with similar errors /usr/bin/launch_awx_task.sh: line 4: /etc/passwd: Permission denied. The redis container starts up normally.

Additional information

No response

[shanemcd]

It's working for most use cases, so there must be some edge case here we haven't accounted for.

We grant write access to /etc/passwd here in the Dockerfile:

awx/tools/ansible/roles/dockerfile/templates/Dockerfile.j2

Lines 257 to 259 in b28e14c

	/etc/passwd \
	/var/lib/awx/rsyslog/rsyslog.conf ; \
	do touch $file ; chmod g+rw $file ; chgrp root $file ; done

You can find a more complete version of this hack, with a detailed explanation, in the default entrypoint used by images produced with ansible-builder:

https://github.com/ansible/ansible-builder/blob/77f54f067a8f23361ae4e061876ad6705965604b/src/ansible_builder/_target_scripts/entrypoint#L6-L26

What distribution of Kubernetes are you using? And what's the underlying container runtime?

[ToxicGLaDOS]

We grant write access to /etc/passwd here in the Dockerfile:

If I'm reading that Dockerfile correctly we're setting read and write permissions on /etc/passwd for the root group. But that doesn't help us if we're just a normal user. In our case UID = GID = 1000, means we still don't have permission.

What distribution of Kubernetes are you using? And what's the underlying container runtime?

We're using AKS which uses containerd as it's runtime.

[shanemcd]

Got it, that makes sense. Typically non-root users are still in the root group.

$ docker run -ti --user=1000 quay.io/centos/centos:stream9 groups
root

This is also true in Kubernetes by default.

From reading the code, the entrypoint from ansible-builder looks a bit more robust and not as failure prone. We may be able to use parts of that here, but I'm not sure. Are you able to build custom AWX images and try it out?

[ToxicGLaDOS]

I might have to set a few things up, but yeah I can do that.

[vshete93]

@ToxicGLaDOS were you able to solve your issue using above solution? I am in the same boat and want to check if above worked for you? TIA.

[ToxicGLaDOS]

After some digging I think I've found that my initial analysis doesn't tell the whole story. While the error /usr/bin/launch_awx_task.sh: line 4: /etc/passwd: Permission denied does show up it isn't fatal (that portion appears before the set -e). The fatal part is later in launch_awx_rsyslog.sh and launch_awx_task.sh. launch_awx_web.sh doesn't error fatally. In particular, line 23 of launch_awx_rsyslog.sh

cat << EOF > /var/lib/awx/rsyslog/rsyslog.conf
action(type="omfile" file="/dev/null")
EOF

fails because /var/lib/awx/rsyslog is owned by root with 700 permissions.

And launch_awx_task.sh fails to open a socket on line 22 exec supervisord -c /etc/supervisord_task.conf

/usr/bin/launch_awx_task.sh: line 4: /etc/passwd: Permission denied
[wait-for-migrations] Waiting for database migrations...
[wait-for-migrations] Attempt 1 of 30
Instance Group already registered controlplane
Instance Group already registered default
Successfully registered instance None
(changed: True)
Error: Cannot open an HTTP server: socket.error reported errno.EACCES (13)
For help, use /usr/local/bin/supervisord -h

[vshete93]

@ToxicGLaDOS so where did you exactly change the permissions for/var/lib/awx/rsyslog? Does that has to be in both Dockerfiles with images quay-io/ansible/awx:22.2.0 ? and quay-io/ansible/awx-ee:22.2.0 and run as a user test_user?

Something like below:

FROM quay-io/ansible/awx:22.2.0

USER root
RUN chmod -c 0644 /etc/passwd* /etc/group*
RUN useradd -m -s /usr/bin test_user
RUN chmod 755 /var/lib/awx/rsyslog
USER test_user
ADD --chmod=755 --chown=test_user:test_user launch_awx_task.sh launch_awx_rsyslog.sh /usr/bin/

[ToxicGLaDOS]

@vshete93 I didn't actually find a fix in our own Dockerfile like that yet. I've been trying to put together a fix in awx itself. I think the issue is that we interact with a lot of directories that are owned by root and aren't world readable/writeable (/var/lib/awx, /var/lib/awx/rsyslog, /var/run/supervisor). Moving those interactions into /tmp seems to work, albeit a strange place for some of the things that need to be moved (pid files and sockets).