Tasks fails with a permission denied on /tmp at start

[menardorama]

• Bug Report
• UI

SUMMARY

After having successfully deployed AWX on an Openshift cluster following this blog post (https://developers.redhat.com/blog/2017/10/16/guide-starting-use-awx-top-openshift-upstream-red-hat-ansible-tower/); we are unable to start a deployement. The fetch of the playbook is failing on a "permission denied on /tmp"

ENVIRONMENT

• AWX version: 1.0.1.93
• AWX install method: openshift
• Ansible version: 2.4.0.0
• Operating System: Linux
• Web Browser: Chrome

STEPS TO REPRODUCE

Define a new deployement.

• Project creation is working fine and git repo is correctly fetched (using git credentials)
• Inventory is defined (some variables in the json field of one group)
• Create the template and associate everything.

We have a user dedicated to ansible on targeted hosts and are sudoers NOPASSWD.

The fetch of the of the repo seems to work as the Project name appear with a green point on the task detail.

EXPECTED RESULTS

The task should start and working correctly

ACTUAL RESULTS

The task hang in error with the following trace :

Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 786, in run
safe_args = self.build_safe_args(instance, **kwargs)
File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 1161, in build_safe_args
return self.build_args(job, display=True, **kwargs)
File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 1088, in build_args
args = ['ansible-playbook', '-i', self.build_inventory(job, **kwargs)]
File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 681, in build_inventory
with open(path, 'w') as f:
IOError: [Errno 13] Permission denied: u'/tmp/awx_28_Ee1gSY/inventory'

ADDITIONAL INFORMATION

The same scenario is being run correctly on an Ansible Tower (evaluation mode) version 3.1.3

[mulbc]

I'm hitting the same issue - reproducing this should be easy - It happens right after the installation even with the demo credentials/project/inventory:

    Image ID:           docker-pullable://ansible/awx_web@sha256:ab915ca59d8f1459b79489fac1cfd3e854a45a64b6bdf9fd2c108d8889e93bfc
    Image ID:           docker-pullable://ansible/awx_task@sha256:a746070100fc0cd8f827e506c5dc28289518ec206d919cabd39548e35d7cae95

Some more context from the awx-celery log

[2017-10-24 16:31:00,281: DEBUG/MainProcess] Task accepted: awx.main.tasks.update_inventory_computed_fields[5afd12f8-f83c-4a39-8ce9-e54a532675e8] pid:408
2017-10-24 16:31:00,309 DEBUG    awx.main.models.inventory Going to update inventory computed fields
[2017-10-24 16:31:00,309: DEBUG/Worker-5] Going to update inventory computed fields
2017-10-24 16:31:00,332 WARNING  awx.main.tasks %s encountered an error (rc=%s), please see task stdout for details.
2017-10-24 16:31:00,332 WARNING  awx.main.tasks %s encountered an error (rc=%s), please see task stdout for details.
[2017-10-24 16:31:00,332: WARNING/Worker-4] %s encountered an error (rc=%s), please see task stdout for details.
[2017-10-24 16:31:00,362: ERROR/MainProcess] Task awx.main.tasks.run_job[4caf0590-1977-43f1-9e08-2d40292c382a] raised unexpected: TaskError('%s encountered an error (rc=%s), please see task stdout for details.',)
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 469, in _wrapped
    return f(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 889, in run
    raise TaskError(instance, rc)
TaskError: %s encountered an error (rc=%s), please see task stdout for details.
[2017-10-24 16:31:00,366: INFO/MainProcess] Received task: awx.main.tasks.handle_work_error[ade6215a-92f6-4274-b02f-5968583dd21f]
[2017-10-24 16:31:00,370: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x6da0b18> (args:(u'awx.main.tasks.handle_work_error', u'ade6215a-92f6-4274-b02f-5968583dd21f', [u'4caf0590-1977-43f1-9e08-2d40292c382a'], {u'subtasks': [{u'type': u'job', u'id': 2}]}, {u'utc': True, u'is_eager': False, u'chord': None, u'group': u'388c71bb-1c06-4f16-8f96-4f2a231440c0', u'args': [u'4caf0590-1977-43f1-9e08-2d40292c382a'], u'retries': 0, u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u'tower'}, u'expires': None, u'hostname': 'celery@localhost', u'task': u'awx.main.tasks.handle_work_error', u'callbacks': None, u'correlation_id': u'ade6215a-92f6-4274-b02f-5968583dd21f', u'errbacks': None, u'timelimit': [None, None], u'taskset': u'388c71bb-1c06-4f16-8f96-4f2a231440c0', u'kwargs': {u'subtasks': [{u'type': u'job', u'id': 2}]}, u'eta': None, u'reply_to': u'9cd61b69-c9e9-3047-b04e-4aaccb7e2828', u'id': u'ade6215a-92f6-4274-b02f-5968583dd21f', u'headers': {}}) kwargs:{})
[2017-10-24 16:31:00,373: DEBUG/MainProcess] Task accepted: awx.main.tasks.handle_work_error[ade6215a-92f6-4274-b02f-5968583dd21f] pid:150
2017-10-24 16:31:00,404 DEBUG    awx.main.tasks Executing error task id ade6215a-92f6-4274-b02f-5968583dd21f, subtasks: [{u'type': u'job', u'id': 2}]
2017-10-24 16:31:00,404 DEBUG    awx.main.tasks Executing error task id ade6215a-92f6-4274-b02f-5968583dd21f, subtasks: [{u'type': u'job', u'id': 2}]
[2017-10-24 16:31:00,404: DEBUG/Worker-4] Executing error task id ade6215a-92f6-4274-b02f-5968583dd21f, subtasks: [{u'type': u'job', u'id': 2}]
2017-10-24 16:31:00,429 DEBUG    awx.main.models.inventory Finished updating inventory computed fields
[2017-10-24 16:31:00,429: DEBUG/Worker-5] Finished updating inventory computed fields```

[manoj3832]

hello friends ,
i think you are missing "user" parameter when starting containers . try with adding "user: root" in compose or what ever you are using . goal is to enable root user in aws_task and aws_web container . i was also facing same issue in swarm cluster with docker-compose . it works when i enable user field in compose. you can also check installer/local_docker/task/main.yml where they enable user:root in web and task container configuration.

[cevich]

i think you are missing "user" parameter

IIUC, the openshift containers run as user 'awx' and drop privileges by design.

[cevich]

Using the latest AWX from github, running the bog-standard inventory & install.yml (docker-based install) this problem is NOT reproducible (demo playbook works fine). This issue must be related specifically to usage under openshift.

[howels]

I've hit this issue also. This core problem is that whilst Docker allows containers to run as root, for reasons of security OpenShift forces best-practice by default and ensures that containers are run with uid != 0. This minimises the risk of container to host breakout attacks and makes multi-tenant environments more secure. So I would suggest that the bug is valid in both cases as AWX should follow best-practice and run as a non-root user unless there is a hard requirement otherwise.

[bbrfkr]

+1

[skk2010]

+1
installed from docker hub

dockerhub_base=ansible
dockerhub_version=latest

[lotjuh]

+1 (running the dockers in AWS ECS)

[midekra]

+1
I'm hitting this issue aswell. Running on-premise Openshift Origin.

As a workaround I have created a new serviceaccount inside the project and added a scc for this SA:

oc create serviceaccount awx
oadm policy add-scc-to-user anyuid system:serviceaccount:<namespace>:awx

I've then added an annotation to my deployement:

...
...
# Deployment Config
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    labels:
      name: awx-web-deploy
      service: django
    annotations:
      openshift.io/scc: privileged
    name: awx
...
...

And also added the serviceaccount and securityContext to the container template:

...
...
     spec:
        serviceAccount: awx
        serviceAccountName: awx
...
...
            securityContext:
              privileged: false
              runAsUser: 0
...
...

[mprasil]

Can confirm the issue. I'm running the ansible/awx_task image as regular docker container. (We have a docker-based installation of AWX here) Running the image with root user will avoid the problem, however the root user first needs to be added to /etc/passwd, so I've created a custom awx_task image like this:

# Dockerfile
FROM ansible/awx_task:latest

# Switch to root
ADD passwd-root /etc/passwd
USER root

With the following passwd-root:

root:x:0:0:root:/root:/bin/false
awx:x:1000:0:,,,:/var/lib/awx:/bin/bash

Obviously not an ideal solution and the core issue needs to be fixed instead.

[andutt]

Confirming the above problem on Openshift and AWX 1.0.1.167. Any ETA for a fix for this issue?

[matburt]

Hey folks, sorry this fell through the cracks a little bit. Taking a look at this now.

[matburt]

This is now fixed by #657