Ansible operator: cannot create directory ‘/.ansible’: Permission denied

[daquinoaldo]

Type of question

Are you asking about community best practices, how to implement a specific feature, or about general context and help around the operator-sdk?
Help around the operator-sdk

Question

What did you do?

• I created an ansible operator with operator-sdk new visitors-frontend-operator --type=ansible --api-version=example.com/v1 --kind=VisitorsApp.
• I've edited the roles folder and successfully tested locally with operator-sdk run --local.
• I've correctly build, pushed and deployed to my openshift stack (I'm using crc).

What did you expect to see?
The CR resources allocated when I deploy my CR file, as when running the operator locally.

What did you see instead? Under which circumstances?
When I deploy the CR, nothing happens. These are the (verbose) logs of the ansible container of the operator pod.

Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/artifacts/605394647632969758//stdout
ansible-playbook 2.9.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/inventory/hosts as it did not pass its verify_file() method
script declined parsing /tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/inventory/hosts as it did not pass its verify_file() method
auto declined parsing /tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/inventory/hosts as it did not pass its verify_file() method
Set default localhost to localhost
Parsed /tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/inventory/hosts inventory source with ini plugin

Loading callback plugin awx_display of type stdout, v2.0 from /usr/local/lib/python3.6/site-packages/ansible_runner/callbacks/awx_display.py


LAYBOOK: c8d4c1032cd84d38b06aef83dbe58ddd *************************************
Positional arguments: /tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/project/c8d4c1032cd84d38b06aef83dbe58ddd
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/inventory',)
extra_vars: ('@/tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/env/extravars',)
forks: 5

PLAYBOOK: c8d4c1032cd84d38b06aef83dbe58ddd *************************************
Positional arguments: /tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/project/c8d4c1032cd84d38b06aef83dbe58ddd
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/inventory',)
extra_vars: ('@/tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/env/extravars',)
forks: 5

1 plays in /tmp/ansible-operator/runner/example.com/v1/VisitorsApp/myprj/visitorsapp/project/c8d4c1032cd84d38b06aef83dbe58ddd

[WARNING]: Found variable using reserved name: name


LAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: 1000540000

<localhost> EXEC /bin/sh -c 'echo ~1000540000 && sleep 0'

<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /.ansible/tmp/ansible-tmp-1584107823.0948596-231277174841977 `" && echo ansible-tmp-1584107823.0948596-231277174841977="` echo /.ansible/tmp/ansible-tmp-1584107823.0948596-231277174841977 `" ) && sleep 0'

fatal: [localhost]: UNREACHABLE! => {
    "changed": false,
    "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /.ansible/tmp/ansible-tmp-1584107823.0948596-231277174841977 `\" && echo ansible-tmp-1584107823.0948596-231277174841977=\"` echo /.ansible/tmp/ansible-tmp-1584107823.0948596-231277174841977 `\" ), exited with result 1, stderr output: mkdir: cannot create directory ‘/.ansible’: Permission denied\n",
    "unreachable": true
}

I can confirm, by opening a remote shell inside the container, that the current user (that is not root) cannot create the /.ansible directory.

watches.yaml file

---
- version: v1
  group: example.com
  kind: VisitorsApp
  role: /opt/ansible/roles/visitorsapp

build/Dockerfile

FROM quay.io/operator-framework/ansible-operator:v0.15.2

COPY watches.yaml ${HOME}/watches.yaml

COPY roles/ ${HOME}/roles/

Environment

• operator-sdk
  version: "v0.15.2"
  commit: "ffaf278993c8fcb00c6f527c9f20091eb8dd3352"
  go version: "go1.13.3 linux/amd64"

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.2", GitCommit:"94e669a", GitTreeState:"clean", BuildDate:"2020-02-03T23:11:39Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:
  CloudReady Containers

crc version:

crc version: 1.7.0+fa7e558
OpenShift version: 4.3.1 (embedded in binary)

oc version:

Client Version: v4.4.0
Server Version: 4.3.1
Kubernetes Version: v1.16.2

Additional context
Linux Fedora 31 amd64

[camilamacedo86]

Hi @daquinoaldo,

Did you build the project with operator-sdk buil <image:tag>? If not can you try and see if the error will change? If the resulting change please let us know and add here the logs if you start to face another issue.

[daquinoaldo]

Hi @camilamacedo86,
thank you for responding so quickly.

I build the image with operator-sdk build visitors-frontend-operator. Is the tag important?
Also, docker is installed via dnf, latest version of docker-ce.

I have also a visitors-backend-operator, implemented in Go, that is build and deployed in the same way, and it works.

[camilamacedo86]

Hi @daquinoaldo,

Thank you for let us know. I am unable to reproduce it with the memcached ansible sample and minikube. Could you share the repo link of your project? Can we try to run it?

[daquinoaldo]

@camilamacedo86
Sure, I've uploaded the code in my repository visitors-splitted.

You can find the working Go implementation in visitors-backend-operator and the Ansible implementation that caused me the problem in visitors-frontend-operator.
The namespace on OpenShift is myprj.

All the commands I used to run and deploy it are in the README.

[jontow]

I'm hitting this same issue on an unrelated in-development operator. I've just generated it and begun development using operator-sdk v0.16.0. I'm also using crc 1.7.0.

I'm really not familiar with the insides of operator-sdk or the ansible-operator image, but wanted to share what I see.

It looks like something is expecting $HOME to be /opt/ansible, but (and you can see this in the original logs, too) this is getting run as uid 1000540000, which has a $HOME of '/'; not as ansible-operator (uid 1001), which has a $HOME of '/opt/ansible'. The /opt/ansible/.ansible directory gets chmod'd intentionally during the image build:

FROM quay.io/operator-framework/ansible-operator:v0.16.0

COPY requirements.yml ${HOME}/requirements.yml
RUN ansible-galaxy collection install -r ${HOME}/requirements.yml \
 && chmod -R ug+rwx ${HOME}/.ansible
...

Not sure if this helps or is simply a long +1, but thank you for looking at this!

[jontow]

For whatever it is worth, and to temporarily unblock anyone else that runs across this issue while a proper fix is unavailable, I am able to work around the above reported error by simply adding "ln -s ${HOME}/.ansible /.ansible" to the end of the RUN line in my build/Dockerfile:

RUN ansible-galaxy collection install -r ${HOME}/requirements.yml \
 && chmod -R ug+rwx ${HOME}/.ansible \
 && ln -s ${HOME}/.ansible /.ansible

[jmontleon]

For what it's worth I saw this on a 4.3.1 cluster I was diagnosing for someone, but not on a 4.3.8 cluster I tried to reproduce it on. I also didn't see it on two other testing clusters running 4.3.8. I don't know why cluster version would come into play, but thought I would mention it.

[jontow]

Confirmed that this behavior disappears on crc 1.8.0 (OpenShift 4.3.8).

[jmontleon]

I wonder if this could be related to cri-o/cri-o#3186

[estroz]

/cc @fabianvf