Ansible problem after 7.2.1.

[WaughB]

Operating System: Ubuntu 16.04
Docker Version: 18.06.1-ce

Attempted to run

docker run -d -p 8000:8000 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=12345678910' splunk/splunk:latest

and was met with this the container quickly exited. Checking docker logs provided me with:

PLAY [Run default Splunk provisioning] *****************************************
Saturday 29 December 2018  01:25:36 +0000 (0:00:00.034)       0:00:00.034 ***** 

TASK [Gathering Facts] *********************************************************
fatal: [localhost]: FAILED! => {"msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc: 1, err: chown: changing ownership of '/var/tmp/ansible-tmp-1546046736.73-44831970175188/': Operation not permitted\nchown: changing ownership of '/var/tmp/ansible-tmp-1546046736.73-44831970175188/AnsiballZ_setup.py': Operation not permitted\n}). For information on working around this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}
	to retry, use: --limit @/opt/container_artifact/ansible-retry/site.retry

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1   

Saturday 29 December 2018  01:25:36 +0000 (0:00:00.284)       0:00:00.318 ***** 
=============================================================================== 
Gathering Facts --------------------------------------------------------- 0.28s

This happened for tags: 7.2.2, 7.2.3, 7.2, and latest.

[arctan5x]

For the 7.2.3 release, there was a change in how user and permission are handled inside of the container for security reasons. I just pulled splunk/splunk:latest and was unable to reproduce that issue. Also confirmed that the installed Splunk version is 7.2.3. I am using Docker engine 18.09.0 so would you be able to upgrade the Docker engine and give it another shot?

[WaughB]

Updated Docker to version 18.09.0.
Same result. After running:

docker run -d -p 8000:8000 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=12345678910' splunk/splunk:latest

The container quickly exits and docker logs says this:

PLAY [Run default Splunk provisioning] *****************************************
Saturday 29 December 2018  21:18:18 +0000 (0:00:00.035)       0:00:00.035 ***** 

TASK [Gathering Facts] *********************************************************
fatal: [localhost]: FAILED! => {"msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc: 1, err: chown: changing ownership of '/var/tmp/ansible-tmp-1546118298.38-154813452708260/': Operation not permitted\nchown: changing ownership of '/var/tmp/ansible-tmp-1546118298.38-154813452708260/AnsiballZ_setup.py': Operation not permitted\n}). For information on working around this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}
	to retry, use: --limit @/opt/container_artifact/ansible-retry/site.retry

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1   

Saturday 29 December 2018  21:18:18 +0000 (0:00:00.270)       0:00:00.306 ***** 
=============================================================================== 
Gathering Facts --------------------------------------------------------- 0.27s

Are there any other requirements that changed to run 7.2.3?

[arctan5x]

Not really. 7.2.3 image is backward compatible with previous versions of Docker engine. Would you like to jump on a video call to drill down on this issue?

[JanGoerss]

Hi,

i have the same problem from 7.2.1 to (7.2.2, 7.2.3 or latest)

Docker Version : 18
Operating System: Oracle Linux Server 7.6

docker compose config:

...
app_splunk:
  restart: always
  image: splunk/splunk
  container_name: app_splunk
  networks:
    splunknet:
      aliases:
      - app_splunk
  ports:
  - '8000:8000'
  - '8088:8088'
  - '8191:8191'
  - '9997:9997'
  - '1514:1514'
  volumes:
  - /data/app/splunk/etc:/opt/splunk/etc
  - /data/app/splunk/var:/opt/splunk/var
  environment:
  - SPLUNK_START_ARGS=--accept-license --answer-yes
  - SPLUNK_PASSWORD=*******************
  - TZ=Europe/Berlin
  - SPLUNK_ENABLE_LISTEN=9997
  - SPLUNK_ADD=tcp 1514
...

and the log

 Thursday 03 January 2019  08:17:05 +0100 (0:00:00.035)       0:00:00.035 ******

 TASK [Gathering Facts] *********************************************************
 fatal: [localhost]: FAILED! => {"msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc: 1, err: chown: changing ownership of '/var/tmp/ansible-tmp-1546499825.06-70753836802764/': Operation not permitted\nchown: changing ownership of '/var/tmp/ansible-tmp-1546499825.06-70753836802764/AnsiballZ_setup.py': Operation not permitted\n}). For information on working around this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}
      to retry, use: --limit @/opt/container_artifact/ansible-retry/site.retry

 PLAY RECAP *********************************************************************
 localhost                  : ok=0    changed=0    unreachable=0    failed=1

 Thursday 03 January 2019  08:17:05 +0100 (0:00:00.281)       0:00:00.317 ******
 ===============================================================================
 Gathering Facts --------------------------------------------------------- 0.28s```

[arctan5x]

Just got off the call with @WaughB. Seems like the issue exists in linux environment and not MacOS. Current workaround would be using 7.2.1 or below container. Will be working on the fix immediately.

Edit:
Could not reproduce with the same Linux distribution and the same version of Linux Kernal. All other dependencies were identical. After resetting the Docker environment and dependencies, Brett's container started working.

[WaughB]

After speaking with @arctan5x and @nwang92 I decided to completely remove and reinstall everything Docker on my box. These are just the steps I did, I few may seem redundant and excessive but it worked in the end.

sudo apt-get remove docker docker-engine docker.io

dpkg -l | grep -i docker

sudo apt-get purge -y docker-engine docker docker.io docker-ce  
sudo apt-get autoremove -y --purge docker-engine docker docker.io docker-ce 

sudo rm -rf /var/lib/docker
sudo rm /etc/apparmor.d/docker
sudo groupdel docker
sudo rm -rf /var/run/docker.sock
sudo rm -rf  /etc/docker

sudo groupdel docker

After all of that I restarted my system, then proceeded to reinstall everything based on the instructions from the official page. I tried doing just the uninstall instructions from that page but on my first reinstall it did not work. I ended up blending it with the instructions from here. Also made sure to follow the after install instructions located here.

Hope this helps out anyone who is experiencing similar issues.

[arctan5x]

Going to close this issue for now since @WaughB has resolved his case. @JanGoerss let us know if you still run into the same issue after trying out Brett's solution.

[markgifford]

Just wanted to say I had the same issue as OP trying to pull the latest image. Same Docker engine but Ubuntu 17.04 as the host OS.

I updated to:
Docker version 18.09.1, build 4c52b90

But I had the same result. In the end I just grabbed image 7.2.1 and it was fine for me (just looking to test out Splunk, don't necessarily need the latest release).

[arctan5x]

@markgifford Did you remove the Docker engine and all its dependencies completely and reinstalled?

[markgifford]

@arctan5x No, I didn't sorry. Maybe I will try that in a few weeks if I continue using Splunk but for now I'm OK just playing around with 7.2.1.

[arctan5x]

@markgifford Ok.

[figaro001]

The same problem (
2019-01-21 19:25:28,483 p=8 u=ansible | fatal: [localhost]: FAILED! => {"msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc: 1, err: chown: changing ownership of '/var/tmp/ansible-tmp-1548098727.84-181837930380149/': Operation not permitted\nchown: changing ownership of '/var/tmp/ansible-tmp-1548098727.84-181837930380149/AnsiballZ_setup.py': Operation not permitted\n}). For information on working around this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}
) is present in Synology. Version 7.2.1 runs in Docker on Synology without errors.

[arctan5x]

@figaro001 would you try Brett's solution as outlined above?

[figaro001]

I installed the Docker image on the newly purchased Synology NAS. Installed a clean system, Docker and downloaded the image.

[matt-wirth]

Having the same issue. Purging docker assets and reinstalling had no effect. Platform is Ubuntu Server.

[lephino]

Did you guys happen to take a look at #105?

[matt-wirth]

Very curious. Personally not using a snap, using the official docker repo for Ubuntu.

Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:31 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:02:44 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[neogeek83]

I'm experiencing the same issue on a fresh installs of docker.
Info on the system:

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"

uname -r
4.15.0-43-generic

docker -v
Docker version 18.09.1, build 4c52b90

docker pull splunk/splunk:latest
docker run -it -d -p 8000:8000 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=changeme' splunk/splunk:latest

docker logs 51134f959f22

PLAY [Run default Splunk provisioning] *****************************************
Friday 25 January 2019  18:05:17 +0000 (0:00:00.039)       0:00:00.039 ******** 

TASK [Gathering Facts] *********************************************************
fatal: [localhost]: FAILED! => {"msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc: 1, err: chown: changing ownership of '/var/tmp/ansible-tmp-1548439517.96-190759433573090/': Operation not permitted\nchown: changing ownership of '/var/tmp/ansible-tmp-1548439517.96-190759433573090/AnsiballZ_setup.py': Operation not permitted\n}). For information on working around this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}
	to retry, use: --limit @/opt/container_artifact/ansible-retry/site.retry

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1   

Friday 25 January 2019  18:05:18 +0000 (0:00:00.314)       0:00:00.354 ******** 
=============================================================================== 
Gathering Facts --------------------------------------------------------- 0.31s

I can also confirm that if I docker pull splunk/splunk:7.2.1 splunk starts working again.

[acobaugh]

Same error here trying to run 7.2.3. Distro is VoidLinux.

Though, of note, ansible in 7.2.1 gave me a slightly different error after trying to splunkd about an "unusable filesystem" - I attribute that to running on ZFS, which is a problem I'll have to tackle at some point.

% docker version
Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.11.4
 Git commit:
 Built:             Tue Jan 22 09:03:19 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.4
  Git commit:       v18.09.1
  Built:            Tue Jan 22 09:03:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Logs from docker run -d -p 8000:8000 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=foobarbaz123!' splunk/splunk:7.2.3:

PLAY [Run default Splunk provisioning] *****************************************
Thursday 07 February 2019  19:59:00 +0000 (0:00:00.052)       0:00:00.052 *****

TASK [Gathering Facts] *********************************************************
fatal: [localhost]: FAILED! => {"msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc: 1, err: chown: changing ownership of '/var/tmp/ansible-tmp-1549569540.05-226917218255164/': Operation not permitted\nchown: changing ownership of '/var/tmp/ansible-tmp-1549569540.05-226917218255164/AnsiballZ_setup.py': Operation not permitted\n}). For information on working around this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}
	to retry, use: --limit @/opt/container_artifact/ansible-retry/site.retry

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1

Thursday 07 February 2019  19:59:00 +0000 (0:00:00.387)       0:00:00.440 *****
===============================================================================
Gathering Facts --------------------------------------------------------- 0.39s

[neogeek83]

Just a quick followup to my last post, I have been confirm at least in my case, that 'aufs' is still the default file system driver being used for a splunk install and @lephino comment and link to #105 helps. Just for completeness, I will go ahead and include how I fixed this here.

First, I confirmed that I was(by default I did NOT set it anywhere) in fact using aufs I determined this by running:

docker inspect splunk|grep Driver

If you see:

        "Driver": "aufs",
            "Name": "aufs"

Then this is also the case for you. For reference, running docker --version returns:

Docker version 18.09.3, build 774a1f4

And if I run docker info|grep "Storage Driver" I get:

Storage Driver: aufs

So, I clearly have a docker build that is supposed to be using overlay2 as the backing driver by default, but for whatever reason is using the legacy aufs. Let's fix it. First review these and note that you will have to DELETE AND RECREATE ALL CONTAINERS FOR THIS TO WORK.

• https://docs.docker.com/storage/storagedriver/select-storage-driver/
• https://docs.docker.com/storage/storagedriver/overlayfs-driver/

Okay, now let's get to it:

• sudo vim /etc/docker/daemon.json and add: "storage-driver": "overlay2" to the root {} of the json
• Next, restart docker daemon sudo systemctl restart docker
• Next confirm the change took should show overlay2: docker inspect splunk|grep Driver

 "Driver": "overlay2",
            "VolumeDriver": "",
        "GraphDriver": {
                    "DriverOpts": null

• Now you should be able to recreate your containers and they will start using the overlay2 and the splunk permissions error will go away(least it did for me).