permission denied since kernel upgrade from kernel-3.10.0-514.16.1.el7.x86_64 to kernel-3.10.0-514.21.1.el7.x86_64

[ledzepp4eva]

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible

ANSIBLE VERSION

ansible 2.3.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Aug  2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]

CONFIGURATION

OS / ENVIRONMENT

Red Hat Enterprise Linux Server release 7.3 (Maipo)
3.10.0-514.21.1.el7.x86_64

SUMMARY

STEPS TO REPRODUCE

Since the kernel update all playbooks I try to run through the error below

Running any ansible commands for example:

ansible -i playbooks/staging.hosts example.org -m facts

EXPECTED RESULTS

I would get facts back from the server

ACTUAL RESULTS

ansible -i playbooks/staging.hosts example.org -m facts -vvvv
Using /etc/ansible/ansible.cfg as config file
ERROR! Unexpected Exception: [Errno 13] Permission denied
the full traceback was:

Traceback (most recent call last):
  File "/usr/bin/ansible", line 109, in <module>
    exit_code = cli.run()
  File "/usr/lib/python2.7/site-packages/ansible/cli/adhoc.py", line 197, in run
    run_tree=run_tree,
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 102, in __init__
    self._final_q = multiprocessing.Queue()
  File "/usr/lib64/python2.7/multiprocessing/__init__.py", line 218, in Queue
    return Queue(maxsize)
  File "/usr/lib64/python2.7/multiprocessing/queues.py", line 63, in __init__
    self._rlock = Lock()
  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 147, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 13] Permission denied

However if I do downgrade the kernel back it works fine. Please let me know if you believe this isn't any ansible issue and I can speak to redhat about this.

I do run selinux, but setting this to permissive still ends in the same result.

Thanks,
Liam

[robbmanes]

In the event it helps, feel free to ignore:

# ansible --version
ansible 2.3.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Aug  2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]

# uname -r
3.10.0-514.21.1.el7.x86_64

# getenforce
Enforcing

Although it seems like mine works (I tried from EPEL and built from git both):

# ansible -i hosts webservers -m setup
[works]

I don't have the facts module handy though so I just used setup. Judging from the calltrace ansible is just creating a workqueue using the multiprocessing module so I'd suspect that ansible isn't at fault here.

# python
Python 2.7.5 (default, Aug  2 2016, 04:20:16) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> _final_q = multiprocessing.Queue()
>>> type(_final_q)
<class 'multiprocessing.queues.Queue'>

Note this is creating the queue which does use a semaphore in the multiprocessing module, here:

class SemLock(object):

    def __init__(self, kind, value, maxvalue):
        sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
        debug('created semlock with handle %s' % sl.handle)
        self._make_methods()

Long shot, but perhaps something changed with your semaphore permissions? I note this person had similar problems. Note mine, which is working, is like this:

# ls -ld /dev/shm
drwxrwxrwt. 2 root root 40 May 31 11:33 /dev/shm

That being said, I am not sure why one kernel would be affected and the other wouldn't if the above is true. Maybe the ramdisks are different when you reboot between kernels and you can rebuild them with dracut -f?

If it keeps happening despite right semaphore permissions perhaps someone else can chime in with a better idea. Sorry if this perhaps veers us off course.

[robbmanes]

In the event it helps, feel free to ignore:

# ansible --version
ansible 2.3.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Aug  2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]

# uname -r
3.10.0-514.21.1.el7.x86_64

# getenforce
Enforcing

Although it seems like mine works (I tried from EPEL and built from git both):

# ansible -i hosts webservers -m setup
[works]

I don't have the facts module handy though so I just used setup. Judging from the calltrace ansible is just creating a workqueue using the multiprocessing module so I'd suspect that ansible isn't at fault here.

# python
Python 2.7.5 (default, Aug  2 2016, 04:20:16) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> _final_q = multiprocessing.Queue()
>>> type(_final_q)
<class 'multiprocessing.queues.Queue'>

Note this is creating the queue which does use a semaphore in the multiprocessing module, here:

class SemLock(object):

    def __init__(self, kind, value, maxvalue):
        sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
        debug('created semlock with handle %s' % sl.handle)
        self._make_methods()

Long shot, but perhaps something changed with your semaphore permissions? I note this person had similar problems. Note mine, which is working, is like this:

# ls -ld /dev/shm
drwxrwxrwt. 2 root root 40 May 31 11:33 /dev/shm

That being said, I am not sure why one kernel would be affected and the other wouldn't if the above is true. Maybe the ramdisks are different when you reboot between kernels and you can rebuild them with dracut -f?

If it keeps happening despite right semaphore permissions perhaps someone else can chime in with a better idea. Sorry if this perhaps veers us off course.

[ledzepp4eva]

Hey @robbmanes

liam@folk: ~/cdn-ddos ls -ld /dev/shm
drwxr-xr-t. 2 root root 60 Jun  1 06:52 /dev/shm
liam@folk: ~/cdn-ddos sudo chmod 1777 /dev/shm
liam@folk: ~/cdn-ddos ls -ld /dev/shm
drwxrwxrwt. 2 root root 60 Jun  1 06:52 /dev/shm

and now it runs.

I will have a look through the kernel change logs to see what has changed there, but thank you very much for your help.

Liam

[ledzepp4eva]

Hey @robbmanes

liam@folk: ~/cdn-ddos ls -ld /dev/shm
drwxr-xr-t. 2 root root 60 Jun  1 06:52 /dev/shm
liam@folk: ~/cdn-ddos sudo chmod 1777 /dev/shm
liam@folk: ~/cdn-ddos ls -ld /dev/shm
drwxrwxrwt. 2 root root 60 Jun  1 06:52 /dev/shm

and now it runs.

I will have a look through the kernel change logs to see what has changed there, but thank you very much for your help.

Liam