dbus_init() hangs when called in a script that calls another instance via sudo --preserve-env

[mwhahaha]

I'm not certain if this is a jeepney or secretstorage issue, but I'm opening it here. The bug context is that if you have a script (a.py) which performs a secretstorage.dbus_init() and the script uses subprocess to execute another script (b.py) using sudo --preserve-env, the second secretstorage.dbus_init() hangs.

a.py:

#!/usr/bin/env python3
import os
import secretstorage
import secretstorage.exceptions as exceptions
import subprocess


def launch():
    print('a launch()')
    try:
        print('a dbus_init()')
        bus = secretstorage.dbus_init()
    except exceptions.SecretStorageException as e:
        raise RuntimeError("Unable to initialize SecretService: %s" % e)
    p = os.getcwd()
    cmd = ['sudo', '--preserve-env', 'python3', '{}/b.py'.format(p)]
    #cmd = ['python3', '{}/b.py'.format(p)]
    try:
        print('a subprocess')
        subprocess.check_call(cmd)
    except Exception as e:
        print('a exception')
        raise
    print("a done")

if __name__ == '__main__':
    print('a main()')
    launch()

b.py

#!/usr/bin/env python3
import secretstorage
import secretstorage.exceptions as exceptions

def launch():
    print("b launch()")
    try:
        print('b dbus_init()')
        bus = secretstorage.dbus_init()
    except exceptions.SecretStorageException as e:
        raise RuntimeError("Unable to initialize SecretService: %s" % e)
    print("b done")


if __name__ == '__main__':
    print('b main()')
    launch()

Resulting output

[centos@undercloud test]$ ./a.py 
a main()
a launch()
a dbus_init()
a subprocess
b main()
b launch()
b dbus_init()

It will hang on the second dbus_init. A ctrl+c results in this traceback:

^CTraceback (most recent call last):
  File "/home/centos/test/b.py", line 17, in <module>
    launch()
  File "/home/centos/test/b.py", line 9, in launch
    bus = secretstorage.dbus_init()
  File "/usr/lib/python3.6/site-packages/secretstorage/__init__.py", line 37, in dbus_init
    connection = connect_and_authenticate()
  File "/usr/lib/python3.6/site-packages/jeepney/integrate/blocking.py", line 102, in connect_and_authenticate
    conn = DBusConnection(sock)
  File "/usr/lib/python3.6/site-packages/jeepney/integrate/blocking.py", line 40, in __init__
    hello_reply = self.bus_proxy.Hello()
  File "/usr/lib/python3.6/site-packages/jeepney/integrate/blocking.py", line 84, in inner
    return self._connection.send_and_get_reply(msg)
  File "/usr/lib/python3.6/site-packages/jeepney/integrate/blocking.py", line 67, in send_and_get_reply
    self.recv_messages()
  File "/usr/lib/python3.6/site-packages/jeepney/integrate/blocking.py", line 55, in recv_messages
    b = self.sock.recv(4096)
KeyboardInterrupt
a exception
Traceback (most recent call last):
  File "/usr/lib64/python3.6/subprocess.py", line 289, in call
    return p.wait(timeout=timeout)
  File "/usr/lib64/python3.6/subprocess.py", line 1477, in wait
    (pid, sts) = self._try_wait(0)
  File "/usr/lib64/python3.6/subprocess.py", line 1424, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./a.py", line 28, in <module>
    launch()
  File "./a.py", line 20, in launch
    subprocess.check_call(cmd)
  File "/usr/lib64/python3.6/subprocess.py", line 306, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 291, in call
    p.kill()
  File "/usr/lib64/python3.6/subprocess.py", line 1610, in kill
    self.send_signal(signal.SIGKILL)
  File "/usr/lib64/python3.6/subprocess.py", line 1600, in send_signal
    os.kill(self.pid, sig)
PermissionError: [Errno 1] Operation not permitted

This is on CentOS8.

Pip freeze

alembic==1.3.1
amqp==2.5.2
ansible==2.8.8
ansible-config-template==1.0.1
ansible-pacemaker==1.0.4
ansible-role-atos-hsm==0.1.1
ansible-role-chrony==1.0.2
ansible-role-container-registry==1.1.1
ansible-role-openstack-operations==0.0.1
ansible-role-thales-hsm==0.2.1
ansible-role-tripleo-modify-image==1.1.1
ansible-runner==1.4.4
anyjson==0.3.3
aodhclient==1.5.0
appdirs==1.4.3
asn1crypto==0.24.0
attrs==17.4.0
Babel==2.5.1
bcrypt==3.1.7
beautifulsoup4==4.8.2
boto==2.49.0
Bottleneck==1.2.1
cachetools==4.0.0
cffi==1.13.2
chardet==3.0.4
cliff==2.18.0
cmd2==0.9.16
colorama==0.4.1
configobj==5.0.6
contextlib2==0.5.5
croniter==0.3.27
cryptography==2.8
cssselect==0.9.2
cycler==0.10.0
dbus-python==1.2.4
debtcollector==1.22.0
decorator==4.4.0
dib-utils==0.0.11
diskimage-builder==2.33.1.dev18
dnspython==1.15.0
docutils==0.14
dogpile.cache==0.9.0
entrypoints==0.3
eventlet==0.25.1
fasteners==0.14.1
flake8==3.7.7
fluidity-sm==0.2.0
funcsigs==1.0.2
futurist==1.10.0
gevent==1.2.2
gitdb2==2.0.3
GitPython==3.0.2
gpg==1.10.0
greenlet==0.4.13
heat-cfntools==1.4.2
html5lib==0.999999999
idna==2.5
importlib-metadata==0.23
invoke==1.4.0
ironic-python-agent-builder==1.1.1.dev44
iso8601==0.1.11
jeepney==0.4.1
Jinja2==2.10.1
jmespath==0.9.0
jsonpatch==1.21
jsonpointer==1.10
jsonschema==2.6.0
keyring==21.0.0
keystoneauth1==3.18.0
keystonemiddleware==8.0.0
kiwisolver==1.1.0
kombu==4.6.6
lexicon==1.0.0
lockfile==0.11.0
logutils==0.3.5
lxml==4.2.3
Mako==1.0.6.dev0
MarkupSafe==0.23
matplotlib==3.1.1
mccabe==0.6.1
metalsmith==0.16.0
mistral-lib==1.4.0
mock==3.0.5
monotonic==1.5
msgpack==0.6.2
munch==2.3.2
netaddr==0.7.19
netifaces==0.10.6
networkx==2.3
neutron-lib==1.31.0
numexpr==2.7.1
numpy==1.14.3
openstack-heat==13.1.0.dev143
openstacksdk==0.40.0
os-apply-config==11.0.1.dev1
os-client-config==1.34.0.dev6
os-collect-config==11.0.0
os-ken==0.4.1
os-net-config==12.1.1.dev1
os-refresh-config==10.4.1.dev1
os-service-types==1.7.0
os-traits==2.2.0
osc-lib==2.0.0
oslo.cache==1.38.1
oslo.concurrency==3.31.0
oslo.config==6.12.0
oslo.context==2.23.0
oslo.db==6.0.0
oslo.i18n==3.25.1
oslo.log==3.45.2
oslo.messaging==10.5.0
oslo.middleware==3.38.1
oslo.policy==2.4.1
oslo.reports==1.31.1
oslo.rootwrap==5.17.1
oslo.serialization==2.29.2
oslo.service==1.41.1
oslo.upgradecheck==0.3.2
oslo.utils==3.42.1
oslo.versionedobjects==1.37.0
osprofiler==2.9.0
ovs==2.12.0
pandas==0.25.3
paramiko==2.7.1
passlib==1.7.1
Paste==3.2.4
PasteDeploy==2.0.1
paunch==6.0.2.dev2
pbr==5.4.3
pciutils==2.3.6
pecan==1.3.2
perf==0.1
pexpect==4.7.0
Pillow==5.1.1
ply==3.9
prettytable==0.7.2
psutil==5.6.3
ptyprocess==0.5.2
pyasn1==0.4.6
pycadf==2.10.0
pycairo==1.16.3
pycodestyle==2.5.0
pycparser==2.14
pydot==1.4.1
pyflakes==2.1.1
pygobject==3.28.3
pygraphviz==1.5
pyinotify==0.9.6
PyMySQL==0.8.0
PyNaCl==1.3.0
pyngus==2.3.0
pyOpenSSL==18.0.0
pyparsing==2.4.6
pyperclip==1.6.4
PySocks==1.6.8
pystache==0.5.4
python-barbicanclient==4.9.0
python-cinderclient==5.0.0
python-daemon==2.2.3
python-dateutil==2.6.1
python-designateclient==3.0.0
python-dmidecode==3.12.2
python-editor==1.0.4
python-glanceclient==2.17.0
python-heatclient==1.18.0
python-ironic-inspector-client==3.7.0
python-ironicclient==3.1.1
python-keystoneclient==3.22.0
python-linux-procfs==0.6
python-magnumclient==2.16.0
python-manilaclient==1.29.0
python-memcached==1.58
python-mistralclient==3.10.0
python-monascaclient==1.16.0
python-neutronclient==6.14.0
python-novaclient==16.0.0
python-octaviaclient==2.0.0
python-openstackclient==4.0.0
python-qpid-proton==0.30.0
python-saharaclient==2.3.0
python-swiftclient==3.8.1
python-tripleoclient==13.1.1.dev59
python-troveclient==3.2.0
python-zaqarclient==1.12.0
pytz==2017.2
pyudev==0.21.0
PyYAML==5.3
pyzmq==18.1.0
repoze.lru==0.7
requests==2.22.0
requestsexceptions==1.4.0
rfc3986==1.2.0
rhnlib==2.8.6
Routes==2.4.1
rpm==4.14.2
rsa==3.4.2
schedutils==0.6
scipy==1.0.0
SecretStorage==3.1.1
selinux==2.9
sepolicy==1.1
setools==4.2.2
setproctitle==1.1.10
shade==1.32.0
simplegeneric==0.8.1
simplejson==3.17.0
singledispatch==3.4.0.3
six==1.14.0
slip==0.6.4
slip.dbus==0.6.4
smmap2==2.0.3
sos==3.7
soupsieve==1.9.2
SQLAlchemy==1.3.2
sqlalchemy-migrate==0.13.0
sqlparse==0.2.4
statsd==3.2.1
stevedore==1.31.0
syspurpose==1.25.17
tables==3.5.2
Tempita==0.5.1
tenacity==5.1.1
tinyrpc==1.0.3
tripleo-ansible==1.2.1
tripleo-common==12.1.1.dev38
tripleo-heat-templates==12.1.1.dev94
tripleo-image-elements==11.0.2.dev1
tripleo-ipsec==9.2.1
tripleo-puppet-elements==12.1.1.dev1
tripleo-repos==0.0.1.dev96
tripleo-validations==12.1.1.dev4
urllib3==1.25.7
vine==1.3.0
waitress==1.4.2
warlock==1.3.3
wcwidth==0.1.7
webencodings==0.5.1
WebOb==1.8.5
websocket-client==0.56.0
WebTest==2.0.33
Werkzeug==0.16.0
wrapt==1.11.2
yappi==1.0
yaql==1.1.3
zipp==0.5.1

This is the stripped down version of the issue. This also happens if you just import keyring which makes it even harder to track down.

[mwhahaha]

It should be noted this works fine without sudo. It fails with sudo without --preserve-env like so:

a main()
a launch()
a dbus_init()
a subprocess
b main()
b launch()
b dbus_init()
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/secretstorage/__init__.py", line 37, in dbus_init
    connection = connect_and_authenticate()
  File "/usr/lib/python3.6/site-packages/jeepney/integrate/blocking.py", line 90, in connect_and_authenticate
    bus_addr = get_bus(bus)
  File "/usr/lib/python3.6/site-packages/jeepney/bus.py", line 53, in get_bus
    return find_session_bus()
  File "/usr/lib/python3.6/site-packages/jeepney/bus.py", line 42, in find_session_bus
    addr = os.environ['DBUS_SESSION_BUS_ADDRESS']
  File "/usr/lib64/python3.6/os.py", line 669, in __getitem__
    raise KeyError(key) from None
KeyError: 'DBUS_SESSION_BUS_ADDRESS'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/centos/test/b.py", line 9, in launch
    bus = secretstorage.dbus_init()
  File "/usr/lib/python3.6/site-packages/secretstorage/__init__.py", line 43, in dbus_init
    raise SecretServiceNotAvailableException(reason) from ex
secretstorage.exceptions.SecretServiceNotAvailableException: Environment variable DBUS_SESSION_BUS_ADDRESS is unset

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/centos/test/b.py", line 17, in <module>
    launch()
  File "/home/centos/test/b.py", line 11, in launch
    raise RuntimeError("Unable to initialize SecretService: %s" % e)
RuntimeError: Unable to initialize SecretService: Environment variable DBUS_SESSION_BUS_ADDRESS is unset
a exception
Traceback (most recent call last):
  File "./a.py", line 29, in <module>
    launch()
  File "./a.py", line 21, in launch
    subprocess.check_call(cmd)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', 'python3', '/home/centos/test/b.py']' returned non-zero exit status 1.

[mitya57]

This is a quite exotic configuration…

My hypothesis is that:

• When you use sudo, the effective user id is 0, so Jeepney uses that for authentication.
• However if you also pass the --preserve-env, the DBUS_SESSION_BUS_ADDRESS is present in the new environment, so Jeepney uses that to connect to the bus.
• And probably connecting to a user's session bus when authenticated as a different user (root) does not work.

I'm not sure if this can be fixed at all, but if it can then the fix should be in Jeepney, not in SecretStorage. If I am wrong then pull requests are welcome.

[mwhahaha]

Just FYI this worked under 2.3.1. I'm not certain the best way to open issues against Jeepney :/

[mwhahaha]

I found the jeepney issue board. I'll open one over there as well. This isn't as exotic as you think because this is triggered simply by 'import keyring'

[takluyver]

To summarise the discussions on Jeepney: the hang seen on CentOS 8 is a bug in Jeepney, which I aim to fix by raising the same ConnectionResetError that already occurs on some other platforms.

keyring is meant to gracefully handle errors setting up the backend and disable it without crashing on import, and on my machine it does. See code here and here. So I believe that the crashes on importing keyring are/were a bug with that package.

[mitya57]

If https://gitlab.com/takluyver/jeepney/-/merge_requests/13 is merged then no changes are needed in SecretStorage or keyring. SecretStorage will convert ConnectionResetError to SecretServiceNotAvailableException, and keyring will catch that and treat the backend as not viable.

In case there is a new type of exception that jeepney can raise, I am fine with changing SecretStorage to intercept it.

[mwhahaha]

https://gitlab.com/takluyver/jeepney/-/merge_requests/13 resolves it as SecretStorage will bubble up the error which keyring handles.