Skip to content

Conversation

AnTuanHa
Copy link
Contributor

@AnTuanHa AnTuanHa commented Jun 13, 2017

When using Python's multiprocessing module, it throws an AttributeError, complaining that the object does not have the attribute used. This adds the missing attributes and allows them to be pickled.

@AnTuanHa
Copy link
Contributor Author

AnTuanHa commented Jun 19, 2017

@shin- Any chance on gettting this merged? Here's a minimal working example of the error occurring:

Create a local docker-machine: docker-machine create -d virtualbox testmachine
Then run:

import docker
import multiprocessing as mp
import re
import subprocess

def foo(client):
    pass

if __name__ == '__main__':
    cmd = ['docker-machine', 'config', 'testmachine']
    p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, _ = p.communicate()
    output = stdout.decode('utf-8')
    regex = """(--tlsverify\n)?--tlscacert="(.+)"\n--tlscert="(.+)"\n--tlskey="(.+)"\n-H=(.+)"""
    match = re.match(regex, output.strip())
    tlsverify, tlscacert, tlscert, tlskey, host = match.group(1, 2, 3, 4, 5)
    tlsverify = bool(tlsverify)

    if tlsverify:
        base_url = host.replace('tcp://', 'https://')
        tls = docker.tls.TLSConfig(client_cert=(tlscert, tlskey),
                                   ca_cert=tlscacert,
                                   verify=True)


        client = docker.DockerClient(base_url=base_url, tls=tls)

        pool = mp.Pool()
        job = pool.apply_async(foo, args=[client])
        job.wait()

The error:

Process ForkPoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 345, in get
    return _ForkingPickler.loads(res)
  File "/tmp/testlocal/lib/python3.6/site-packages/requests/adapters.py", line 142, in __setstate__
    block=self._pool_block)
  File "/tmp/testlocal/lib/python3.6/site-packages/docker/transport/ssladapter.py", line 40, in init_poolmanager
    'assert_hostname': self.assert_hostname,
AttributeError: 'SSLAdapter' object has no attribute 'assert_hostname'

@shin-
Copy link
Contributor

shin- commented Jun 19, 2017

LGTM - should we do a similar thing with UnixAdapter and NpipeAdapter as well?

@AnTuanHa AnTuanHa force-pushed the add_attrs_for_pickling branch 3 times, most recently from 77f2d4a to 365a18c Compare June 20, 2017 13:36
When using the multiprocessing module, it throws an AttributeError,
complaining that the object does not have the attribute used. This adds
the missing attributes and allows them to be pickled.

Signed-off-by: An Ha <zero579911@gmail.com>
@AnTuanHa AnTuanHa force-pushed the add_attrs_for_pickling branch from 365a18c to 0165a34 Compare June 20, 2017 13:40
@AnTuanHa
Copy link
Contributor Author

@shin- Yup, also rebased on top of latest upstream/master.

@shin-
Copy link
Contributor

shin- commented Jun 22, 2017

Great, thank you!

@shin- shin- merged commit b9e5863 into docker:master Jun 22, 2017
@shin- shin- added this to the 2.4.0 milestone Jun 22, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants