distributing.py example in documentation not working (bit-rot)

[GoogleCodeExporter]

The distributing.py example (for distributing processes across machines)
given at the end of the documentation seems broken, or at least bit-rotten.
I can't even get a simple example (attached) to work: I get various weird
exceptions.

Original issue reported on code.google.com by christia...@gmail.com on 9 Dec 2008 at 9:48

Attachments:

• test.py

[GoogleCodeExporter]

The following patch fixes at least some of the problems encountered while 
trying to
make distributing.py work:

- fix typo in _logger.propagate (not "propogate")
- use new "name" property of process, instead of set_name() and get_name()
- replace managers.transact() (which is not available anymore) by its old
implementation (which uses managers.dispatch)
- Passing string with embedded newlines as argument to "python -c" doesn't seem 
to
work, at least not with tcsh as a login shell. Pass python code snippet that 
uses ";"
to separate statements instead.
- Add missing last argument ("pickle") to managers.Server() 

There's still some bitrot remaining, apparently, as the changes above are not 
enough
to make the simple "test.py" example (attached previously) work.

Original comment by christia...@gmail.com on 9 Dec 2008 at 9:59

Attachments:

• patch1.diff

[GoogleCodeExporter]

After changing the REMOTE_HOSTNAME_HERE with the hostname of my machine (not
"localhost", it is special-cased by distributing.py), I get the following 
backtrace
when running test.py:

Traceback (most recent call last):
  File "./test.py", line 18, in <module>
    main()
  File "./test.py", line 11, in main
    cluster.start()
  File "/home/chrish/distributing-to-upstream/distributing.py", line 204, in start
    self._authkey)
  File "/home/chrish/distributing-to-upstream/distributing.py", line 91, in from_address
    managers.dispatch(conn, None, 'dummy')
  File "/home/chrish/python_modules/multiprocessing/managers.py", line 79, in dispatch
    kind, result = c.recv()
cPickle.UnpicklingError: invalid load key, '#'.

Original comment by christia...@gmail.com on 10 Dec 2008 at 8:52

[GoogleCodeExporter]

cPickle.UnpicklingError: invalid load key, '#'.

I'm running into this same problem with the example. Has anyone figured out how 
to
fix this issue? I know this is an old issue, but any help would be great! 
Thanks!

Original comment by mkan...@gmail.com on 24 Sep 2009 at 6:31

[GoogleCodeExporter]

Commenting here because I googled for this error and couldn't find any leads, so
figured it out myself.

In my case, I suspect this was caused by pickling to the same file twice. The 
first
pickle failed mid-stream, but I changed it and repickled. The file starts with 
some
text clearly referencing something I removed in the second try, so I suspect 
that
some buffer still held data from the failed attempt (the file hadn't gotten 
closed
because the exception kept a reference to it). That buffer could have been in 
Python,
libc, or maybe even the kernel, can't tell. But the "invalid load key" was right
after the overwritten chunk; it was a character in the middle of a text stream 
from
the new pickle. Try %debug (ipython) on the pure-Python pickle module to see 
where it is.

Original comment by kenneth.arnold on 23 Oct 2009 at 6:04

[GoogleCodeExporter]

I got the cPickle.UnpicklingError: invalid load key, '#' error as well. For me 
the problem was the authkey argument to the connection.Client which was passed 
in the 'family' position.

Here is a fix:
-conn = connection.Client(address, authkey)
+conn = connection.Client(address, authkey=authkey)

But now I get another error:
AttributeError: 'AutoProxy[_RemoteProcess]' object has no attribute 'exitcode'

Original comment by henjo2...@gmail.com on 23 May 2012 at 5:04