bpo-35917: Test multiprocessing manager classes and shareable types

[giampaolo]

Related to PR #11664. BPO reference is: https://bugs.python.org/issue35917.

https://bugs.python.org/issue35917

Tests can be run with ./python -m test -v _test_multiprocessing.

[terryjreedy]

PR titles must begin with the bpo number when there is one. Note issue-number check to allow merge. This automatically generates link in details and listing of PR on the issue. I changed rest so not too long.

A news item is also required. Use blurb. (Both checks can be over-ridden by labels, but these are used rarely, for things such as typo fixes.)

[pitrou]

test_multiprocessing_forkserver and test_multiprocessing_spawn fail with a lot of errors here. You might try it for yourself:

./python -m test -m "*Manager*" -v test_multiprocessing_forkserver

[bedevere-bot]

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

And if you don't make the requested changes, you will be poked with soft cushions!

[giampaolo]

OK, it appears failures on Windows are fixed.

[pitrou]

Hmm, I don't think this is the right solution. You must keep the tests inside _test_multiprocessing, because they are pickled up by all three of test_multiprocessing_{fork,forkserver,spawn}. What you should try is to use classmethods for the process target functions, rather than nested functions.

For example:

    @classmethod
    def _test_event(cls, obj):
        assert obj.is_set()
        obj.wait()
        obj.clear()
        obj.wait(0.001)

    def test_event(self):
        o = self.manager.Event()
        o.set()
        self.run_test(self._test_event, o)
        assert not o.is_set()
        o.wait(0.001)

[pitrou]

@giampaolo I think you misunderstood the test failures. A RLock can only be released by the thread/process which acquired it. Example:

>>> import threading                                                                      
>>> lock = threading.RLock()                                                              
>>> lock.acquire()                                                                        
True
>>> t = threading.Thread(target=lock.release)                                             
>>> t.start()                                                                             
Exception in thread Thread-1990:
Traceback (most recent call last):
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
RuntimeError: cannot release un-acquired lock

Same with multiprocessing:

>>> lock = mp.RLock()                                                                     
>>> lock.acquire()                                                                        
True
>>> proc = mp.Process(target=lock.release)                                                
>>> proc.start()                                                                          
Process Process-1:
>>> Traceback (most recent call last):                                                    
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
AssertionError: attempt to release recursive lock not owned by thread

So you should write the RLock and Condition tests differently.

[giampaolo]

Thanks Antoine. That was the problem indeed (got tricked by the fact that for some reason Linux behaves differently). Good to merge?

[pitrou]

It's actually a bit weird that Linux behaves differently. Can you open an issue for that?

[miss-islington]

Thanks @giampaolo for the PR, and @pitrou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

[miss-islington]

Sorry, @giampaolo and @pitrou, I could not cleanly backport this to 3.7 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 2848d9d29914948621bce26bf0d9a89f2e19b97b 3.7

[bedevere-bot]

GH-11780 is a backport of this pull request to the 3.7 branch.

[giampaolo]

Will do. Do you think we should backport this to 3.7 only? Or also for previous releases?

[pitrou]

3.7 only. Backporting to 2.7 would probably be more involved.

[pablogsal]

@pitrou @giampaolo

This PR broke some buildbots with refleak checks because the tests modify the environment (leaking processes or threads or files ...). Some of the failures:

https://buildbot.python.org/all/#builders/80/builds/506
ttps://buildbot.python.org/all/#builders/114/builds/375
https://buildbot.python.org/all/#builders/1/builds/497

I opened https://bugs.python.org/issue35940 to track this.

I think I can find some time tonight to check if I am able to find out where the problem is.