test_multiprocessing fails on systems with HAVE_SEM_OPEN=0 #48020
Comments
|
test_multiprocessing crashes on platforms that lack a working I think the multiprocessing module should disable the functionality Failure message: test test_multiprocessing crashed -- <type 'exceptions.AttributeError'>:
'module' object has no attribute 'SemLock'
Traceback (most recent call last):
File ".//Lib/test/regrtest.py", line 556, in runtest_inner
indirect_test()
File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/test/test_multiprocessing.py",
line 1758, in test_main
ProcessesMixin.pool = multiprocessing.Pool(4)
File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/multiprocessing/__init__.py",
line 226, in Pool
return Pool(processes, initializer, initargs)
File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/multiprocessing/pool.py",
line 84, in __init__
self._setup_queues()
File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/multiprocessing/pool.py",
line 130, in _setup_queues
from .queues import SimpleQueue
File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/multiprocessing/queues.py",
line 22, in <module>
from multiprocessing.synchronize import Lock, BoundedSemaphore,
Semaphore, Condition
File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/multiprocessing/synchronize.py",
line 29, in <module>
SEM_VALUE_MAX = _multiprocessing.SemLock.SEM_VALUE_MAX
AttributeError: 'module' object has no attribute 'SemLock'
1 test failed:
test_multiprocessing |
|
Which platforms is this appearing on? |
|
On Thu, 4 Sep 2008, Jesse Noller wrote:
OpenBSD, with this section added to setup.py: @@ -1269,6 +1268,14 @@ class PyBuildExt(build_ext): + elif platform.startswith('openbsd'): |
|
2.6rc1 shows the same failure traceback on FreeBSD 6.3, which is covered |
|
So the bug is actually in the multiprocessing module rather than the I guess _multiprocessing could always define a dummy SemLock or (it would be great to see this fixed for 2.6) -d |
|
Bumping up _ I'll need help with a patch |
|
Looking at mp.synchronize, the whole module really does depend on a Once that is identified, then it should be possible to code either a C |
|
For 2.6/3.0 it would probably be best to just disable the module |
|
Jesse, how much (if any) of the rest of the package will work without (Unfortunately, our OpenBSD and FreeBSD buildbots are so unreliable that |
|
I've done some more digging into this for the FreeBSD case. FreeBSD 6.3 and 7.0 both have sem_open, and the man pages suggest it If I change HAVE_SEM_OPEN to 1 in setup.py, the _multiprocessing module On 7.0 amd64, the top of the backtrace scrolls off screen so I can't get To try and remove threads from the equation, due to FreeBSD 6.3 having gcc -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 It appears that some support is there for a single threaded build, but If its not to be supported on single threaded builds (which would be a I'm still trying to understand the core dump in the multithreaded build
Any suggests on how/where to dig further on this? |
|
Oops - meant to add that the actual reported cause of the core dump is Also, the OpenBSD man pages make clear that shared semaphores aren't |
|
I've been thinking about this - Right now, having a working mp.synchronize My recommendation (I'm going to work on a patch to do this) is to not |
|
Well, even if 2.6 slipped (which it is looking probably won't happen), |
|
Agreed - Jesse, can you work up a patch that generates a clean import Improving the *BSD and single-threaded build compatibility of the This may also be worth mentioning in the release notes - Barry's call on |
|
Hows this error look: >>> import multiprocessing
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jesse/open_source/subversion/python-
trunk/Lib/multiprocessing/__init__.py", line 178, in <module>
" function, see issue 3770.")
multiprocessing.SemaphoreImportError: This platform lacks a functioning
sem_open implementation and thereforce, the required synchronization
primitives needed will not function, see issue 3770.
>>> |
|
looks good to me |
On Mon, Sep 29, 2008 at 11:20 AM, Jesse Noller <report@bugs.python.org> wrote:
>
> Jesse Noller <jnoller@gmail.com> added the comment:
>
> Hows this error look:
>
>>>> import multiprocessing
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/Users/jesse/open_source/subversion/python-
> trunk/Lib/multiprocessing/__init__.py", line 178, in <module>
> " function, see issue 3770.")
> multiprocessing.SemaphoreImportError: This platform lacks a functioning
> sem_open implementation and thereforce, the required synchronization
> primitives needed will not function, see issue 3770.Is "thereforce" an actual word? Otherwise it looks fine to me. |
Yeah, I caught that. Rather than disable the entire package, which |
|
Here's a patch, works on my machine. Please review it and make sure it |
|
Could use confirmation from Damien and Andrew that they now see the |
|
I can confirm that the patch works on OpenBSD -current. Only one nit: Does this line in Lib/test/test_multiprocessing.py need to be there? +#import multiprocessing.SemaphoreImportError |
|
On Sep 29, 2008, at 6:36 PM, Damien Miller <report@bugs.python.org>
Oops - my bad. I'll remove it and check it tonight |
|
checked in r66688, lowering from rb to crit to address post 2.6 final |
|
The checked in change has the planned effect on FreeBSD 6.3 i386. I |
|
POSIX semaphores should be fixed in 8-CURRENT, pending MFC. There are rtld + malloc issues in FreeBSD. Python multiprocessing's In the meantime, you may wish to try building Python 2.6 on FreeBSD http://people.freebsd.org/~bms/dump/python26-fbsd-pth.patch thanks! |
|
Sorry, I hit the wrong bug |
|
Closing; we've removed hard-coded platform variables for a better autoconf |
|
I'm trying to follow the history of this issue, as it does not seem fully resolved to me. While trying to package something for Debian, one of the buildd 0 logs for hurd-i386 points to this issue as the cause of a failure 1. This occurs with Python 3.7, though Debian's Python 3.7 may involve some distro patches on top of upstream source. If the build log is inaccessible or incomprehensible, the relevant sections are: ...
The following additional packages will be installed:
... libpython3-stdlib libpython3.7-minimal libpython3.7-stdlib ... python3 python3-minimal python3.7 python3.7-minimal
...
/usr/bin/ctest --force-new-ctest-process -j1
Test project /<<PKGBUILDDIR>>/obj-i686-gnu
Start 1: tests
1/1 Test #1: tests ............................***Failed 0.10 sec
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 28, in <module>
from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/usr/lib/python3.7/lib-dynload/_multiprocessing.cpython-37m-i386-gnu.so)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/tests/run-tests.py", line 46, in <module>
print_lock = multiprocessing.Lock()
File "/usr/lib/python3.7/multiprocessing/context.py", line 66, in Lock
from .synchronize import Lock
File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 32, in <module>
" synchronization primitives needed will not" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.Do you know what is happening here? Maybe the compounded ImportError I'm seeing *was* the resolution referred to in prior comments, but if so it seems strange to reference this issue in the error message. The hurd-i386 platform is not a priority for me, so if the answer is "multiprocessing doesn't work there" I am perfectly OK with this. Also I realise I am asking you to recall >10 year old memories so I apologise in advance for any re-opened wounds. If you need to see the source for what buildd is actually testing here, it's the debian/v2020.11.01-1 tag of 2. |
|
Matthew, this tracker issue has been closed for over 10 years; posting new questions or comments to such is unlikely to get a response (I just happened to see your comment in passing). If you believe you have identified a new issue with current versions of Python, you should search for newer issues and, if appropriate, open a new one with a reference to this one. However, before doing that, I would do a web search on "cannot import name 'SemLock'"; you should find a number of references to situations that seem similar to yours with the cause being the lack of a working sem_open implementation as noted in the ImportError message, a condition that might just be a system configuration issue on the build system. Good luck! |
…r multiprocessing support (GH-126531) Docs: Remove the logging howto potential promise of multiprocessing support in the future. Stick to the facts and suggestions, don't provide hope where we're not going to implement complexity that we'd rather the user implement themselves when needed.
…Handler multiprocessing support (pythonGH-126531) Docs: Remove the logging howto potential promise of multiprocessing support in the future. Stick to the facts and suggestions, don't provide hope where we're not going to implement complexity that we'd rather the user implement themselves when needed. (cherry picked from commit 1fe67df) Co-authored-by: Gregory P. Smith <greg@krypto.org>
…eHandler multiprocessing support (GH-126531) (#126540) gh-48020: [docs] Remove the logging howto suggested future FileHandler multiprocessing support (GH-126531) Docs: Remove the logging howto potential promise of multiprocessing support in the future. Stick to the facts and suggestions, don't provide hope where we're not going to implement complexity that we'd rather the user implement themselves when needed. (cherry picked from commit 1fe67df) Co-authored-by: Gregory P. Smith <greg@krypto.org>
…Handler multiprocessing support (pythonGH-126531) Docs: Remove the logging howto potential promise of multiprocessing support in the future. Stick to the facts and suggestions, don't provide hope where we're not going to implement complexity that we'd rather the user implement themselves when needed.
…Handler multiprocessing support (pythonGH-126531) Docs: Remove the logging howto potential promise of multiprocessing support in the future. Stick to the facts and suggestions, don't provide hope where we're not going to implement complexity that we'd rather the user implement themselves when needed.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs
The text was updated successfully, but these errors were encountered: