New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Update multiprocessing plugin for Python 3.4 #57

Closed
wants to merge 10 commits into
base: develop
from

Conversation

Projects
None yet
2 participants
@nmoehrle

nmoehrle commented Feb 19, 2018

The multiprocessing module was subject to a larger refactoring in Python 3.4 (python/cpython@84ed9a6) -- the multiprocessing.forking module was split up into multiple modules reduction, spawn, popen_fork*, popen_spawn_* which broke the multiprocessing plugin. This PR is an attempt to fix this but is work in progress and should serve as an discussion platform.

kayhayen and others added some commits Jan 1, 2018

Restore runners for use from git checkout and source releases.
* This allows zero setup usage of Nuitka without PYTHONPATH being
  changed at all.
Cleanup, the "LIBDIR" trick is now obsolete.
* We used to have runners patched at install time with the correct
  path to add, but that was moved into the "__main__" here for no
  good reason.
Add back runners for tests and checkouts
* The "-m nuitka" needs PYTHONPATH tricks to find the proper variant to
  use, the runner is supposed to know, so lets use it there.

* This restores the out of the box ability to take a source snapshot and
  run it.
Fix, for modules the "__package__" should be dynamic.
* Only at load time, the value is decided, so do not statically optimize
  for "__package__" as it is subject to change in those cases.

* Also, do not assign module attributes in the node tree, merely have
  special references for them, generated in case of no other assignment
  being seen. This affects "__loader__" and "__spec__" mostly

* The new nodes were moved to a new nodes module, along with the node
  that handles "__file__" already.
@kayhayen

This comment has been minimized.

Collaborator

kayhayen commented Apr 25, 2018

Hello,

are you still intend on working on this? This would be really good to see finished.

Yours,
Kay

@kayhayen

This comment has been minimized.

Collaborator

kayhayen commented Apr 25, 2018

I cherry-picked your commits on to factory, and it compiled a multiprocessing test program fine, do you think it's still WIP, and why?

@kayhayen

This comment has been minimized.

Collaborator

kayhayen commented Apr 26, 2018

Seems that multiprocessing does work on Linux and alike, but not on Windows and at least Python3, even before 3.4, as 3.3 also failed for me.

Are you intent on working for Windows still? Otherwise this seems good and I will merge at least this part, rebasing your branch ought to be trivial. Give a sign of life please. :-)

@kayhayen

This comment has been minimized.

Collaborator

kayhayen commented Apr 26, 2018

Ok, stupid me, I forgot that in fact the plugin is only even used on Windows, so nothing new is really working, although I think Python 3.4 on Linux might have started to work now, but that can't be real.

@kayhayen

This comment has been minimized.

Collaborator

kayhayen commented Apr 26, 2018

So I figured out the missing bits. The multiprocessing plugin so far only worked if you also provided --standalone or --recurse-to=multiprocessing, so it could monkey patch things while it loads them. That wasn't the case with my test program. And it shouldn't have to be. So now it works out of the box, if you enable the plugin, and doesn't hang anymore.

I think we can close this. Let me know if I am wrong.

@kayhayen kayhayen closed this Apr 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment