Standardize runner scripts #1

Merged
merged 4 commits into from Nov 17, 2015

Conversation

Projects
None yet
2 participants
Contributor

didrocks commented Nov 17, 2015

When running dep11-generator script in our infra from git in a virtualenv, we would get some stacktrace (see below).

The issue is that __spec__ (new in python 3.4) needs to be define when using a pool of process with the multiprocessing module, if initialized with set_start_method('forkserver').

When using the setuptools generated script (in the script section), those are using __import__() to load the executable scripts, which doesn't set those properties. The bug seems thus to be in that tool.
However, a more correct/modern way is to use the module entrypoints() for setup.py. This one generate in a similar way the executable script, but does something way more standard like what we have in import <module_name>:

  • all code is now in the module itself, enabling everything to be pre-compiled in most distribution.
  • it doesn't use and abuse __import__ to load modules

I thus added this refactoring and moving the code to the dep11 module themselves main() method. I did rewrite the scripts themselves that I guess you are using for testing, but they are not installed/shipped anymore by setuptools. That way, we use a similar way to load and start our local scripts for testing than the one installed on the system.


The stacktrace was:

Traceback (most recent call last):
  File "/home/ubuntu/appstream/bin/dep11-generator", line 4, in <module>
    __import__('pkg_resources').run_script('dep11==0.2', 'dep11-generator')
  File "/home/ubuntu/appstream/lib/python3.4/site-packages/pkg_resources/__init__.py", line 735, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/ubuntu/appstream/lib/python3.4/site-packages/pkg_resources/__init__.py", line 1652, in run_script
    exec(code, namespace, namespace)
  File "/home/ubuntu/appstream/lib/python3.4/site-packages/dep11-0.2-py3.4.egg/EGG-INFO/scripts/dep11-generator", line 806, in <module>
    main()
  File "/home/ubuntu/appstream/lib/python3.4/site-packages/dep11-0.2-py3.4.egg/EGG-INFO/scripts/dep11-generator", line 763, in main
    gen.process_suite(args[2])
  File "/home/ubuntu/appstream/lib/python3.4/site-packages/dep11-0.2-py3.4.egg/EGG-INFO/scripts/dep11-generator", line 254, in process_suite
    with mp.Pool(maxtasksperchild=16) as pool:
  File "/usr/lib/python3.4/hmultiprocessing/context.py", line 118, in Pool
    context=self.get_context())
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 168, in __init__
    self._repopulate_pool()
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 233, in _repopulate_pool
    w.start()
  File "/usr/lib/python3.4/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.4/multiprocessing/context.py", line 281, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.4/multiprocessing/popen_forkserver.py", line 36, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.4/multiprocessing/popen_fork.py", line 21, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.4/multiprocessing/popen_forkserver.py", line 43, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/usr/lib/python3.4/multiprocessing/spawn.py", line 173, in get_preparation_data
    main_mod_name = getattr(main_module.__spec__, "name", None)
AttributeError: 'module' object has no attribute '__spec__'

didrocks added some commits Nov 17, 2015

Move dep11-generator and dep11-validate as importable scripts
That will enable us to use entrypoints setuptools functionality to generate
the script rather than easyinstall ones.
Remove thus __main__ and ensure all initialization is done in main()
Readd local runner scripts
Those scripts are only for local debugging, they won't be installed by
setuptools.
Use entry points instead of scripts
Entry points is more standard in the way to ship init binaries:
- all code is now in the module itself, enabling everything to be pre-compiled
  in most distribution.
- it doesn't use and abuse __import__ to load modules, which, doesn't set
  __spec__ (new in python 3.4). __spec__ is necessary in multiprocessing module
  pool initialization when you multiprocessing.set_start_method('forkserver')
  as in dep11.

That way, we use a similar way to load and start our local scripts for testing
than the one installed on the system.
Owner

ximion commented Nov 17, 2015

Looks very good, thank you for the oatch and the debugging work! :-)

ximion added a commit that referenced this pull request Nov 17, 2015

@ximion ximion merged commit 5bb308d into ximion:master Nov 17, 2015

@didrocks didrocks deleted the didrocks:setuptools_scripts branch Nov 17, 2015

iainlane pushed a commit to iainlane/appstream-dep11 that referenced this pull request Feb 19, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment