Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setuptools: Problems when including ruamel.yaml #778

Closed
el-dee opened this issue Nov 10, 2019 · 3 comments
Closed

setuptools: Problems when including ruamel.yaml #778

el-dee opened this issue Nov 10, 2019 · 3 comments
Assignees
Labels
Milestone

Comments

@el-dee
Copy link
Contributor

@el-dee el-dee commented Nov 10, 2019

Probably due to the strange name of the package, including ruamel.yaml needs extra steps :

to include it, one needs the following lines :

    'include_modules': {'*': ['_ruamel_yaml', 'ruamel']},
    'package_data_dirs': {'ruamel': [('ruamel/*', 'ruamel', {})]},

_ruamel_yaml is needed to retrieve the wheel and ruamel to trigger the package_data_dirs entry. Package_data_dirs entry is needed to copy the .py files from the wheel.

Using 'ruamel.*.*' causes setuptools to crash :

Building runtime for platform: manylinux1_x86_64
Traceback (most recent call last):
  File "setup.py", line 91, in 
    setup(**config)
  File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Developer/Panda3D/direct/dist/commands.py", line 382, in run
    self.build_runtimes(platform, True)
  File "/Developer/Panda3D/direct/dist/commands.py", line 711, in build_runtimes
    create_runtime(appname, scriptname, False)
  File "/Developer/Panda3D/direct/dist/commands.py", line 641, in create_runtime
    freezer.addModule(incmod)
  File "/Developer/Panda3D/direct/dist/FreezeTool.py", line 1042, in addModule
    guess, fromSource, text))
  File "/Developer/Panda3D/direct/dist/FreezeTool.py", line 996, in _gatherSubmodules
    modules = self.getModuleStar(parentName)
  File "/Developer/Panda3D/direct/dist/FreezeTool.py", line 945, in getModuleStar
    file, pathname, description = imp.find_module(baseName, path)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/imp.py", line 270, in find_module
    "not {}".format(type(path)))
RuntimeError: 'path' must be None or a list, not < class '_frozen_importlib_external._NamespacePath'>

See https://discourse.panda3d.org/t/setuptools-how-to-properly-include-ruamel-yaml/25298

@rdb rdb added this to the 1.10.5 milestone Nov 10, 2019
@Moguri Moguri added the deployment label Nov 12, 2019
@Moguri Moguri added the bug label Dec 3, 2019
@rdb

This comment has been minimized.

Copy link
Member

@rdb rdb commented Dec 11, 2019

It looks like ruamel's top-level package doesn't have an __init__.py, but it does ship with this bizarre -nspkg.pth file (expanded for readability):

import sys, types, os
has_mfs = sys.version_info > (3, 5)
p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('ruamel',))
importlib = has_mfs and __import__('importlib.util')
has_mfs and __import__('importlib.machinery')
m = has_mfs and sys.modules.setdefault('ruamel', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('ruamel', [os.path.dirname(p)])))
m = m or sys.modules.setdefault('ruamel', types.ModuleType('ruamel'))
mp = (m or []) and m.__dict__.setdefault('__path__',[])
(p not in mp) and mp.append(p)

It looks like this might be part of an implementation of namespace packages, which may deserve further attention in deploy-ng.

I don't really understand your fix since the ruamel wheel doesn't have any data files. Including the .py files as data files is not satisfactory since the intent of deploy-ng is to compile the Python into bytecode and embed it into the executable.

@el-dee

This comment has been minimized.

Copy link
Contributor Author

@el-dee el-dee commented Dec 11, 2019

I don't really understand your fix since the ruamel wheel doesn't have any data files. Including the .py files as data files is not satisfactory since the intent of deploy-ng is to compile the Python into bytecode and embed it into the executable.

It's not a fix, it's a crude workaround to get ruamel.yaml working in a distributed app. I didn't spend days, but I could not find a way to have the .py files from ruamel.yaml detected and byte coded by deploy-ng.

@rdb rdb self-assigned this Dec 11, 2019
@rdb

This comment has been minimized.

Copy link
Member

@rdb rdb commented Dec 11, 2019

OK, got it. Copy-pasting the ruamel source into your project, and adding an __init__.py into the ruamel folder, might also be an effective workaround.

Apparently, since PEP 420, any folder can act as a namespace package. For example, this works:

$ mkdir thingy
$ python -c 'import thingy; print(thingy)'
<module 'thingy' (namespace)>

The modulefinder that comes with the standard Python library cannot even handle this; we'll have to add custom handling for this in FreezeTool.py. I'll take care of this for 1.10.5.

@rdb rdb closed this in c3052f3 Dec 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.