openSUSE packages failing when built with Nuitka

[jayvdb]

Project https://build.opensuse.org/project/show/home:jayvdb:Nuitka - Python 3 only

The collection of packages contains the initial set of dependencies needed to bootstrap a standard Python installation, with all build dependencies and necessary testing tools needed to verify the built packages. Still quite a few unresolved packages, preventing building of larger frameworks like Django.

Nuitka build failures:

• testtools assert locals_dict_name not in locals_dict_handles, locals_dict_name : globals_testtools$_compat2x : "Cannot recurse to import module 'testtools/_compat2x.py' (..) because of 'SyntaxError'" (Fixed by deleting testtools/_compat2x.py on Python 3)
• Js2Py assert locals_dict_name not in locals_dict_handles, locals_dict_name: globals_js2py$internals$speed : "Cannot recurse to import module 'js2py/internals/speed.py' (...) because of 'SyntaxError'"; bypassing that results in OOM in clang and gcc
• PasteDeploy assert locals_dict_name not in locals_dict_handles, locals_dict_name : globals_paste$deploy$paster_templates (_Fixed with mv paste/deploy/paster_templates.py paste/deploy/paster_templates/__init__.py)
• Pygments assert locals_dict_name not in locals_dict_handles, locals_dict_name : globals_tests$support (Fixed by removing tests/)
• Twisted Twisted build_with_Nuitka fails to find twisted.news #531
• datrie & numpy & greenlet Missing both compile_packages and py_modules, aborting...
• pyzmq gcc: /tmp/timer_createamusva27.c:2: undefined reference to timer_create'`
• PyInstaller Error, 'hook-PIL.Image' is not a proper python module name.

Missing artifacts (#530 is a workaround for someone of these):

• Jinja2 - Jinja2.tests not available https://github.com/Nuitka/Nuitka/issues/525
• PySocks - sockshandler discarded
• Cython - cython -> Cython (maybe ok?)
• peewee - packages peewee and pwiz discarded
• cryptography - ModuleNotFoundError: No module named 'cryptography.hazmat.bindings._constant_time'
• pytest-runner ptr.so created; pytest_runner.so omitted

Seg faults

• Werkzeug only occurs in test_shared_data_middleware
• hypothesis - only occurs with clang, not gcc
• Pympler only occurs in test test_stracker_create_summary (test_sys_mods fails for less concerning reasons)

As I go through getting those packages to build, any which build but cause other failures I will disable "Use for Build Flag" e.g. https://build.opensuse.org/repositories/home:jayvdb:Nuitka/python-setuptools . I will attempt to adjust them so the tests fail.

Builds but breaks other packages:

• appdirs - see path.py
• setuptools
• py
• pytest
• pbr
• pytz
• virtualenv
• alabaster
• execnet - inspect.getsource(source) fails, breaking pytest
• testscenarios
• kombu
• pip
• eventlet, especially eventlet.patcher.SysModulesSaver
• zope.*, sphinxcontrib.*, paste.* zope components built with Nuitka are all called zope.so #524
• Werkzeug breaks Flask AttributeError: module 'werkzeug' has no attribute 'test'

Test failures:

• psutil, 10 failures
• xmlschema test_element_tree_import_script fails
• path.py TestSpecialPaths case fails (5 tests)
• Paste test_egg_parser fails
• bs4 test_out_of_range_entity fails
• pytest-shutil test_run_in_subprocess_passes_stderr fails
• Jinja2 - test_safe_format_safety and test_safe_format_all_okay fail
• kombu test_import_no_ssl
• ZConfig test_existing_file & test_existing_path
• Werkzeug test_default_stream_factory[True-???] (two tests) test_shared_data_middleware (seg fault)
• cliff lots of failures, but may be only a few common problems, and maybe VM setup related
• scons - test/fixture/wrapper.py & test/fixture/wrapper_with_args.py
• pytest-cov tests fail after test_funcarg_not_active PASSED [ 52%]
• mock - autospec functionality: TypeError: Nuitka doesn't support __defaults__ size changes - c.f. graphviz tests/conftest.py which uses this.

Convertion to setuptools pending:

• PyYAML TypeError: dist must be a Distribution instance
• gpgme error: each element of 'ext_modules' option must be an Extension instance or 2-tuple - workaround was create alternative python-gpg which is only the Python component, but also without the tests.

[jayvdb]

I've mostly figured out cryptography, and cryptography_vectors.

I needed to copy all of the subdirectories of cryptography_vectors to the top level of site-packages (hopefully not too many conflicts with other packages) -- a patch should be able to fix this easily, or a Nuitka plugin maybe. I guess there must be an existing issue about missing data files. #521 is an umbrella issue I created about that.

Then for cryptography, I first tried combinations of force-inclusions, like --include-plugin-directory cryptography.hazmat.bindings --include-module cryptography.hazmat.bindings._padding --include-module cryptography.hazmat.bindings._constant_time --include-module cryptography.hazmat.bindings._openssl --include-module cryptography.hazmat.bindings._padding.abi3 --include-module cryptography.hazmat.bindings._constant_time.abi3 --include-module cryptography.hazmat.bindings._openssl.abi3 but that didnt work, and there wasnt any warnings about those either, which feels like it should emit something.

I also tried doing build_ext first, and then rename cryptography/hazmat/bindings/_constant_time.abi3.so -> cryptography/hazmat/bindings/_constant_time.so in case the PEP 3149 names were causing the problem. Nuitka still didnt pick up the extra .so

Then I did the following before performing the build, and success.

echo 'from cryptography.hazmat.bindings import _padding' >>  src/cryptography/hazmat/bindings/__init__.py
echo 'from cryptography.hazmat.bindings import _constant_time' >>  src/cryptography/hazmat/bindings/__init__.py
echo 'from cryptography.hazmat.bindings import _openssl' >>  src/cryptography/hazmat/bindings/__init__.py

[kayhayen]

I don't think this is being worked on, is it? I would like to see this closed therefore. Let me know if I am mistaken.