(bisected) ImportError on meson configure

[Akaricchi]

Describe the bug

Traceback (most recent call last):
  File "/home/akari/.local/bin/meson", line 33, in <module>
    sys.exit(load_entry_point('meson==0.63.99', 'console_scripts', 'meson')())
  File "/home/akari/.local/lib/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/mesonmain.py", line 280, in main
    return run(sys.argv[1:], launcher)
  File "/home/akari/.local/lib/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/mesonmain.py", line 271, in run
    return CommandLineParser().run(args)
  File "/home/akari/.local/lib/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/mesonmain.py", line 72,in __init__
    from . import mconf, mdist, minit, minstall, mintro, msetup, mtest, rewriter, msubprojects, munstable_coredata,mcompile, mdevenv
  File "/home/akari/.local/lib/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/mconf.py", line 22, in <module>
    from . import build
  File "/home/akari/.local/lib/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/build.py", line 33, in <module>
    from .mesonlib import (
ImportError: cannot import name 'EnvironmentVariables' from 'mesonbuild.mesonlib' (/home/akari/.local/lib/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/mesonlib/__init__.py)

To Reproduce
Run meson configure in any directory.

Expected behavior
No crash.

system parameters

• meson meson-0.63.99 (bug introduced in commit 2dfd952)

CC @xclaesse

[xclaesse]

Try reconfiguring with --wipe. When you track meson master we don't guarantee an existing build dir keeps working

[Akaricchi]

It fails when running meson configure in any context, even outside of any build directory at all.

[xclaesse]

I think you have a mismatch of meson versions installed. Try clearing your .local and reinstall meson.

[Akaricchi]

@xclaesse I've just tried it in a clean venv, and unfortunately it still doesn't work. It has no access to system or user site-packages. It still produces the same error.

Traceback (most recent call last):
  File "/tmp/test/bin/meson", line 33, in <module>
    sys.exit(load_entry_point('meson==0.63.99', 'console_scripts', 'meson')())
  File "/tmp/test/lib64/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/mesonmain.py", line 280, in main
    return run(sys.argv[1:], launcher)
  File "/tmp/test/lib64/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/mesonmain.py", line 271, in run
    return CommandLineParser().run(args)
  File "/tmp/test/lib64/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/mesonmain.py", line 72, in __init__
    from . import mconf, mdist, minit, minstall, mintro, msetup, mtest, rewriter, msubprojects, munstable_coredata, mcompile, mdevenv
  File "/tmp/test/lib64/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/mconf.py", line 22, in <module>
    from . import build
  File "/tmp/test/lib64/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/build.py", line 33, in <module>
    from .mesonlib import (
ImportError: cannot import name 'EnvironmentVariables' from 'mesonbuild.mesonlib' (/tmp/test/lib64/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/mesonlib/__init__.py)

I can reproduce it in the interpreter within the venv as well:

Python 3.10.7 (main, Sep 11 2022, 08:41:56) [GCC]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import mesonbuild

In [2]: mesonbuild
Out[2]: <module 'mesonbuild' from '/tmp/test/lib64/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/__init__.py'>

In [3]: from mesonbuild import build
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In [3], line 1
----> 1 from mesonbuild import build

File /tmp/test/lib64/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/build.py:33
     31 from . import mlog
     32 from . import programs
---> 33 from .mesonlib import (
     34     HoldableObject, SecondLevelHolder,
     35     File, MesonException, MachineChoice, PerMachine, OrderedSet, listify,
     36     extract_as_list, typeslistify, stringlistify, classify_unity_sources,
     37     get_filenames_templates_dict, substitute_values, has_path_sep,
     38     OptionKey, PerMachineDefaultable, OptionOverrideProxy,
     39     MesonBugException, EnvironmentVariables
     40 )
     41 from .compilers import (
     42     is_object, clink_langs, sort_clink, all_languages,
     43     is_known_suffix, detect_static_linker
     44 )
     45 from .interpreterbase import FeatureNew, FeatureDeprecated

ImportError: cannot import name 'EnvironmentVariables' from 'mesonbuild.mesonlib' (/tmp/test/lib64/python3.10/site-packages/meson-0.63.99-py3.10.egg/mesonbuild/mesonlib/__init__.py)

In [4]: import sys

In [5]: sys.path
Out[5]:
['/tmp/test/bin',
 '/usr/lib64/python310.zip',
 '/usr/lib64/python3.10',
 '/usr/lib64/python3.10/lib-dynload',
 '',
 '/tmp/test/lib64/python3.10/site-packages',
 '/tmp/test/lib64/python3.10/site-packages/meson-0.63.99-py3.10.egg',
 '/tmp/test/lib/python3.10/site-packages',
 '/tmp/test/lib/python3.10/site-packages/meson-0.63.99-py3.10.egg']

Meson was installed with python3 ./setup.py install -O2, though -O2 does not make a difference here.

[eli-schwartz]

Try reconfiguring with --wipe. When you track meson master we don't guarantee an existing build dir keeps working

The traceback comes from mesonmain.py, in from . import mconf -- in turn, mconf.py does from . import build, and build.py does from .mesonlib import ( which fails because mesonlib/__init__.py does not contain EnvironmentVariables

Now, the interesting thing about this claim is that it is about meson configure, when literally any meson command will import the same files (other than --internal) during the startup mesonmain command line parser. In other words, the error report is that Meson is so fundamentally, utterly broken, that attempting to use it in any way shape or form, even just invoking it with the --help option, across any subcommand without exception, doesn't work. We should have full 100% CI failures across every individual test case, and likely also when trying to execute the unittest harness too, and attempting to test that commit before submitting a PR should be so broken that it's mind boggling that it ever got git commited.

This seems bogus, of course. There must be a better explanation.

The other problem is that:

• the traceback was bisected to commit 2dfd952
• the traceback is because mesonbuild/mesonlib/__init__.py doesn't provide a particular import
• that __init__.py was explicitly removed in the commit that, supposedly, this bug was traced back to

Conclusion: this is a very real bug... but it's not a Meson bug.

I think you have a mismatch of meson versions installed. Try clearing your .local and reinstall meson.

I don't think this is a mismatch of Meson versions installed, something that typically refers to having one version in /usr/lib/python3.X and another version in ~/.local/lib/python3.X.

I think this is a deeply, deeply, deeply broken setuptools invocation. Using python setup.py anything uses a build directory in which the source files are copied, preserved, and kept around in perpetuity, even when the original is deleted. Then you created an egg install, which is basically zipping up the build directory, including stale old files that were deleted from the project. A couple of those files were bad news.

No matter how many times you uninstalled Meson, or used a venv, the bug was in your source tree (but not in meson's git repository). The general lesson to take from here is that python setup.py clean -a or git clean -id is basically mandatory. Some old versions of pip "solved" this by making a copy of the entire project, including .git/, in /tmp and building from there, so you never have build artifacts in the source directory... but that was removed because it was really slow 🤣 and now pip install . shares all the problems of python setup.py install.

Fortunately, building from an sdist downloaded from PyPI works flawlessly. Also flawless: low-level tooling on top of setuptools, such as distro build scripts.

The general solution to this is eli-schwartz@6a24f13 -- Meson will be growing the ability to install python pyproject.toml metadata, create wheels, etc. -- and it will dogfood this in order to be fully self-sufficient for building and installing Meson without relying on external dependencies such as setuptools. Conveniently, this also means that we can rely on Meson's actually accurate knowledge of what needs to be installed.

[Akaricchi]

@eli-schwartz your assessment seems correct, running setup.py clean -a and reinstalling helped indeed. It never crossed my mind to suspect a contaminated build directory…

Now, the interesting thing about this claim is that it is about meson configure, when literally any meson command will import the same files (other than --internal) during the startup mesonmain command line parser.

Every other command was affected as well, I'm honestly not sure what led me to believe it was just meson configure.

[eli-schwartz]

It's an obscure setuptools edge case. Well, I say obscure because how often do people change a foo/*.py module to foo.py? In nearly any other case, you just end up with additional junk files that are never actually used, so it doesn't matter how broken those unused files are.

Still, I anticipate using ./meson.py setup builddir to install Meson itself, it really will be so much better.

[xclaesse]

@eli-schwartz thanks for the explanation, I did not know all that!