OSX + homebrew python@3.9: pybuilder/_vendor/virtualenv.py change_prefix does not consider symlinks, raises invalid AssertionError #771

hoxu · 2021-04-28T07:41:46Z

I have installed Python 3 on macOS 11.3 (Big Sur) with brew install python@3.9. Homebrew installs Python 3.9 under /usr/local/Cellar/python@3.9, and symlinks /usr/local/opt/python@3.9 -> ../Cellar/python@3.9/3.9.4.

python3 is a symlink /usr/local/bin/python3 -> ../Cellar/python@3.9/3.9.4/bin/python3.
pip3 is a symlink /usr/local/bin/pip3 -> ../Cellar/python@3.9/3.9.4/bin/pip3.

$ head -1 $(which pip3)
#!/usr/local/opt/python@3.9/bin/python3.9

I used pip3 install --user pybuilder to install pybuilder:

$ which pyb
/Users/username/Library/Python/3.9/bin/pyb
$ head -1 $(which pyb)
#!/usr/local/opt/python@3.9/bin/python3.9

$ pyb
PyBuilder version 0.12.10
Build started at 2021-04-28 10:36:15
------------------------------------------------------------
------------------------------------------------------------
BUILD FAILED - AssertionError: Filename /usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py does not start with any of these prefixes: ['/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/Extras/lib/python', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/Users/username/Library/Python/3.9/lib/python/site-packages', '/Users/username/.local/lib/python/3.9/site-packages', '/Users/username/Library/Python/3.9/site-packages', '/Library/Python/3.9/site-packages'] (pybuilder/_vendor/virtualenv.py:1272)
------------------------------------------------------------
Build finished at 2021-04-28 10:36:15
Build took 0 seconds (64 ms)

AssertionError is raised from change_prefix:

pybuilder/src/main/python/pybuilder/_vendor/virtualenv.py

Line 1272 in 6ac1ef4

    
           raise AssertionError("Filename {} does not start with any of these prefixes: {}".format(filename, prefixes))

The text was updated successfully, but these errors were encountered:

dindurthy · 2021-05-03T22:05:53Z

Are there any workarounds? We're completely blocked.

arcivanov · 2021-05-04T01:11:07Z

Hello, unfortunately I'm completely swamped right now with work.

We don't test with Brew Python, but we do test with PyEnv Python on Mac and it works: https://travis-ci.com/github/pybuilder/pybuilder/jobs/502734233

I would highly recommend using PyEnv with Brew regardless as Brew Python contains the following warning:

Important: Python may be upgraded to a newer version at any time. Consider using a version manager such as pyenv if you require stability of minor or patch versions for virtual environments.

https://docs.brew.sh/Homebrew-and-Python

arcivanov · 2021-05-04T01:26:12Z

I see why this is happening and it's brew-specific, unfortunately.

arcivanov · 2021-05-04T01:26:54Z

@hoxu could you please run the build with pyb -vX so that the full exception will be printed out?

hoxu · 2021-05-04T05:47:31Z

Are there any workarounds? We're completely blocked.

@dindurthy you could use OSX-shipped python3 for pybuilder execution by prepending /usr/bin to PATH. Here is an example with which:

$ which -a python3
/usr/local/bin/python3
/usr/bin/python3
$ python3 --version
Python 3.9.4
$ PATH=/usr/bin:$PATH which -a python3
/usr/bin/python3
/usr/local/bin/python3
/usr/bin/python3
$ PATH=/usr/bin:$PATH python3 --version
Python 3.8.2

hoxu · 2021-05-04T05:52:42Z

@arcivanov:

+ pyb -vX
[DEBUG] Registered Python environment 'system': CPython version 3.9.4.final.0 on darwin in ['/Users/username/XXX/vebuild/bin/python3.9']
PyBuilder version 0.12.10
Build started at 2021-05-04 08:48:51
------------------------------------------------------------
[DEBUG] Loading project module from ./build.py
[DEBUG] Setting up plugins VEnv at '/Users/username/XXX/.pybuilder/plugins/cpython-3.9.4.final.0'
Traceback (most recent call last):
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/cli.py", line 461, in main
    reactor.prepare_build(property_overrides=options.property_overrides,
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/reactor.py", line 203, in prepare_build
    self._setup_plugin_directory(reset_plugins, no_venvs)
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/reactor.py", line 517, in _setup_plugin_directory
    plugin_env = per["pybuilder"] = PythonEnv(plugin_dir, self).create_venv(with_pip=True,
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/python_env.py", line 209, in create_venv
    create_venv(self._env_dir,
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/python_env.py", line 429, in create_venv
    virtualenv.create_environment(
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/_vendor/virtualenv.py", line 1161, in create_environment
    install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages=site_packages, clear=clear, symlink=symlink)
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/_vendor/virtualenv.py", line 1450, in install_python
    copy_required_modules(home_dir, symlink)
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/_vendor/virtualenv.py", line 1340, in copy_required_modules
    dst_filename = change_prefix(filename, dst_prefix)
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/_vendor/virtualenv.py", line 1272, in change_prefix
    raise AssertionError("Filename {} does not start with any of these prefixes: {}".format(filename, prefixes))
AssertionError: Filename /usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py does not start with any of these prefixes: ['/Users/username/XXX/vebuild/Extras/lib/python', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/Users/username/Library/Python/3.9/lib/python/site-packages', '/Users/username/XXX/vebuild', '/Users/username/.local/lib/python/3.9/site-packages', '/Users/username/Library/Python/3.9/site-packages', '/Library/Python/3.9/site-packages']
------------------------------------------------------------
BUILD FAILED - AssertionError: Filename /usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py does not start with any of these prefixes: ['/Users/username/XXX/vebuild/Extras/lib/python', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/Users/username/Library/Python/3.9/lib/python/site-packages', '/Users/username/XXX/vebuild', '/Users/username/.local/lib/python/3.9/site-packages', '/Users/username/Library/Python/3.9/site-packages', '/Library/Python/3.9/site-packages'] (pybuilder/_vendor/virtualenv.py:1272)
------------------------------------------------------------
Build finished at 2021-05-04 08:48:51
Build took 0 seconds (64 ms)

dindurthy · 2021-05-04T20:06:57Z

Changing PATH won't work without breaking too many other things, unfortunately. I think the only workaround is to not use brew python.

hoxu · 2021-08-25T08:38:21Z

So change_prefix function is comparing /usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py to sys.prefix, that points to symlink-containing /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9 on Homebrew python.

@arcivanov: wouldn't this be fixed by simply changing:

 def change_prefix(filename, dst_prefix):
-    prefixes = [sys.prefix]
+    prefixes = [os.path.realpath(sys.prefix)]

>>> sys.prefix
'/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9'
>>> os.path.realpath(sys.prefix)
'/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9'

If so, would you accept PR with the above change?

arcivanov · 2021-09-05T04:31:10Z

If you could could you please retest with latest --pre and let me know?

hoxu · 2021-09-06T05:59:09Z

First reproduction with 0.12.10:

$ pyb
PyBuilder version 0.12.10
Build started at 2021-09-06 08:48:40
------------------------------------------------------------
------------------------------------------------------------
BUILD FAILED - AssertionError: Filename /usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py does not start with any of these prefixes: ['/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/Extras/lib/python', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/Users/username/Library/Python/3.9/lib/python/site-packages', '/Users/username/.local/lib/python/3.9/site-packages', '/Users/username/Library/Python/3.9/site-packages', '/Library/Python/3.9/site-packages'] (pybuilder/_vendor/virtualenv.py:1272)
------------------------------------------------------------
Build finished at 2021-09-06 08:48:40
Build took 0 seconds (68 ms)

Then test with --pre:

$ pip3 install --user --pre --upgrade pybuilder
<clip>
Successfully installed pybuilder-0.13.0.dev20210905004524
$ pyb
PyBuilder version 0.13.0.dev20210905004524
Build started at 2021-09-06 08:50:08
------------------------------------------------------------
------------------------------------------------------------
BUILD FAILED - StopIteration:  (virtualenv/run/plugin/discovery.py:22)
------------------------------------------------------------
Build finished at 2021-09-06 08:50:08
Build took 0 seconds (153 ms)

A different error, I call it progress :)

$ pyb -vX
[DEBUG] Registered Python environment 'system': CPython version 3.9.6.final.0 on darwin in ['/usr/local/opt/python@3.9/bin/python3.9']
PyBuilder version 0.13.0.dev20210905004524
Build started at 2021-09-06 08:54:14
------------------------------------------------------------
[DEBUG] Loading project module from ./build.py
[DEBUG] Setting up plugins VEnv at '/Users/username/XXX/.pybuilder/plugins/cpython-3.9.6.final.0'
Traceback (most recent call last):
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/cli.py", line 461, in main
    reactor.prepare_build(property_overrides=options.property_overrides,
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/reactor.py", line 203, in prepare_build
    self._setup_plugin_directory(reset_plugins, no_venvs)
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/reactor.py", line 517, in _setup_plugin_directory
    plugin_env = per["pybuilder"] = PythonEnv(plugin_dir, self).create_venv(with_pip=True,
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/python_env.py", line 214, in create_venv
    create_venv(self._env_dir,
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/python_env.py", line 453, in create_venv
    virtualenv.cli_run(args, setup_logging=False)
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/_vendor/virtualenv/run/__init__.py", line 30, in cli_run
    of_session = session_via_cli(args, options, setup_logging, env)
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/_vendor/virtualenv/run/__init__.py", line 48, in session_via_cli
    parser, elements = build_parser(args, options, setup_logging, env)
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/_vendor/virtualenv/run/__init__.py", line 69, in build_parser
    discover = get_discover(parser, args)
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/_vendor/virtualenv/run/plugin/discovery.py", line 22, in get_discover
    default=next(iter(choices)),
StopIteration
------------------------------------------------------------
BUILD FAILED - StopIteration:  (virtualenv/run/plugin/discovery.py:22)
------------------------------------------------------------
Build finished at 2021-09-06 08:54:14
Build took 0 seconds (129 ms)

Is this a regression?

arcivanov · 2021-09-06T06:36:20Z

Yes and no. 😄 This is apparently a bug in virtualenv, which is bundled in PyB. Let me see if VEnv can fix it on their end.

arcivanov · 2021-09-06T08:41:49Z

@hoxu
Would you kindly please:

pip3 install --user virtualenv into your homebrewed Python environment.
virtualenv -vvv --with-traceback ~/testvenv
Capture the output and post it here? Does it fail?

hoxu · 2021-09-06T08:58:54Z

$ realpath $(which pip3)
/usr/local/Cellar/python@3.9/3.9.7/bin/pip3
$ pip3 install --user virtualenv
Collecting virtualenv
  Downloading virtualenv-20.7.2-py2.py3-none-any.whl (5.3 MB)
     |████████████████████████████████| 5.3 MB 3.6 MB/s
Requirement already satisfied: six<2,>=1.9.0 in /usr/local/lib/python3.9/site-packages (from virtualenv) (1.15.0)
Collecting backports.entry-points-selectable>=1.0.4
  Downloading backports.entry_points_selectable-1.1.0-py2.py3-none-any.whl (6.2 kB)
Collecting distlib<1,>=0.3.1
  Downloading distlib-0.3.2-py2.py3-none-any.whl (338 kB)
     |████████████████████████████████| 338 kB 4.0 MB/s
Collecting filelock<4,>=3.0.0
  Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting platformdirs<3,>=2
  Downloading platformdirs-2.3.0-py3-none-any.whl (13 kB)
Installing collected packages: platformdirs, filelock, distlib, backports.entry-points-selectable, virtualenv
Successfully installed backports.entry-points-selectable-1.1.0 distlib-0.3.2 filelock-3.0.12 platformdirs-2.3.0 virtualenv-20.7.2
$ realpath $(which virtualenv)
/Users/username/Library/Python/3.9/bin/virtualenv
$ virtualenv -vvv --with-traceback /tmp/testvenv
<clip>
1790 created virtual environment CPython3.9.7.final.0-64 in 1677ms
  creator CPython3Posix(dest=/private/tmp/testvenv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/username/Library/Application Support/virtualenv)
    added seed packages: pip==21.2.3, setuptools==57.4.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator [WARNING __main__:19]
$ echo $?
0

So nope, doesn't fail. Let me know if you need the full output regardless.

arcivanov · 2021-09-06T09:04:25Z

Could you please post the shebang of the virtualenv file (first line)?
And yes please post the whole output.

hoxu · 2021-09-07T05:30:25Z

$ head -n1 $(which virtualenv)
#!/usr/local/opt/python@3.9/bin/python3.9
$ realpath /usr/local/opt/python@3.9/bin/python3.9
/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/bin/python3.9

$ virtualenv -vvv --with-traceback /tmp/testvenv
114 setup logging to NOTSET [DEBUG report:43]
140 find interpreter for spec PythonSpec(path=/usr/local/opt/python@3.9/bin/python3.9) [INFO builtin:62]
140 proposed PythonInfo(spec=CPython3.9.7.final.0-64, exe=/usr/local/opt/python@3.9/bin/python3.9, platform=darwin, version='3.9.7 (default, Sep  3 2021, 12:37:55) \n[Clang 12.0.5 (clang-1205.0.22.9)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:69]
140 accepted PythonInfo(spec=CPython3.9.7.final.0-64, exe=/usr/local/opt/python@3.9/bin/python3.9, platform=darwin, version='3.9.7 (default, Sep  3 2021, 12:37:55) \n[Clang 12.0.5 (clang-1205.0.22.9)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:71]
143 filesystem is not case-sensitive [DEBUG info:28]
186 create virtual environment via CPython3Posix(dest=/private/tmp/testvenv, clear=False, no_vcs_ignore=False, global=False) [INFO session:52]
186 create folder /private/tmp/testvenv/bin [DEBUG _sync:25]
186 create folder /private/tmp/testvenv/lib/python3.9/site-packages [DEBUG _sync:25]
187 write /private/tmp/testvenv/pyvenv.cfg [DEBUG pyenv_cfg:34]
187     home = /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9 [DEBUG pyenv_cfg:38]
187     implementation = CPython [DEBUG pyenv_cfg:38]
187     version_info = 3.9.7.final.0 [DEBUG pyenv_cfg:38]
187     virtualenv = 20.7.2 [DEBUG pyenv_cfg:38]
187     include-system-site-packages = false [DEBUG pyenv_cfg:38]
187     base-prefix = /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9 [DEBUG pyenv_cfg:38]
187     base-exec-prefix = /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9 [DEBUG pyenv_cfg:38]
187     base-executable = /usr/local/opt/python@3.9/bin/python3.9 [DEBUG pyenv_cfg:38]
187 symlink /usr/local/opt/python@3.9/bin/python3.9 to /private/tmp/testvenv/bin/python [DEBUG _sync:44]
188 create virtualenv import hook file /private/tmp/testvenv/lib/python3.9/site-packages/_virtualenv.pth [DEBUG api:95]
189 create /private/tmp/testvenv/lib/python3.9/site-packages/_virtualenv.py [DEBUG api:98]
190 ============================== target debug ============================== [DEBUG session:54]
190 debug via /private/tmp/testvenv/bin/python /Users/username/Library/Python/3.9/lib/python/site-packages/virtualenv/create/debug.py [DEBUG creator:224]
190 {
  "sys": {
    "executable": "/private/tmp/testvenv/bin/python",
    "_base_executable": "/private/tmp/testvenv/bin/python",
    "prefix": "/private/tmp/testvenv",
    "base_prefix": "/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9",
    "real_prefix": null,
    "exec_prefix": "/private/tmp/testvenv",
    "base_exec_prefix": "/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9",
    "path": [
      "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python39.zip",
      "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9",
      "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload",
      "/private/tmp/testvenv/lib/python3.9/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.9.7 (default, Sep  3 2021, 12:37:55) \n[Clang 12.0.5 (clang-1205.0.22.9)]",
  "makefile_filename": "/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin/Makefile",
  "os": "<module 'os' from '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py'>",
  "site": "<module 'site' from '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site.py'>",
  "datetime": "<module 'datetime' from '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/datetime.py'>",
  "math": "<module 'math' from '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/math.cpython-39-darwin.so'>",
  "json": "<module 'json' from '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py'>"
} [DEBUG session:55]
248 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/username/Library/Application Support/virtualenv) [INFO session:59]
251 got embed update of distribution setuptools from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/setuptools.json [DEBUG via_disk_folder:135]
252 got embed update of distribution wheel from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/wheel.json [DEBUG via_disk_folder:135]
252 got embed update of distribution pip from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/pip.json [DEBUG via_disk_folder:135]
256 got embed update of distribution setuptools from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/setuptools.json [DEBUG via_disk_folder:135]
256 got embed update of distribution pip from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/pip.json [DEBUG via_disk_folder:135]
256 got embed update of distribution wheel from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/wheel.json [DEBUG via_disk_folder:135]
257 install setuptools from wheel /Users/username/Library/Python/3.9/lib/python/site-packages/virtualenv/seed/wheels/embed/setuptools-57.4.0-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
257 install pip from wheel /Users/username/Library/Python/3.9/lib/python/site-packages/virtualenv/seed/wheels/embed/pip-21.2.3-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
257 install wheel from wheel /Users/username/Library/Python/3.9/lib/python/site-packages/virtualenv/seed/wheels/embed/wheel-0.37.0-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
258 Attempting to acquire lock 4507837008 on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [DEBUG filelock:270]
258 Attempting to acquire lock 4507839552 on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [DEBUG filelock:270]
258 Attempting to acquire lock 4507839648 on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [DEBUG filelock:270]
259 Lock 4507839552 acquired on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [INFO filelock:274]
259 Lock 4507839648 acquired on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [INFO filelock:274]
259 build install image for pip-21.2.3-py3-none-any.whl to /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any [DEBUG base:52]
259 Lock 4507837008 acquired on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [INFO filelock:274]
259 build install image for wheel-0.37.0-py2.py3-none-any.whl to /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any [DEBUG base:52]
259 build install image for setuptools-57.4.0-py3-none-any.whl to /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any [DEBUG base:52]
287 Attempting to release lock 4507839648 on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [DEBUG filelock:315]
287 Lock 4507839648 released on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [INFO filelock:318]
288 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any/wheel-0.37.0.dist-info to /private/tmp/testvenv/lib/python3.9/site-packages/wheel-0.37.0.dist-info [DEBUG _sync:52]
335 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any/wheel to /private/tmp/testvenv/lib/python3.9/site-packages/wheel [DEBUG _sync:52]
412 Attempting to release lock 4507837008 on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [DEBUG filelock:315]
412 Lock 4507837008 released on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [INFO filelock:318]
413 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/setuptools-57.4.0.dist-info to /private/tmp/testvenv/lib/python3.9/site-packages/setuptools-57.4.0.dist-info [DEBUG _sync:52]
414 copy /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any/wheel-0.37.0.virtualenv to /private/tmp/testvenv/lib/python3.9/site-packages/wheel-0.37.0.virtualenv [DEBUG _sync:52]
417 generated console scripts wheel-3.9 wheel3 wheel wheel3.9 [DEBUG base:48]
427 copy /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/distutils-precedence.pth to /private/tmp/testvenv/lib/python3.9/site-packages/distutils-precedence.pth [DEBUG _sync:52]
432 copy /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/setuptools-57.4.0.virtualenv to /private/tmp/testvenv/lib/python3.9/site-packages/setuptools-57.4.0.virtualenv [DEBUG _sync:52]
433 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/setuptools to /private/tmp/testvenv/lib/python3.9/site-packages/setuptools [DEBUG _sync:52]
529 Attempting to release lock 4507839552 on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [DEBUG filelock:315]
529 Lock 4507839552 released on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [INFO filelock:318]
530 copy /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any/pip-21.2.3.virtualenv to /private/tmp/testvenv/lib/python3.9/site-packages/pip-21.2.3.virtualenv [DEBUG _sync:52]
530 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any/pip to /private/tmp/testvenv/lib/python3.9/site-packages/pip [DEBUG _sync:52]
657 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/pkg_resources to /private/tmp/testvenv/lib/python3.9/site-packages/pkg_resources [DEBUG _sync:52]
684 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/_distutils_hack to /private/tmp/testvenv/lib/python3.9/site-packages/_distutils_hack [DEBUG _sync:52]
688 generated console scripts  [DEBUG base:48]
1776 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any/pip-21.2.3.dist-info to /private/tmp/testvenv/lib/python3.9/site-packages/pip-21.2.3.dist-info [DEBUG _sync:52]
1785 generated console scripts pip-3.9 pip3 pip3.9 pip [DEBUG base:48]
1786 add activators for Bash, CShell, Fish, PowerShell, Python [INFO session:64]
1789 write /private/tmp/testvenv/pyvenv.cfg [DEBUG pyenv_cfg:34]
1789    home = /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9 [DEBUG pyenv_cfg:38]
1789    implementation = CPython [DEBUG pyenv_cfg:38]
1789    version_info = 3.9.7.final.0 [DEBUG pyenv_cfg:38]
1789    virtualenv = 20.7.2 [DEBUG pyenv_cfg:38]
1789    include-system-site-packages = false [DEBUG pyenv_cfg:38]
1789    base-prefix = /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9 [DEBUG pyenv_cfg:38]
1789    base-exec-prefix = /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9 [DEBUG pyenv_cfg:38]
1789    base-executable = /usr/local/opt/python@3.9/bin/python3.9 [DEBUG pyenv_cfg:38]
1790 created virtual environment CPython3.9.7.final.0-64 in 1677ms
  creator CPython3Posix(dest=/private/tmp/testvenv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/username/Library/Application Support/virtualenv)
    added seed packages: pip==21.2.3, setuptools==57.4.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator [WARNING __main__:19]

arcivanov · 2021-09-12T22:57:31Z

I can't seem to reproduce this issue with BigSur and Homebrew on GitHub Actions. Could you please see these logs and tell me whether anything jumps at you as different from your system configuration?

https://github.com/pybuilder/pybuilder/commit/618512b0f356e3314b1218ae7c0192b9fea49c6b/checks/3581148294/logs

hoxu · 2021-09-13T05:58:00Z

I noticed one difference:

$ which python3
/usr/local/bin/python3
$ realpath $(which python3)
/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/bin/python3.9
$ l /usr/local/opt/python@3.9
lrwxr-xr-x  1 username  admin  26 Sep  6 09:21 /usr/local/opt/python@3.9 -> ../Cellar/python@3.9/3.9.7

I seem to be using python3 from /usr/local/bin/python3, while in GitHub Actions you add /usr/local/opt/python@3.9/bin to PATH.

2021-09-12T22:24:47.3742910Z �[36;1mecho "/usr/local/opt/python@3.9/bin" >> $GITHUB_PATH�[0m

Both symlinks seem to exist for me, though.

I don't think this is relevant.

Does the GitHub Actions workflow work with pybuilder 0.12.10? I can't test with --pre because of the StopIteration issue.

arcivanov · 2021-09-13T05:59:40Z

I might have supplied you with the wrong logs. One sec.

arcivanov · 2021-09-13T06:01:53Z

Yeah sorry, wrong logs. Correct link: https://github.com/pybuilder/pybuilder/commit/618512b0f356e3314b1218ae7c0192b9fea49c6b/checks/3581148294/logs

2021-09-12T22:25:46.2604910Z To reinstall 3.9.7, run:
2021-09-12T22:25:46.2605500Z   brew reinstall python@3.9
2021-09-12T22:25:46.2734340Z ##[group]Run pwd
2021-09-12T22:25:46.2735090Z �[36;1mpwd�[0m
2021-09-12T22:25:46.2735500Z �[36;1mwhich python3�[0m
2021-09-12T22:25:46.2735920Z �[36;1mpython3 --version�[0m
2021-09-12T22:25:46.2736410Z �[36;1mpython3 -m pip --version�[0m
2021-09-12T22:25:46.2736880Z �[36;1mecho GITHUB_REF $GITHUB_REF�[0m
2021-09-12T22:25:46.2737570Z �[36;1mecho GITHUB_HEAD_REF $GITHUB_HEAD_REF�[0m
2021-09-12T22:25:46.2738150Z �[36;1mecho GITHUB_BASE_REF $GITHUB_BASE_REF�[0m
2021-09-12T22:25:46.2925070Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2021-09-12T22:25:46.2925760Z env:
2021-09-12T22:25:46.2926200Z   DEPLOY_BRANCHES: refs/heads/master
2021-09-12T22:25:46.2926690Z   DEPLOY_PYTHONS: 3.9
2021-09-12T22:25:46.2927090Z   DEPLOY_OSES: Linux
2021-09-12T22:25:46.2927560Z   PYB_ARGS: -E ci -v -X analyze install
2021-09-12T22:25:46.2928230Z   TWINE_USERNAME: pybuilder-travis2
2021-09-12T22:25:46.2928770Z   TWINE_PASSWORD: 
2021-09-12T22:25:46.2929180Z   SKIP_JOB: false
2021-09-12T22:25:46.2929660Z ##[endgroup]
2021-09-12T22:25:46.3201590Z /Users/runner/work/pybuilder/pybuilder
2021-09-12T22:25:46.3225010Z /usr/local/opt/python@3.9/bin/python3
2021-09-12T22:25:46.3843640Z Python 3.9.7
2021-09-12T22:25:48.2166530Z pip 21.2.4 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
2021-09-12T22:25:48.2455010Z GITHUB_REF refs/pull/786/merge
2021-09-12T22:25:48.2456760Z GITHUB_HEAD_REF pr-769
2021-09-12T22:25:48.2459170Z GITHUB_BASE_REF master

arcivanov · 2021-09-13T06:06:31Z

That's the thing - StopIteration issue doesn't reproduce either on GHA macos 11 + Homebrew. I wonder if it's some standard environment on a laptop that doesn't appear on macos boxes...
I can hypothetically build a 0.12 on GHA and see what's up, but 0.12 contains an old version of bundled VEnv as well, which was never tested with Python 3.9 and latest version of setuptools and wheel, so it's a whole other can of worms...

arcivanov · 2021-09-13T06:10:31Z

I seem to be using python3 from /usr/local/bin/python3, while in GitHub Actions you add /usr/local/opt/python@3.9/bin to PATH.

Now this is interesting... Did you add that to PATH? Because I think I saw Homebrew not updating that on GHA, that's why python location check is there. I'll test tomorrow if it makes a difference and will look at whether Brew updates that symlink.

hoxu · 2021-09-13T06:11:33Z

Can you check what this outputs on GHA?

$ python3 -c 'import sys; print(sys.path); print(sys.prefix)'
['', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/Users/username/Library/Python/3.9/lib/python/site-packages', '/usr/local/lib/python3.9/site-packages']
/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9

hoxu · 2021-09-13T06:13:32Z

Did you add that to PATH?

I have /usr/local/bin in PATH, but it does not make difference which way python3 is found. Build still fails with AssertionError even if prefixed with PATH=/usr/local/opt/python@3.9/bin:$PATH.

arcivanov · 2021-09-13T06:16:21Z

That is weird... Are you on M1 by any chance? Or Intel?

arcivanov · 2021-09-13T06:23:25Z

https://github.com/pybuilder/pybuilder/actions/runs/1228406688

hoxu · 2021-09-13T06:24:13Z

Not M1.

macOS Big Sur 11.5.2 on MacBook Pro (13-inch, 2019, Four Thunderbolt 3 ports).

arcivanov · 2021-09-13T06:24:24Z

This is the homebrewed one: https://github.com/pybuilder/pybuilder/runs/3583700812?check_suite_focus=true

arcivanov · 2021-09-13T06:25:51Z

python3 -c 'import sys; print(sys.path); print(sys.prefix); print(sys.base_prefix); print(sys.exec_prefix); print(sys.executable);'

['', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/site-packages']
/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9
/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9
/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9
/usr/local/opt/python@3.9/bin/python3.9

arcivanov · 2021-09-13T06:29:28Z

Is your PyBuilder installed in --user or system?

arcivanov · 2021-09-13T14:06:17Z

@hoxu?

hoxu · 2021-09-14T05:18:29Z

I run pybuilder mostly with a custom build script in a virtualenv like this:

python3 -m venv vebuild
. vebuild/bin/activate
pip --disable-pip-version-check install pybuilder

But I get the same AssertionError outside virtualenv with pybuilder installed with pip3 install --user pybuilder==0.12.10.

Apologies for not having enough time to dig properly into this.

Maybe @dindurthy could take a look at the GitHub Actions workflow logs and see if there are any differences to local execution?

hoxu · 2021-09-14T05:20:09Z

Same exception with pybuilder installed withpip3 install pybuilder as well.

hoxu · 2021-09-14T05:58:27Z

['', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/site-packages']
/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9
/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9
/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9
/usr/local/opt/python@3.9/bin/python3.9

Here is a diff from mine to GHA:

  ['',
   '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python39.zip',
   '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9',
   '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload',
-  '/Users/username/Library/Python/3.9/lib/python/site-packages',
   '/usr/local/lib/python3.9/site-packages']
  /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9

arcivanov · 2021-09-15T05:32:33Z

Rejoice! I was able to reproduce this issue on Linux :) At least the StopIteration part that is.

hoxu · 2021-09-15T08:55:55Z

@arcivanov let me know when --pre with #794 is available, so that I can try to reproduce this with the StopIteration fix. Good job fixing it :)

arcivanov · 2021-09-15T09:38:22Z

@hoxu go ahead please with --pre testing. :)

arcivanov · 2021-09-16T14:51:32Z

@hoxu did you have a chance to try it out?

hoxu · 2021-09-17T07:39:28Z

Working on other projects, but a quick test with --pre PyBuilder version 0.13.0.dev20210917052200 & Python 3.9.7 seems to indicate that this issue is fixed! Awesome, good job!

I think this can be closed after a release.

arcivanov · 2021-09-17T07:58:22Z

Glad to hear! The release is a bit away, as an issue in virtualenv has been discovered as well.

Vendorize latest virtualenv with cleanup fixes fixes pybuilder#795, pybuilder#777, pybuilder#774, pybuilder#771

Vendorize latest virtualenv with cleanup fixes Bump version to 0.13.0 fixes pybuilder#795, pybuilder#777, pybuilder#774, pybuilder#771

arcivanov mentioned this issue Sep 6, 2021

VirtualEnv fails with Homebrewed Python 3.9 on macOS Big Sur pypa/virtualenv#2176

Closed

arcivanov mentioned this issue Sep 8, 2021

pybuilder fails on osx m1 #774

Closed

arcivanov added the bug label Sep 14, 2021

arcivanov mentioned this issue Sep 15, 2021

StopIteration: (virtualenv/run/plugin/discovery.py:22) with 0.13.0 #793

Closed

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Sep 26, 2021

Prepare 0.13.0

a85db1c

Vendorize latest virtualenv with cleanup fixes fixes pybuilder#795, pybuilder#777, pybuilder#774, pybuilder#771

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Sep 26, 2021

Prepare 0.13.0

c61bec6

Vendorize latest virtualenv with cleanup fixes Bump version to 0.13.0 fixes pybuilder#795, pybuilder#777, pybuilder#774, pybuilder#771

arcivanov mentioned this issue Sep 26, 2021

Prepare 0.13.0 #802

Merged

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Sep 26, 2021

Prepare 0.13.0

957a5de

Vendorize latest virtualenv with cleanup fixes Bump version to 0.13.0 fixes pybuilder#795, pybuilder#777, pybuilder#774, pybuilder#771

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Sep 26, 2021

Prepare 0.13.0

7d0dc08

Vendorize latest virtualenv with cleanup fixes Bump version to 0.13.0 fixes pybuilder#795, pybuilder#777, pybuilder#774, pybuilder#771

arcivanov closed this as completed Sep 27, 2021

OSX + homebrew python@3.9: pybuilder/_vendor/virtualenv.py change_prefix does not consider symlinks, raises invalid AssertionError #771

OSX + homebrew python@3.9: pybuilder/_vendor/virtualenv.py change_prefix does not consider symlinks, raises invalid AssertionError #771

Comments

hoxu commented Apr 28, 2021

dindurthy commented May 3, 2021

arcivanov commented May 4, 2021 • edited Loading

arcivanov commented May 4, 2021

arcivanov commented May 4, 2021

hoxu commented May 4, 2021

hoxu commented May 4, 2021

dindurthy commented May 4, 2021

hoxu commented Aug 25, 2021

arcivanov commented Sep 5, 2021

hoxu commented Sep 6, 2021

arcivanov commented Sep 6, 2021

arcivanov commented Sep 6, 2021

hoxu commented Sep 6, 2021 • edited Loading

arcivanov commented Sep 6, 2021

hoxu commented Sep 7, 2021

arcivanov commented Sep 12, 2021

hoxu commented Sep 13, 2021

arcivanov commented Sep 13, 2021

arcivanov commented Sep 13, 2021

arcivanov commented Sep 13, 2021

arcivanov commented Sep 13, 2021

hoxu commented Sep 13, 2021

hoxu commented Sep 13, 2021

arcivanov commented Sep 13, 2021

arcivanov commented Sep 13, 2021

hoxu commented Sep 13, 2021

arcivanov commented Sep 13, 2021

arcivanov commented Sep 13, 2021

arcivanov commented Sep 13, 2021

arcivanov commented Sep 13, 2021

hoxu commented Sep 14, 2021

hoxu commented Sep 14, 2021

hoxu commented Sep 14, 2021

arcivanov commented Sep 15, 2021 • edited Loading

hoxu commented Sep 15, 2021

arcivanov commented Sep 15, 2021

arcivanov commented Sep 16, 2021

hoxu commented Sep 17, 2021

arcivanov commented Sep 17, 2021

arcivanov commented May 4, 2021 •

edited

Loading

hoxu commented Sep 6, 2021 •

edited

Loading

arcivanov commented Sep 15, 2021 •

edited

Loading