Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

isolated build env should ignore deps #1239

Closed
webknjaz opened this issue Apr 1, 2019 · 12 comments
Closed

isolated build env should ignore deps #1239

webknjaz opened this issue Apr 1, 2019 · 12 comments

Comments

@webknjaz
Copy link
Contributor

@webknjaz webknjaz commented Apr 1, 2019

So it looks like when you combine isolated_build = true with deps = -rrequirements.txt in combines deps from pyproject.toml and deps entries expecting that those would be direct dist specs and tries to parse them as such.

Python 3.7.1 + tox 3.8.4.

$ tox -e metadata-validation --notest -vv 
using tox.ini: ~/src/github/ansible/molecule/tox.ini (pid 8030)
  removing ~/src/github/ansible/molecule/.tox/log
using tox-3.8.4 from ~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/__init__.py (pid 8030)
.package start: getenv ~/src/github/ansible/molecule/.tox/.package
.package cannot reuse: no previous config ~/src/github/ansible/molecule/.tox/.package/.tox-config1
.package create: ~/src/github/ansible/molecule/.tox/.package
  removing ~/src/github/ansible/molecule/.tox/.package
setting PATH=~/src/github/ansible/molecule/.tox/.package/bin:~/.pyenv/versions/3.7.1/bin:~/.pyenv/libexec:~/.pyenv/plugins/python-build/bin:~/.pyenv/plugins/pyenv-virtualenv/bin:~/.pyenv/plugins/pyenv-update/bin:~/.pyenv/plugins/pyenv-installer/bin:~/.pyenv/plugins/pyenv-doctor/bin:~/.nvm/versions/node/v9.8.0/bin:~/.pyenv/shims:/usr/lib/llvm/7/bin:/usr/lib/llvm/6/bin:/usr/lib/llvm/5/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:~/.pyenv/bin
[8077] ~/src/github/ansible/molecule/.tox$ ~/.pyenv/versions/3.7.1/bin/python3.7 -m virtualenv --system-site-packages --python ~/.pyenv/versions/3.7.1/bin/python3.7 .package
Already using interpreter ~/.pyenv/versions/3.7.1/bin/python3.7
Using base prefix '~/.pyenv/versions/3.7.1'
~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
New python executable in ~/src/github/ansible/molecule/.tox/.package/bin/python3.7
Also creating executable in ~/src/github/ansible/molecule/.tox/.package/bin/python
Installing setuptools, pip, wheel...done.
.package installdeps: setuptools >= 40.8.0, setuptools_scm >= 1.15.0, setuptools_scm_git_archive >= 1.0, wheel, -rlint-requirements.txt, -rtest-requirements.txt
setting PATH=~/src/github/ansible/molecule/.tox/.package/bin:~/.pyenv/versions/3.7.1/bin:~/.pyenv/libexec:~/.pyenv/plugins/python-build/bin:~/.pyenv/plugins/pyenv-virtualenv/bin:~/.pyenv/plugins/pyenv-update/bin:~/.pyenv/plugins/pyenv-installer/bin:~/.pyenv/plugins/pyenv-doctor/bin:~/.nvm/versions/node/v9.8.0/bin:~/.pyenv/shims:/usr/lib/llvm/7/bin:/usr/lib/llvm/6/bin:/usr/lib/llvm/5/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:~/.pyenv/bin
[8144] ~/src/github/ansible/molecule$ ~/src/github/ansible/molecule/.tox/.package/bin/python -m pip install 'setuptools >= 40.8.0' 'setuptools_scm >= 1.15.0' 'setuptools_scm_git_archive >= 1.0' wheel -rlint-requirements.txt -rtest-requirements.txt
Double requirement given: wheel==0.30.0 (from -r test-requirements.txt (line 12)) (already in wheel, name='wheel')
ERROR: invocation failed (exit code 1)
ERROR: could not install deps [setuptools >= 40.8.0, setuptools_scm >= 1.15.0, setuptools_scm_git_archive >= 1.0, wheel, -rlint-requirements.txt, -rtest-requirements.txt]; v = InvocationError("~/src/github/ansible/molecule/.tox/.package/bin/python -m pip install 'setuptools >= 40.8.0' 'setuptools_scm >= 1.15.0' 'setuptools_scm_git_archive >= 1.0' wheel -rlint-requirements.txt -rtest-requirements.txt", 1)
.package finish: getenv ~/src/github/ansible/molecule/.tox/.package after 3.18 seconds
.package start: get-build-requires ~/src/github/ansible/molecule/.tox/.package
setting PATH=~/src/github/ansible/molecule/.tox/.package/bin:~/.pyenv/versions/3.7.1/bin:~/.pyenv/libexec:~/.pyenv/plugins/python-build/bin:~/.pyenv/plugins/pyenv-virtualenv/bin:~/.pyenv/plugins/pyenv-update/bin:~/.pyenv/plugins/pyenv-installer/bin:~/.pyenv/plugins/pyenv-doctor/bin:~/.nvm/versions/node/v9.8.0/bin:~/.pyenv/shims:/usr/lib/llvm/7/bin:/usr/lib/llvm/6/bin:/usr/lib/llvm/5/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:~/.pyenv/bin
[8151] ~/src/github/ansible/molecule$ ~/src/github/ansible/molecule/.tox/.package/bin/python ~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/helper/build_requires.py setuptools.build_meta '' >.tox/.package/log/.package-0.log
.package finish: get-build-requires ~/src/github/ansible/molecule/.tox/.package after 0.30 seconds
Traceback (most recent call last):
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 90, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 3739, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 3400, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 2711, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3042, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__
    requirement_string[e.loc:e.loc + 8]))
pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'-rlint-r'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/.pyenv/versions/3.7.1/bin/tox", line 10, in <module>
    sys.exit(cmdline())
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/session/__init__.py", line 43, in cmdline
    main(args)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/session/__init__.py", line 68, in main
    exit_code = session.runcommand()
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/session/__init__.py", line 192, in runcommand
    return self.subcommand_test()
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/session/__init__.py", line 208, in subcommand_test
    venv.package = self.hook.tox_package(session=self, venv=venv)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/callers.py", line 201, in _multicall
    return outcome.get_result()
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/callers.py", line 76, in get_result
    raise ex[1].with_traceback(ex[2])
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/__init__.py", line 16, in tox_package
    session.package, session.dist = get_package(session)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/__init__.py", line 29, in get_package
    package = acquire_package(config, session)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/__init__.py", line 40, in acquire_package
    path = build_package(config, session)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/builder/__init__.py", line 9, in build_package
    return build(config, session)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/builder/isolated.py", line 34, in build
    base_build_deps = {pkg_resources.Requirement(r.name).key for r in package_venv.envconfig.deps}
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/builder/isolated.py", line 34, in <setcomp>
    base_build_deps = {pkg_resources.Requirement(r.name).key for r in package_venv.envconfig.deps}
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3044, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "'-rlint-r'"
@gaborbernat
Copy link
Member

@gaborbernat gaborbernat commented Apr 1, 2019

Although the error is kinda cryptic this works as expected. If you define deps at base level (testenv) you're expected to clear it for the isolated build env:

[testenv:.package]
deps = 

Will work.

@webknjaz
Copy link
Contributor Author

@webknjaz webknjaz commented Apr 1, 2019

So why can't tox do it automatically?

@gaborbernat
Copy link
Member

@gaborbernat gaborbernat commented Apr 1, 2019

We could... to be fair. Would make isolated build kinda special and not follow other env override logic, but I assume we already differ enough...

@gaborbernat gaborbernat changed the title [BUG] isolated_build broken with indirect deps isolated build env should ignore deps Apr 1, 2019
@webknjaz
Copy link
Contributor Author

@webknjaz webknjaz commented Apr 1, 2019

Yeah, at least make it more obvious what's happening and maybe yell at -rrequirements.txt in deps when first seeing it instead of making some internal parse failure leak...

@gaborbernat
Copy link
Member

@gaborbernat gaborbernat commented Apr 1, 2019

https://github.com/tox-dev/tox/blob/master/src/tox/config/__init__.py#L1138 beside description we should probably also ignore deps (aka overwrite to an empty array) 👍Want to do a PR with a test?

GitHub
Command line driven CI frontend and development task automation tool - tox-dev/tox

@webknjaz
Copy link
Contributor Author

@webknjaz webknjaz commented Apr 1, 2019

Maybe, but not right now :) Trying to refactor packaging in another project...

@webknjaz
Copy link
Contributor Author

@webknjaz webknjaz commented Apr 2, 2019

@gaborbernat after hitting pypa/pip#6264 (comment) I thought that maybe this env could also disable system-site-packages for isolated venv as well?

@gaborbernat
Copy link
Member

@gaborbernat gaborbernat commented Apr 2, 2019

Agreed. Though I would personally remove and banish that option entirely. I don't think it ever is a good idea to have it.

@gaborbernat
Copy link
Member

@gaborbernat gaborbernat commented Apr 2, 2019

What's the reason your project uses it @webknjaz?

@gaborbernat gaborbernat mentioned this issue Apr 2, 2019
@helpr helpr bot added the pr-available label Apr 2, 2019
@helpr helpr bot added pr-merged and removed pr-available labels Apr 2, 2019
@webknjaz
Copy link
Contributor Author

@webknjaz webknjaz commented Apr 3, 2019

@gaborbernat the reason is that it uses libselinux python wrapper, which is not distributed via PyPI and linked against a lot of system things and only distributed via OS package managers meaning that it's either this or symlinking things into tox's venvs. It also speeds up some things when envs are heavy. It wasn't my idea but I couldn't come up with a better solution... I was trying to come up with a repackaging of selinux as manylinux1 wheels but it doesn't seem possible at all.

Ref: ansible-community/molecule#1823 (comment)

@gaborbernat
Copy link
Member

@gaborbernat gaborbernat commented Apr 3, 2019

I've done the fix yesterday and will release today 🤔

@webknjaz
Copy link
Contributor Author

@webknjaz webknjaz commented Apr 3, 2019

Nice, thanks!

@tox-dev tox-dev locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants