Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Environment variable usage schema {env:XXX} is not working #99

Closed
pytoxbot opened this issue Sep 17, 2016 · 15 comments
Closed

Environment variable usage schema {env:XXX} is not working #99

pytoxbot opened this issue Sep 17, 2016 · 15 comments

Comments

@pytoxbot
Copy link

Version info: tox-1.4.3, tox-1.5.dev7 (current head/tip)

The documentation describes that environment variables can be used/referenced as shown above. But there seems to be a number of issues.

PROBLEMS:

  • Environment variables that are setup via "setenv= ..." can not be referenced in "commands= ..." part.
  • Option --showconfig does not evaluate "setenv= ..." parts in testenv section before the output is shown.
#!ini
[testenv:example]
commands=
    echo ENVIRONMENT-VAR foo: {env:foo}
    echo ENVIRONMENT-VAR bar: {env:bar}
setenv =
    foo = Alice
    bar = Bob
deps=

ADDITIONAL NOTE:
Current repository tip has a problem with {homedir} placeholder when indexserver= local=file://{homedir}/.pip_downloads is used. This is working in box-1.4.3, but no longer working in tox-1.5.dev7.
@pytoxbot
Copy link
Author

Original comment by @hpk42

add changelog: fix issue99: the {env:...} substitution now properly uses environment
settings from the setenv section. Thanks Itxaka Serrano.

→ <>

@pytoxbot
Copy link
Author

Original comment by @hpk42

Merged in itxaka/tox/fix_env_use (pull request #169)

Tries to fix #99

→ <<cset 763eeed49a7c>>

1 similar comment
@pytoxbot
Copy link
Author

Original comment by @hpk42

Merged in itxaka/tox/fix_env_use (pull request #169)

Tries to fix #99

→ <<cset 763eeed49a7c>>

@pytoxbot
Copy link
Author

Original comment by @hpk42

thanks, looks good apart from one bit where i commented. Looking forward to merge :)

@pytoxbot
Copy link
Author

Original comment by @Itxaka

@hpk42 There is already a PR for this https://bitbucket.org/hpk42/tox/pull-requests/169/tries-to-fix-99/diff

@pytoxbot
Copy link
Author

Original comment by @hpk42

@mkumatag FYI tox doesn't have a lot of regular contributors, i am afraid. myself, I only get to look at things from time to time and prioritize things according to my own needs and customers.

@Itxaka yes, your analysis is correct i think. Haven't looked if there maybe is a quick way to fix it and have setenv be honored during configuration reading time already. Happy if you find something and can do a PR -- the functional test you basically already have.

@pytoxbot
Copy link
Author

Original comment by mkumatag

Even I'm looking for this.! I'm surprise to see this defect with no activity for more than 2 years :(

@pytoxbot
Copy link
Author

Original comment by @Itxaka

This is still open and not working on current tox:

#!python

$ tox --version
2.1.1 imported from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc

$ cat tox.ini 
[tox]
envlist = py27
skipsdist=True

[testenv:py27]
setenv =
    PYTHONPATH={toxinidir}
commands =
    ls {env:PYTHONPATH}%                                                                                                                                                                                     

$ tox
Traceback (most recent call last):
  File "/usr/local/bin/tox", line 11, in <module>
    sys.exit(cmdline())
  File "/usr/local/lib/python2.7/dist-packages/tox/session.py", line 38, in main
    config = prepare(args)
  File "/usr/local/lib/python2.7/dist-packages/tox/session.py", line 26, in prepare
    config = parseconfig(args)
  File "/usr/local/lib/python2.7/dist-packages/tox/config.py", line 219, in parseconfig
    parseini(config, inipath)
  File "/usr/local/lib/python2.7/dist-packages/tox/config.py", line 669, in __init__
    self.make_envconfig(name, section, reader._subs, config)
  File "/usr/local/lib/python2.7/dist-packages/tox/config.py", line 697, in make_envconfig
    res = meth(env_attr.name, env_attr.default)
  File "/usr/local/lib/python2.7/dist-packages/tox/config.py", line 864, in getargvlist
    return _ArgvlistReader.getargvlist(self, s)
  File "/usr/local/lib/python2.7/dist-packages/tox/config.py", line 1015, in getargvlist
    commands.append(cls.processcommand(reader, current_command))
  File "/usr/local/lib/python2.7/dist-packages/tox/config.py", line 1045, in processcommand
    new_word = reader._replace(word)
  File "/usr/local/lib/python2.7/dist-packages/tox/config.py", line 981, in _replace
    return RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/local/lib/python2.7/dist-packages/tox/config.py", line 977, in _replace_match
    return handler(match)
  File "/usr/local/lib/python2.7/dist-packages/tox/config.py", line 921, in _replace_env
    (envkey, envkey))
tox.ConfigError: ConfigError: substitution env:'PYTHONPATH': unknown environment variable 'PYTHONPATH'

Also happens if used anywhere else as far as my tests go.

Its been reported since 2 years ago and there has been no advance on it, do you need any help on it? Been looking at the code myself but found it quite difficult to wrap my head around it on how to fix it properly.

Im guessing that _replace_env is failing because is trying to check if the requested env is on os.environ while parsing the config, but the env variables are not set until the runtests action is launched (line 330 in venv.py: env = self._getenv(testcommand=True)

Will try to see if there is a good solution to fix this :)

@pytoxbot
Copy link
Author

Original comment by @hpk42

Seems indeed that setenv is not respected in commands.

@pytoxbot
Copy link
Author

Original comment by @coagulant

Could you please reopen? since the "Environment variables that are setup via "setenv= ..." can not be referenced in "commands= ..." part." still holds true.

@pytoxbot
Copy link
Author

Original comment by @jenisys

@Krisztian
Well actually my tox -e init is working as expected with tox 1.4.3. The reason for this is in the setenv= PIP_INDEX_URL = ... in the testenv:init. It provides pip with the pypi URL (as needed) for online usage.

I assume you just inspected the "tox.ini" (or tried it out with current tip).

Otherwise, thx for the fix.

@pytoxbot
Copy link
Author

Original comment by @hpk42

Fix #99 regression - modify regular expression to be non-greedy

→ <<cset 8dd1345f8c32>>

@pytoxbot
Copy link
Author

Original comment by @e3krisztian

Thanks!
From this info I was able to reproduce the regression.

My previous fix introduced a bug, that causes multiple substitutions on same line to fail (non-greedy -> greedy re).
This should be easy to fix (on the way).

(BTW your tox.ini fails until the indexserver default entry is edited for online use - at least for the first time both tox -e init and tox -e py27 fails)

NOTE: the discussion so far was about a regression, not about the main issue regarding substitutions of the format {env:variable}, so the issue is NOT resolved even if the regression is fixed.

@pytoxbot
Copy link
Author

Original comment by @jenisys

The problem is already with tox --showconfig.
The 1.5dev version just pukes on me when "{homedir}" is used in "tox.ini".

shell:xxx [525]. tox --version
1.5.dev7 imported from /.../tox/tox/__init__.pyc

shell:xxx [526]. tox --showconfig
Traceback (most recent call last):
  File "/usr/local/bin/tox", line 9, in <module>
    load_entry_point('tox==1.5.dev7', 'console_scripts', 'tox')()
  File "/.../tox/tox/_cmdline.py", line 24, in main
    config = parseconfig(args, 'tox')
  File "/.../tox/tox/_config.py", line 42, in parseconfig
    parseini(config, inipath)
  File "/.../tox/tox/_config.py", line 202, in __init__
    for line in reader.getlist(toxsection, "indexserver"):
  File "/.../tox/tox/_config.py", line 371, in getlist
    s = self.getdefault(section, name, None)
  File "/.../tox/tox/_config.py", line 463, in getdefault
    x = self._replace(x)
  File "/.../tox/tox/_config.py", line 570, in _replace
    return rexpattern.sub(replace_func, x)
  File "/.../tox/tox/_config.py", line 561, in _replace_match_no_quote
    return self._replace_match(match, quote=False)
  File "/.../tox/tox/_config.py", line 552, in _replace_match
    raise tox.exception.ConfigError("No support for the %s substitution type" % sub_type)
tox.ConfigError: ConfigError: No support for the homedir}/.pip/downloads/simple
local1  = file substitution type

NOTES:
I am basically using a local PyPI directory structure with all required, already downloaded packages. The corrected value is indexserver= local=file://{homedir}/.pip_downloads/simple.

For a real-world example, see:
https://github.com/behave/behave/blob/master/tox.ini
There is a also the script that builds the local package index for me.

@pytoxbot
Copy link
Author

Original comment by @e3krisztian

Re ADDITIONAL NOTE

Could you describe how the indexserver= local=file://{homedir}/.pip_downloads worked?

Could you give a minimal, but complete tox.ini exposing the problem and whatever setup needed - e.g. what exactly is in .pip_downloads?

What are the outputs of tox -r -v (both 1.4.3 and 1.5dev7)?

@pytoxbot pytoxbot mentioned this issue Sep 17, 2016
Closed
@tox-dev tox-dev locked and limited conversation to collaborators Jan 14, 2021
hukkin pushed a commit to hukkin/tox that referenced this issue Dec 4, 2022
@hpk42
Merged in itxaka/tox/fix_env_use (pull request tox-dev#169)
4e6fb6c 
Tries to fix tox-dev#99
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pytoxbot