Fix tests that require isolation via virtualenv #239

abravalheri · 2019-07-15T21:36:24Z

This PR attempts to solve #238.
In order to do so, it replaces pytest-virtualenv dependency by custom code based on stdlib's venv.
To workaround the apparent inability of venv to install pip when running from inside another venv, we use the global python3 available.

abravalheri · 2019-07-17T10:33:37Z

.travis.yml

@@ -7,13 +7,17 @@ matrix:
  fast_finish: true
  include:
    - os: linux
+      dist: xenial


For some reason, travis insists in using an old version of ubuntu, and this version have problems using apt (gpg keys issue)

abravalheri · 2019-07-17T15:44:38Z

I think I am happy with the code now. Sorry if I took me a while, every time we have a Travis issue, the process in painfully iterative.

To be fair, there are a few more things that this PR do:

Switch to Travis built-in way of installing packages instead of manually using apt or brew
Remove duplication for the system tests (e.g., before: a lot of run functions around, now: everything goes to the same run) and centralize helper functions/fixtures in a single place.
Remove test dependencies on pytest-shutils and pytest-virtualenv
Use stdlib's venv module to isolate tests
Better run helper (with better support to env and verbose
Remove workaround needed to make Pipenv (and other dotenv programs) work with pytest-virtualenv
Remove tests to ensure requirements.txt is installed.
Remove some unnecessary things since now we are using Python > 3.4

Unfortunately this PR also removes the ability of using coverage inside the venvs used during tests.

(P.S.: The falling test in Travis happens because we are trying to use Python 3.5 but we require 3.6 minimum)

coveralls · 2019-07-26T00:38:20Z

Pull Request Test Coverage Report for Build 1718

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 95.643%

Totals
Change from base Build 1701:	0.0%
Covered Lines:	1135
Relevant Lines:	1177

💛 - Coveralls

Python3 has the `venv` module built-in. By creating a custom fixture using it, we have more control and can address the limitations of the external package

This is a workaround instead of usinv `venv` python API, since pip is never installed when the running python is already inside a virtual env Unfortunatelly running with coverage is not implemented

Since we are transitioning versions, any outdated extension will cause the tests to break by introducing incompatible version ranges for PyScaffold

… since it is more or less a system test anyway (end-to-end test)

… not installed by default in ubuntu

… so we avoid SSL errors

… to check if the SSL error goes away...

… in an attempt to solve the SSL errors

… in yet other attempt to solve pip SSL errors.

… explicitly, since the other attempts to use env vars and a pip.conf/pip.ini file do not seem to solve the SSL error in Windows [CI]

… so we can understand what else is going wrong

… because it seems that the SSL error in windows is not going anywhere... (using the trusted-host flag, the associated env var or pip.ini in windows, is enough to make pip work, however, `setup.py` seems to ignore all of that and raise SSL errors)

Since get-pip does not solve the problem with the SSL certificates, this commit returns to use `python -m pip install --upgrade pip`, because it should be faster. Additionally, it adds certifi, since a comment in pypa/pip#4156 points that setuptools uses it to solve certificates...

… in system tests, as an attempt to solve of the errors described in pyscaffold#244

… the env var in `.cirrus.yml` should be enough, it is not necessary to repeat it everywhere.

… in system tests

… in system tests, with a very naive conversion function

… for system tests. This commit improves the previous one by removing the windows specific path workarounds. Instead of comparing the path that was found by `shell which` with the venv, call `python which` restricting the search to the venv.

… in system tests. This way Python 3.5 can also be included.

… for xfail

[system tests]

FlorianWilhelm · 2019-10-16T19:12:39Z

@abravalheri, thanks for this PR. I ported most of it to version 3.2.3 and the current v4.0.x branch. I stayed with pytest-virtualenv and it seems to works. Let me know if you want to have something else merged.

Going to Cirrus was a great idea btw. My suggestion would be to have a version 3.3 first with --cirrus and also update pyscaffoldext-custom-extension to use that instead of --travis. Also, all current extension should be updated first. After that, we could go on working on 4.0.

You can just close this PR if you think everything is ported.

abravalheri requested a review from FlorianWilhelm July 15, 2019 21:36

abravalheri force-pushed the fix-venv-tests branch 4 times, most recently from 32949c8 to 1254e7a Compare July 15, 2019 23:16

abravalheri commented Jul 17, 2019

View reviewed changes

abravalheri mentioned this pull request Jul 17, 2019

test_install.py failing and virtualenv issues #238

Closed

abravalheri force-pushed the fix-venv-tests branch 4 times, most recently from 9fecc98 to de289c2 Compare July 17, 2019 14:49

abravalheri mentioned this pull request Jul 18, 2019

Allow inplace updates #138

Closed

abravalheri force-pushed the fix-venv-tests branch from 353aaa0 to d1758ce Compare July 26, 2019 00:20

abravalheri force-pushed the fix-venv-tests branch 6 times, most recently from 4265747 to 1b05f12 Compare July 27, 2019 01:10

abravalheri added 9 commits July 27, 2019 11:14

Attempt to replace pytest-virtualenv with buitin

0e3923e

Python3 has the `venv` module built-in. By creating a custom fixture using it, we have more control and can address the limitations of the external package

Use global python to create the venv

4c00067

This is a workaround instead of usinv `venv` python API, since pip is never installed when the running python is already inside a virtual env Unfortunatelly running with coverage is not implemented

Avoid installing extensions for now

87b102e

Since we are transitioning versions, any outdated extension will cause the tests to break by introducing incompatible version ranges for PyScaffold

Move test_install to the system folder

80f0bb7

… since it is more or less a system test anyway (end-to-end test)

Remove debugging statements

65d9e2f

Attempts to solve travis issue with pytest3-venv

2e7945d

… not installed by default in ubuntu

Attempt to use conda's python in travis

cfc6c51

Attempt to use the same version of python for venv

5696fd7

Force exact version of python for venv in tests

6032c1c

Workaround shlex behaviour in Windows for tests

751b3d2

abravalheri force-pushed the fix-venv-tests branch from fbd463a to fd50ca8 Compare July 27, 2019 11:03

Add trusted hosts for windows tests

981d59a

… so we avoid SSL errors

abravalheri force-pushed the fix-venv-tests branch 2 times, most recently from f09f785 to ea40b07 Compare July 27, 2019 12:47

Force pip update on Windows (system tests)

cac4fd4

… to check if the SSL error goes away...

abravalheri force-pushed the fix-venv-tests branch from ea40b07 to cac4fd4 Compare July 27, 2019 18:16

Add a pip.conf to Windows [CI]

34aec6a

… in an attempt to solve the SSL errors

abravalheri force-pushed the fix-venv-tests branch from 0db031c to 34aec6a Compare July 27, 2019 20:13

abravalheri added 4 commits July 27, 2019 21:33

Add PIP_CONFIG_FILE env var to Windows [CI]

2249acb

… in yet other attempt to solve pip SSL errors.

Pass trusted-host flag to pip in system tests

691af69

… explicitly, since the other attempts to use env vars and a pip.conf/pip.ini file do not seem to solve the SSL error in Windows [CI]

Increase verbosity in system tests

6315757

… so we can understand what else is going wrong

abravalheri mentioned this pull request Jul 28, 2019

Some tests failing on Windows #244

Closed

abravalheri added 2 commits July 28, 2019 02:45

Use bdist_dumb --relative instead of bdist

344d61b

… in system tests, as an attempt to solve of the errors described in pyscaffold#244

abravalheri force-pushed the fix-venv-tests branch from 2e9951e to 344d61b Compare July 28, 2019 02:08

abravalheri added 7 commits July 28, 2019 03:18

Simplify system tests by removing trusted-host

150cc38

… the env var in `.cirrus.yml` should be enough, it is not necessary to repeat it everywhere.

Add xfail marks to windows edge cases

7d82a20

… in system tests

Address msys/mingw vs native windows path diffs

86c88d5

… in system tests, with a very naive conversion function

Simplify verificationof exec inside venv

d8490ea

… for system tests. This commit improves the previous one by removing the windows specific path workarounds. Instead of comparing the path that was found by `shell which` with the venv, call `python which` restricting the search to the venv.

Replace ModuleNotFoundError with ImportError

1dc7ae7

… in system tests. This way Python 3.5 can also be included.

Consider GitNoConfigured exception on test_info

adcfb55

… for xfail

Avoid windows errors on teardown of venvs

245b751

[system tests]

This was referenced Jul 28, 2019

Potential fix for #238 #246

Closed

Update #246 with changes from #239 #247

Closed

FlorianWilhelm force-pushed the v4.0.x branch from 2fb1533 to 9484068 Compare October 12, 2019 19:43

abravalheri mentioned this pull request Oct 23, 2019

Use sys.executable in tests #254

Merged

FlorianWilhelm closed this Nov 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix tests that require isolation via virtualenv #239

Fix tests that require isolation via virtualenv #239

abravalheri commented Jul 15, 2019

abravalheri Jul 17, 2019

abravalheri commented Jul 17, 2019

coveralls commented Jul 26, 2019 •

edited

FlorianWilhelm commented Oct 16, 2019

Fix tests that require isolation via virtualenv #239

Fix tests that require isolation via virtualenv #239

Conversation

abravalheri commented Jul 15, 2019

abravalheri Jul 17, 2019

Choose a reason for hiding this comment

abravalheri commented Jul 17, 2019

coveralls commented Jul 26, 2019 • edited

Pull Request Test Coverage Report for Build 1718

💛 - Coveralls

FlorianWilhelm commented Oct 16, 2019

coveralls commented Jul 26, 2019 •

edited