ansible-test - prefer venv over virtualenv on Python 3

[samdoran]

SUMMARY

Also pin virtualenv to 16.7.10 for older Mac OS X systems. This was the version being installed anway with the previous constraint (<20).

On systems with Python 3, now prefer venv over virtualenv. Test to see if venv is functional since some systems have a non-functional venv installation (such as Debian).

Supersedes #72778
Fixes: #72738

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

changelogs/fragments/ansible-test-venv-virtualenv-fallback.yml
test/lib/ansible_test/_data/injector/virtualenv-isolated.sh
test/lib/ansible_test/_data/injector/virtualenv.sh
test/lib/ansible_test/_data/setup/remote.sh

[samdoran]

@felixfontein @dmsimard Can you try out this change and see if it resolves the problems you were having with virtualenv.sh?

[webknjaz]

@samdoran what is the rationale for this change? Third-party virtualenv is often better for py3 too: it's faster starting v20.

[mattclay]

@webknjaz ansible-test favors venv over virtualenv on Python 3 already. This change just updates the virtualenv scripts to follow the same behavior. Using venv avoids additional requirements needed by virtualenv>=20 which complicates management of the test environment.

[webknjaz]

Oh, I thought it'd be nice to make use of the new plugin APIs virtualenv now provides one day. Like having custom seeders.

[felixfontein]

@samdoran seems to work, at least I was able to use your branch and remove the installation of virtualenv in community.hashi_vault and CI still passes: ansible-collections/community.hashi_vault#30

[samdoran]

@webknjaz For Python 3.9, we need to move to a new version of virtualenv (>20) since <20 was failing tests in Python 3.9. virtualenv >20 brings in distlib, filelock, six, and appdirs as deps whereas virtualenv <20 had no deps. Additional deps bring in more potential instability (we'd have to pin more deps and update them as things break).

All that is why we would rather just use venv if it is available and get away from installing virtualenv during remote instance creation.

[felixfontein]

@samdoran any chance of this being backported to stable-2.9 and stable-2.10? (We'd need that to get rid of the hack :) )

[samdoran]

I don't think so since this is a change in behavior.

[mattclay]

@felixfontein @samdoran Perhaps we could do one of the following to improve the transition while maintaining backwards compatibility?

1. Require setting an environment variable to opt-in to the new behavior. This would only be needed for the backports.
2. Provide another version of the scripts (such as a _v2 suffixed) that would provide the new behavior. This would be needed in all supported branches.

I think the first option is the most appealing, since it wouldn't require a separate script and the additional logic would only be needed for the backports.

[mattclay]

The more I think about this, the more I'm wondering if there really is much of a backwards compatibility concern here. Yes, we're using a different tool to create the virtual environment, but it should provide the same behavior in the end.

If a virtual environment was created before, it should still be. If it was failing to create a virtual environment before, then the test would already be failing -- making it pass shouldn't be a problem.

[felixfontein]

As long as it suffices to set the env variable in the test scripts (and not when calling ansible-test), that's fine for me. But your new argument sounds reasonable to me as well.

[mattclay]

@felixfontein I've opened backports for stable-2.10 and stable-2.9:

• Support venv in ansible-test virtualenv scripts #73163
• Support venv in ansible-test virtualenv scripts #73164

After discussing with @relrod we decided to go with opt-in behavior, just to be safe. To opt-in to the new behavior in those branches, tests will need to set the ANSIBLE_TEST_PREFER_VENV environment variable. So something like this would work:

export ANSIBLE_TEST_PREFER_VENV=1
source virtualenv.sh

Whereas before the tests simply used:

source virtualenv.sh

This should allow a single test configuration to work across 2.9, 2.10 and devel once the backports are merged.

[felixfontein]

@mattclay great, thanks! I've tried it out in ansible-collections/community.hashi_vault#45, works perfectly!

[mattclay]

@felixfontein Looks good. I recommend including a comment when setting the env var that it is only needed while supporting 2.9 and 2.10, so that it will be more obvious in the future when it can be removed.