Cannot run ansible-test sanity behind a proxy

[df-cgdm]

Summary

When I try to run ansible-test it tries to download files from ansible-ci-files.s3.amazonaws.com/ansible-test.
The problem is that I'm behind a proxy
Even by setting env http_proxy it's not working

Issue Type

Bug Report

Component Name

bin/ansible-test

Ansible Version

$ ansible --version
ansible [core 2.12.3]
  config file = /home/adm-dfournout/.ansible.cfg
  configured module search path = ['/home/adm-dfournout/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/adm-dfournout/.local/lib/python3.9/site-packages/ansible
  ansible collection location = /home/adm-dfournout/dev/ansible/collections
  executable location = /home/adm-dfournout/.local/bin/ansible
  python version = 3.9.10 (main, Mar 16 2022, 17:21:43) [GCC 8.3.0]
  jinja version = 3.0.3
  libyaml = True

Configuration

$ ansible-config dump --only-changed
COLLECTIONS_PATHS(/home/adm-dfournout/.ansible.cfg) = ['/home/adm-dfournout/dev/ansible/collections']
DEFAULT_REMOTE_USER(/home/adm-dfournout/.ansible.cfg) = adm-dfournout@adm.corp
DEFAULT_TIMEOUT(/home/adm-dfournout/.ansible.cfg) = 40
DEFAULT_VAULT_PASSWORD_FILE(/home/adm-dfournout/.ansible.cfg) = /home/adm-dfournout/.vault_password
HOST_KEY_CHECKING(/home/adm-dfournout/.ansible.cfg) = False
INTERPRETER_PYTHON(/home/adm-dfournout/.ansible.cfg) = auto

OS / Environment

debian

Steps to Reproduce

''''

ansible-test sanity
Running sanity test "action-plugin-docs"
Running sanity test "ansible-doc"
Downloading pip 21.3.1 bootstrap script: https://ansible-ci-files.s3.amazonaws.com/ansible-test/get-pip-21.3.1.py
.....
''''

Expected Results

I expect to be able to run ansible-test either by setting the proxy.
I should also be able to run ansible-test without connection to internet (I use it likes that in version 2.10)

Actual Results

Running sanity test "action-plugin-docs"
Running sanity test "ansible-doc"
Downloading pip 21.3.1 bootstrap script: https://ansible-ci-files.s3.amazonaws.com/ansible-test/get-pip-21.3.1.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.9/http/client.py", line 1447, in connect
    super().connect()
  File "/usr/local/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/usr/local/lib/python3.9/socket.py", line 844, in create_connection
    raise err
  File "/usr/local/lib/python3.9/socket.py", line 832, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 322, in <module>
  File "<stdin>", line 72, in main
  File "<stdin>", line 94, in bootstrap
  File "<stdin>", line 199, in download_file
  File "/usr/local/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/local/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/local/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>
ERROR: Python 3.9 at "/home/adm-dfournout/.ansible/test/venv/sanity.ansible-doc/3.9/3a197293/bin/python" does not have pip available.

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description:

• bin/ansible-test

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[jborean93]

This is most likely because ansible-test strips out env vars in the code it is running to ensure a clean slate. Will have to add http_proxy and the other relevant proxy vars to the ones to pass through. The code that defines what is passed through is at

ansible/test/lib/ansible_test/_internal/util.py

Lines 361 to 378 in 7cb581e

	optional = (
	'LD_LIBRARY_PATH',
	'SSH_AUTH_SOCK',
	# MacOS High Sierra Compatibility
	# http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html
	# Example configuration for macOS:
	# export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
	'OBJC_DISABLE_INITIALIZE_FORK_SAFETY',
	'ANSIBLE_KEEP_REMOTE_FILES',
	# MacOS Homebrew Compatibility
	# https://cryptography.io/en/latest/installation/#building-cryptography-on-macos
	# This may also be required to install pyyaml with libyaml support when installed in non-standard locations.
	# Example configuration for brew on macOS:
	# export LDFLAGS="-L$(brew --prefix openssl)/lib/ -L$(brew --prefix libyaml)/lib/"
	# export CFLAGS="-I$(brew --prefix openssl)/include/ -I$(brew --prefix libyaml)/include/"
	'LDFLAGS',
	'CFLAGS',
	)

.

[mkot02]

I have the same problem. It looks like it changed between ansible-core 2.11 and 2.12.

When doing this with python3.6 and ansible-core 2.11.x sanity test passes, but when I use python3.8+ and ansible-core 2.12 I'm getting the same error.

[mattclay]

@df-cgdm @mkot02 I'm looking at options for dealing with this. How do you have pip configured to use a proxy in your environment?

[mkot02]

@mattclay I have global.proxy and global.cert options set in /etc/pip.conf file as well as environment variables for proxy (lowercase and uppercase).

[mattclay]

@df-cgdm @mkot02 As a temporary work-around, can you manually download the pip bootstrapping scripts before running ansible-test sanity?

The file(s) to download will depend on which Python version(s) are being used to run the sanity tests:

For Python 3.6+:

https://ci-files.testing.ansible.com/ansible-test/get-pip-21.3.1.py -> ~/.ansible/test/cache/get_pip_21_3_1.py

For Python 2.7 and 3.5:

https://ci-files.testing.ansible.com/ansible-test/get-pip-20.3.4.py -> ~/.ansible/test/cache/get_pip_20_3_4.py

For Python 2.6:

https://ci-files.testing.ansible.com/ansible-test/get-pip-9.0.3.py -> ~/.ansible/test/cache/get_pip_9_0_3.py

NOTE: The URLs above are from the devel version of ansible-test, but they're backed by the same S3 bucket which uses the URL format, which is used in ansible-core 2.12.3: https://ansible-ci-files.s3.amazonaws.com/ansible-test/get-pip-{version}.py

[ketank1000]

Is there any option to skip this installation and use local environment ?

[mattclay]

@ketank1000 I recommend using the --docker default option. If you're not able to use that container, have you tried the work-around I described in my previous comment?

[ketank1000]

@mattclay I actually have cicd pipline and need to run the ansible-test command in docker itself, I have everything installed in docker container and want to avoid any further installation and use local environment.

Do we have any option like that ?

[mattclay]

@ketank1000 If you include the docker cli in your container image and expose the docker socket to the running container, then you won't need to install anything to use the --docker default option. The default container includes all the packages needed to run the sanity tests. That is what we do in our CI pipelines.

If that isn't an option for you, you could build your container using the default container as a base image -- although you'd need to make sure to use the correct image for the version of ansible-core you were using.

If neither of those work for you, then I can provide some tips on how to include the necessary requirements in your own container, but I don't recommend this approach. It requires the most setup and maintenance, since you'll need to keep in sync with the requirements of ansible-core as they change. Let me know if you'd like to try this approach.

[felipempda]

I am facing the same issue and @mattclay workaround worked for me.
But the point here seems to be whether or not we could have a way of bootstraping pip without the get-pip script.
It looks like all virtualenvs created by sanity are using option "--without-pip" as of #77028

In core-2.11.0 that was not the case and if you managed to install pip by a proxy future virtualenvs created would already have pip installed and that problem would not arrive. Even though that seems to create issues in some cases, couldn't we have a way to decide that via an argument ?

Ex:
You can check this by cleaning up generated virtualenv and running ansible-test with debug -vvv:

$ ansible-test sanity --test ansible-doc -vvv
Running sanity test "ansible-doc"
...
Run command: /usr/bin/python3.8 -m venv --without-pip ~/.ansible/test/venv/sanity.ansible-doc/3.8/948d72e0

This seems to be defined in the code here but there is no way to decide that via an argument I think:

test/lib/ansible_test/_internal/venv.py
def create_virtual_environment(args,  # type: EnvironmentConfig
                               python,  # type: PythonConfig
                               path,  # type: str
                               system_site_packages=False,  # type: bool
                               pip=False,  # type: bool <<< here  
                               ):  # type: (...) -> bool`

If I manage to change pip to True here and If I create an "empty" get-script (AKA not using it) it seems to work:
echo "" > ~/.ansible/test/cache/get_pip_21_3_1.py

What do you think? Just my two cents.

[mkot02]

I have the same situation when using --docker option - ansible-test doesn't pass proxy settings and podman pull command fails because it cannot connect to the registry.

[unkaputtbar112]

Since I've searched a bit to find a solution and find it quite of strange, when I got the error message and even a link to this issue here. I may as well leave it for sharing, how i've solved it as workaround.

Here is what I did, downloaded the get-pip21.3.1.py and modified this line and made sure it is available in ~/.ansible/test/cache/get_pip_21_3_1.py:
return ["install", "--upgrade", "--force-reinstall"] + args
to
return ["install", "--upgrade", "--force-reinstall", "--proxy", "http://proxyurl:8080"] + args
(you could also do with args.append or args.extend, I just did it quick & dirty)

That fixed part of it, where it bootstraps the pip, unfortunately he still wants to install a lot of stuff after that for the virtualenv it builds, so I added a line to my .gitlab-ci.yml as well:
pip3 config set global.proxy http://proxyurl:8080

After doing both, it just went through. May it be of some help to somebody else.