ubuntu-latest has broken pre-installed ansible tool (please remove)

[ssbarnea]

Description
The preinstalled ansible on GHA is broken because it cannot be called from within an activated python virtualenvs. The root cause is a broken DEB package but pre-installing it is doing more damage.

Area for Triage:

Question, Bug, or Feature?:

Virtual environments affected

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04

Image version
all ubuntu images

Expected behavior

• Either not having ansible installed at all and relying on user to install it, or
• Install a working version of ansible and assure regression does not happen.

Actual behavior

ansible --version # works
vritualenv foo
source foo/bin/activate
ansible --version
(foo) root@buntu:~# ansible --version
Traceback (most recent call last):
  File "/usr/bin/ansible", line 34, in <module>
    from ansible import context
ModuleNotFoundError: No module named 'ansible'

Please note that installing ansible using pip or pipx at --user or even root level does install a working copy of ansible, one that works regardless if a virtualenv is active or not.

As a first step, please remove the broken copy.

This bug impacts the testing of Python based projects where 99% are using virtualenvs for testing, likely with tox.

[webknjaz]

I bet this problem is not specific to the Ansible package — Debian derivatives are known to ignore the existence of virtualenvs: https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e#gistcomment-3674863

[webknjaz]

Have you checked if Ubuntu's head -n /usr/bin/ansible has #!/usr/bin/python or #!/usr/bin/env python? I bet it's the latter.

[ssbarnea]

Yes, I can confirm that the broken Ubuntu one, is using #!/usr/bin/env python shebang line, which will obviously break inside activated virtualenvs.

While the env based shebang may be seen as portable and prefered for standalone scripts, it is clearly not a good idea for tools that rely on external libraries, as virtualenv activation will likely break those tools as they will endup calling another python interpreter, one that does not have access to their dependencies.

[al-cheb]

Hey, @ssbarnea
I think we can replace the ansible pkg from apt repo and install it using pipx:

    steps:
      - run: sudo apt remove ansible
      - name: ansible
        run: |
          sudo pipx install ansible-base
          virtualenv foo
          source foo/bin/activate
          ansible --version 

[ssbarnea]

I would recommend to keep the tool in PATH while doing the swap from distro ansible to pip(x)-ansible. Keep in mind that by default Ubuntu does not have %HOME/.local/bin in PATH (but other major distros do have it, like Fedora).

pipx could be one path for sure, I started using it only recently so I am not yet sure how well it works on each platform and if there are hidden surprises to be expected. My old approach was to using pip install --user.

If we install Ansible from another location but is not found in default PATH, or if activation of a virtualenv is required, we degrade the UX for some users.

If GHA team would agree to make the already documented workaround of echo "$HOME/.local/bin" >> $GITHUB_PATH implicit (making not needed anymore), everyone will benefit.

[al-cheb]

I would recommend to keep the tool in PATH while doing the swap from distro ansible to pip(x)-ansible. Keep in mind that by default Ubuntu does not have %HOME/.local/bin in PATH (but other major distros do have it, like Fedora).

I don't think it makes sense to store both versions due to the /opt/pipx_bin path takes precedence over /usr/bin/.

/home/linuxbrew/.linuxbrew/bin
/home/linuxbrew/.linuxbrew/sbin
/opt/pipx_bin
....

pipx could be one path for sure, I started using it only recently so I am not yet sure how well it works on each platform and if there are hidden surprises to be expected. My old approach was to using pip install --user.

A user and user's home directory at the end of generation process is deleted.

If we install Ansible from another location but is not found in default PATH, or if activation of a virtualenv is required, we degrade the UX for some users.

The /opt/pipx_bin is already in PATH.

[miketimofeev]

@ssbarnea just fyi: we're going to add %HOME/.local/bin to the PATH in the nearest future

[al-cheb]

@ssbarnea,We are planning to switch installation approach from apt to pipx. Could you please confirm if it works for you?

[al-cheb]

@ssbarnea , The new image with ansible from pipx has been deployed.

[drewdavies]

In switching from apt to pipx, dependencies like boto3 and botocore appear to become inaccessible to Ansible. Eg:

fatal: [default]: FAILED! => {"changed": false, "msg": "AnsibleError: An unhandled exception occurred while running the lookup plugin 'aws_ssm'. Error was a <class 'ansible.errors.AnsibleError'>, original message: botocore and boto3 are required for aws_ssm lookup."}

I was able to resolve my range of broken workflows, by injecting those two packages into the Ansible virtual environment via an additional step:

- name: Install Ansible dependencies
  run: pipx inject ansible-base boto3 botocore

[ssbarnea]

Yes, that is the price we have to pay for using pipx, I am aware of it. Mainly what pipx does is that it hides the management of the virtualenv used for installing these commands. Still, when you want to add extra dependencies, you need to do it like this.

[al-cheb]

Feel free to open the thread if you have any concerns.