test_ansible_controller fails to use pip to install packages on Lunar

[blackboxsw]

Bug report

Seen in jenkins daily integration logs on tip of main, affects cloud-init 23.1 and later.

Steps to reproduce the problem

Environment details

• Cloud-init version: 23.1
• Operating System Distribution: Ubuntu Lunar
• Cloud provider, platform or installer type: lxd_vm

cloud-init logs

Seen @ https://jenkins.canonical.com/server-team/view/cloud-init/job/cloud-init-integration-lunar-lxd_vm/119/testReport/junit/tests.integration_tests.modules/test_ansible/test_ansible_controller/

excerpt

"'/usr/lib/python3/dist-packages/cloudinit/config/cc_ansible.py'>) failed\n"
 'Traceback (most recent call last):\n'
 '  File "/usr/lib/python3/dist-packages/cloudinit/config/modules.py", line '
 '256, in _run_modules\n'
 '    ran, _r = cc.run(\n'
 '              ^^^^^^^\n'
 '  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 67, in run\n'
 '    return self._runners.run(name, functor, args, freq, clear_on_fail)\n'
 '           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
 '  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 172, in '
 'run\n'
 '    results = functor(**args)\n'
 '              ^^^^^^^^^^^^^^^\n'
 '  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ansible.py", line '
 '179, in handle\n'
 '    ansible.install(package_name)\n'
 '  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ansible.py", line '
 '142, in install\n'
 '    self.do_as([*cmd, "--upgrade", "pip"])\n'
 '  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ansible.py", line '
 '99, in do_as\n'
 '    return self.distro.do_as(command, self.run_user, **kwargs)\n'
 '           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
 '  File "/usr/lib/python3/dist-packages/cloudinit/distros/__init__.py", line '
 '1003, in do_as\n'
 '    return subp.subp(\n'
 '           ^^^^^^^^^^\n'
 '  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 335, in '
 'subp\n'
 '    raise ProcessExecutionError(\n'
 'cloudinit.subp.ProcessExecutionError: Unexpected error while running '
 'command.\n'
 "Command: ['su', '-', 'ansible', '-c', 'env PATH=$PATH /usr/bin/python3 -m "
 "pip install --user --upgrade pip']\n"
 'Exit code: 1\n'
 'Reason: -\n'
 'Stdout: \n'
 'Stderr: error: externally-managed-environment\n'
 '        \n'
 '        × This environment is externally managed\n'
 '        ╰─> To install Python packages system-wide, try apt install\n'
 '            python3-xyz, where xyz is the package you are trying to\n'
 '            install.\n'
 '            \n'
 '            If you wish to install a non-Debian-packaged Python package,\n'
 '            create a virtual environment using python3 -m venv '
 'path/to/venv.\n'
 '            Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make\n'
 '            sure you have python3-full installed.\n'
 '            \n'
 '            If you wish to install a non-Debian packaged Python '
 'application,\n'
 '            it may be easiest to use pipx install xyz, which will manage a\n'
 '            virtual environment for you. Make sure you have pipx installed.\n'
 '            \n'
 '            See /usr/share/doc/python3.11/README.venv for more information.\n'
 '        \n'
 '        note: If you believe this is a mistake, please contact your Python '
 'installation or OS distribution provider. You can override this, at the risk '
 'of breaking your Python installation or OS, by passing '
 '--break-system-packages.\n'
 '        hint: See PEP 668 for the detailed specification.\n'

ansible-controller-test.cloud-init.log