pip virtualenv creates chdir twice when using a relative path #81522

xeals · 2023-08-16T07:21:17Z

Summary

pip module creates the chdir argument twice when creating a virtual environment through venv.

Issue Type

Bug Report

Component Name

pip

Ansible Version

$ ansible --version
ansible [core 2.15.1]
  config file = /home/xeal/work/aitb-ansible/ansible.cfg
  configured module search path = ['/home/xeal/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /nix/store/a0pnfr3z94ws2vclc465wf9m41xjqsy3-python3.10-ansible-core-2.15.1/lib/python3.10/site-packages/ansible
  ansible collection location = /home/xeal/.ansible/collections:/usr/share/ansible/collections
  executable location = /nix/store/a0pnfr3z94ws2vclc465wf9m41xjqsy3-python3.10-ansible-core-2.15.1/bin/ansible
  python version = 3.10.12 (main, Jun  6 2023, 22:43:10) [GCC 12.3.0] (/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/bin/python3.10)
  jinja version = 3.1.2
  libyaml = True

Configuration

# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all
CONFIG_FILE() = /home/xeal/work/proj/ansible.cfg
DEFAULT_HOST_LIST(/home/xeal/work/proj/ansible.cfg) = ['/home/xeal/work/proj/inventory']
DEFAULT_PRIVATE_KEY_FILE(env: ANSIBLE_PRIVATE_KEY_FILE) = /home/xeal/work/proj/ansible.pem
DEFAULT_ROLES_PATH(/home/xeal/work/proj/ansible.cfg) = ['/home/xeal/work/proj/galaxy', '/home/xeal/work/proj/roles']
DEFAULT_VAULT_PASSWORD_FILE(env: ANSIBLE_VAULT_PASSWORD_FILE) = /home/xeal/work/proj/keys/vault
EDITOR(env: EDITOR) = emacs
PAGER(env: PAGER) = less

CONNECTION:
==========

paramiko_ssh:
____________
private_key_file(env: ANSIBLE_PRIVATE_KEY_FILE) = /home/xeal/work/proj/ansible.pem

ssh:
___
private_key_file(env: ANSIBLE_PRIVATE_KEY_FILE) = /home/xeal/work/proj/ansible.pem

OS / Environment

Targeting Ubuntu 18.04.05, Python 3.6

Steps to Reproduce

Run below playbook.

- hosts: localhost
  tasks:
    - name: Check out repo
      ansible.builtin.git:
        repo: https://github.com/timofurrer/try
        version: master
        dest: checkout-dir
    - name: Check out virtual environment
      ansible.builtin.pip:
        chdir: checkout-dir
        requirements: requirements.txt
        virtualenv: venv
        virtualenv_command: python3 -m venv venv

(arbitrary repository reproducing the issue)

Expected Results

Ansible creates checkout-dir/venv containing a virtual environment with packages from checkout-dir/requirements-dev.txt.

Actual Results

Ansible creates virtual environments at both checkout-dir/venv and checkout-dir/checkout-dir/venv, but only the latter contains packages from requirements.txt.

changed: [localhost] => {
    "changed": true,
    "cmd": [
        "checkout-dir/venv/bin/pip3",
        "install",
        "-r",
        "requirements.txt"
    ],
    "invocation": {
        "module_args": {
            "chdir": "checkout-dir",
            "editable": false,
            "executable": null,
            "extra_args": null,
            "name": null,
            "requirements": "requirements.txt",
            "state": "present",
            "umask": null,
            "version": null,
            "virtualenv": "venv",
            "virtualenv_command": "python3 -m venv venv",
            "virtualenv_python": null,
            "virtualenv_site_packages": false
        }
    },
    "name": null,
    "requirements": "requirements.txt",
    "state": "present",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "Collecting click (from -r requirements.txt (line 1))\n  Using cached https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl\nCollecting importlib-metadata; python_version < \"3.8\" (from click->-r requirements.txt (line 1))\n  Using cached https://files.pythonhosted.org/packages/a0/a1/b153a0a4caf7a7e3f15c2cd56c7702e2cf3d89b1b359d1f1c5e59d68f4ce/importlib_metadata-4.8.3-py3-none-any.whl\nCollecting zipp>=0.5 (from importlib-metadata; python_version < \"3.8\"->click->-r requirements.txt (line 1))\n  Using cached https://files.pythonhosted.org/packages/bd/df/d4a4974a3e3957fd1c1fa3082366d7fff6e428ddb55f074bf64876f8e8ad/zipp-3.6.0-py3-none-any.whl\nCollecting typing-extensions>=3.6.4; python_version < \"3.8\" (from importlib-metadata; python_version < \"3.8\"->click->-r requirements.txt (line 1))\n  Using cached https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fad776cca7e1704396d043f89effd3a0e/typing_extensions-4.1.1-py3-none-any.whl\nInstalling collected packages: zipp, typing-extensions, importlib-metadata, click\nSuccessfully installed click-8.0.4 importlib-metadata-4.8.3 typing-extensions-4.1.1 zipp-3.6.0\n",
    "stdout_lines": [
        "Collecting click (from -r requirements.txt (line 1))",
        "  Using cached https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl",
        "Collecting importlib-metadata; python_version < \"3.8\" (from click->-r requirements.txt (line 1))",
        "  Using cached https://files.pythonhosted.org/packages/a0/a1/b153a0a4caf7a7e3f15c2cd56c7702e2cf3d89b1b359d1f1c5e59d68f4ce/importlib_metadata-4.8.3-py3-none-any.whl",
        "Collecting zipp>=0.5 (from importlib-metadata; python_version < \"3.8\"->click->-r requirements.txt (line 1))",
        "  Using cached https://files.pythonhosted.org/packages/bd/df/d4a4974a3e3957fd1c1fa3082366d7fff6e428ddb55f074bf64876f8e8ad/zipp-3.6.0-py3-none-any.whl",
        "Collecting typing-extensions>=3.6.4; python_version < \"3.8\" (from importlib-metadata; python_version < \"3.8\"->click->-r requirements.txt (line 1))",
        "  Using cached https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fad776cca7e1704396d043f89effd3a0e/typing_extensions-4.1.1-py3-none-any.whl",
        "Installing collected packages: zipp, typing-extensions, importlib-metadata, click",
        "Successfully installed click-8.0.4 importlib-metadata-4.8.3 typing-extensions-4.1.1 zipp-3.6.0"
    ],
    "version": null,
    "virtualenv": "checkout-dir/venv"
}

Code of Conduct

I agree to follow the Ansible Code of Conduct

The text was updated successfully, but these errors were encountered:

ansibot · 2023-08-16T07:22:01Z

Files identified in the description:

lib/ansible/modules/pip.py

If these files are incorrect, please update the component name section of the description or use the component bot command.

mkrizek · 2023-08-25T12:52:20Z

pip module creates the chdir argument twice when creating a virtual environment through venv.
...

- hosts: localhost
  tasks:
    - name: Check out repo
      ansible.builtin.git:
        repo: https://github.com/timofurrer/try
        version: master
        dest: checkout-dir
    - name: Check out virtual environment
      ansible.builtin.pip:
        chdir: checkout-dir
        requirements: requirements.txt
        virtualenv: venv
        virtualenv_command: python3 -m venv venv

First part of the problem here is I think that the virtualenv is created twice because the above playbook specifies it twice. Once in virtualenv parameter and once in virtualenv_command. I think the virtualenv_command should just say python3 -m venv.

Second part of the problem is that it looks like specifying chdir as a relative path does not work which might be a bug.

As a workaround (or working solution?) this should work:

- hosts: localhost
  gather_facts: false
  tasks:
    - git:
        repo: https://github.com/timofurrer/try
        version: master
        dest: /tmp/checkout-dir

    - pip:
        chdir: /tmp/checkout-dir
        requirements: requirements.txt                                                                                                                                             
        virtualenv: venv
        virtualenv_command: python3 -m venv

xeals · 2023-08-27T23:27:22Z

First part of the problem here is I think that the virtualenv is created twice because the above playbook specifies it twice. Once in virtualenv parameter and once in virtualenv_command. I think the virtualenv_command should just say python3 -m venv.

With this in mind the behaviour seems that virtualenv is created, virtualenv_command is run to create the env, then virtualenv/bin/activate is sourced, so the command should initialise the env in the working directory (python -m venv .). Cheers.

update: it seems virtualenv is quietly provided as an argument to the command, so as you say, the . isn't required.

Second part of the problem is that it looks like specifying chdir as a relative path does not work which might be a bug.

It does work for me, and in my experience is relative to the working directory of the target (usually the home directory on a remote host). I've relied on this behaviour in most of my playbooks, whether or not it's defined.

mkrizek · 2023-08-28T11:07:31Z

First part of the problem here is I think that the virtualenv is created twice because the above playbook specifies it twice. Once in virtualenv parameter and once in virtualenv_command. I think the virtualenv_command should just say python3 -m venv.

With this in mind the behaviour seems that virtualenv is created, virtualenv_command is run to create the env, then virtualenv/bin/activate is sourced, so the command should initialise the env in the working directory (python -m venv .). Cheers.

update: it seems virtualenv is quietly provided as an argument to the command, so as you say, the . isn't required.

Second part of the problem is that it looks like specifying chdir as a relative path does not work which might be a bug.

It does work for me, and in my experience is relative to the working directory of the target (usually the home directory on a remote host). I've relied on this behaviour in most of my playbooks, whether or not it's defined.

Interesting, with chdir specified as a relative path:

- hosts: container
  gather_facts: false                                                                           
  tasks:
    - git:
        repo: https://github.com/timofurrer/try
        version: master
        dest: /checkout-dir

    - pip:
        chdir: checkout-dir
        requirements: requirements.txt
        virtualenv: venv
        virtualenv_command: python3 -m venv

I get:

PLAY [container] ******************************************************************************************************************************************************************************************************************************************

TASK [git] ************************************************************************************************************************************************************************************************************************************************
changed: [container]

TASK [pip] ************************************************************************************************************************************************************************************************************************************************
fatal: [container]: FAILED! => {"changed": false, "msg": "Unable to find pip in the virtualenv, checkout-dir/venv, under any of these names: pip3, pip. Make sure pip is present in the virtualenv."}

PLAY RECAP ************************************************************************************************************************************************************************************************************************************************
container                  : ok=1    changed=1    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

because the virtualenv is created in checkout-dir/checkout-dir/venv/ instead of checkout-dir/venv.

The following POC patch seems to fix the problem:

diff --git a/lib/ansible/modules/pip.py b/lib/ansible/modules/pip.py
index 9a3daf61b80..da7a33cd19d 100644
--- a/lib/ansible/modules/pip.py
+++ b/lib/ansible/modules/pip.py
@@ -686,8 +686,10 @@ def main():
     env = module.params['virtualenv']
 
     venv_created = False
-    if env and chdir:
-        env = os.path.join(chdir, env)
+    if env:
+        if chdir:
+            env = os.path.join(chdir, env)
+        env = os.path.abspath(env)
 
     if umask and not isinstance(umask, int):
         try:

ansibot added bug This issue/PR relates to a bug. needs_triage Needs a first human triage before being processed. affects_2.15 module This issue/PR relates to a module. labels Aug 16, 2023

s-hertel added needs_verified This issue needs to be verified/reproduced by maintainer and removed needs_triage Needs a first human triage before being processed. labels Aug 17, 2023

mkrizek added verified This issue has been verified/reproduced by maintainer and removed needs_verified This issue needs to be verified/reproduced by maintainer labels Aug 25, 2023

s-hertel added P3 Priority 3 - Approved, No Time Limitation easyfix This issue is considered easy to fix by aspiring contributors. and removed P3 Priority 3 - Approved, No Time Limitation labels Apr 17, 2024

yctomwang linked a pull request May 3, 2024 that will close this issue

Fix unable to find pip in the virtualenv when using relative path #83189

Open

ansibot added the has_pr This issue has an associated PR. label May 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip virtualenv creates chdir twice when using a relative path #81522

pip virtualenv creates chdir twice when using a relative path #81522

xeals commented Aug 16, 2023 •

edited

ansibot commented Aug 16, 2023

mkrizek commented Aug 25, 2023

xeals commented Aug 27, 2023 •

edited

mkrizek commented Aug 28, 2023

pip virtualenv creates chdir twice when using a relative path #81522

pip virtualenv creates chdir twice when using a relative path #81522

Comments

xeals commented Aug 16, 2023 • edited

Summary

Issue Type

Component Name

Ansible Version

Configuration

OS / Environment

Steps to Reproduce

Expected Results

Actual Results

Code of Conduct

ansibot commented Aug 16, 2023

mkrizek commented Aug 25, 2023

xeals commented Aug 27, 2023 • edited

mkrizek commented Aug 28, 2023

xeals commented Aug 16, 2023 •

edited

xeals commented Aug 27, 2023 •

edited