`venv_cache_path` not a valid keyword argument when defining a task.virtualenv in taskflow api

[themantalope]

Apache Airflow version

2.8.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

After declaring a task.virtualenv using a decorator and the TaskFlow API, the DAG has an import error:

Broken DAG: [/opt/airflow/dags/<dag>.py] Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 437, in apply_defaults
    result = func(self, **kwargs, default_args=default_args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 794, in __init__
    raise AirflowException(
airflow.exceptions.AirflowException: Invalid arguments were passed to _PythonVirtualenvDecoratedOperator (task_id: <task name>). Invalid arguments were:
**kwargs: {'venv_cache_path': '/opt/airflow/dags/venvs/'}

What you think should happen instead?

As best I can tell per the documentation, venv_cache_path should be a valid keyword argument for the virtualenv task decorator.

How to reproduce

Please try to define a task.virtualenv with a venv_cache_path.

Operating System

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Taragolis]

Are you sure that you use Airflow 2.8.0? I've checked this simple DAG under Airflow 2.8.0 image

from airflow import DAG
from airflow.decorators import task

with DAG(dag_id="issue_36775", schedule=None, catchup=False, tags=["36775", "venv_cache_path"]):
    @task.virtualenv(venv_cache_path="/tmp/venvs", requirements=["requests"])
    def my_task(s: str):
        return "".join(reversed(s))

    my_task("foo")

And everything works fine

[2024-01-15, 08:18:59 UTC] {taskinstance.py:1957} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: issue_36775.my_task manual__2024-01-15T08:18:58.465450+00:00 [queued]>
[2024-01-15, 08:18:59 UTC] {taskinstance.py:1957} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: issue_36775.my_task manual__2024-01-15T08:18:58.465450+00:00 [queued]>
[2024-01-15, 08:18:59 UTC] {taskinstance.py:2171} INFO - Starting attempt 1 of 1
[2024-01-15, 08:18:59 UTC] {taskinstance.py:2192} INFO - Executing <Task(_PythonVirtualenvDecoratedOperator): my_task> on 2024-01-15 08:18:58.465450+00:00
[2024-01-15, 08:18:59 UTC] {standard_task_runner.py:60} INFO - Started process 334 to run task
[2024-01-15, 08:18:59 UTC] {standard_task_runner.py:87} INFO - Running: ['***', 'tasks', 'run', 'issue_36775', 'my_task', 'manual__2024-01-15T08:18:58.465450+00:00', '--job-id', '4', '--raw', '--subdir', 'DAGS_FOLDER/issue_36775.py', '--cfg-path', '/tmp/tmp7hmguvg9']
[2024-01-15, 08:18:59 UTC] {standard_task_runner.py:88} INFO - Job 4: Subtask my_task
[2024-01-15, 08:18:59 UTC] {task_command.py:423} INFO - Running <TaskInstance: issue_36775.my_task manual__2024-01-15T08:18:58.465450+00:00 [running]> on host e53269fa53b7
[2024-01-15, 08:18:59 UTC] {taskinstance.py:2481} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='issue_36775' AIRFLOW_CTX_TASK_ID='my_task' AIRFLOW_CTX_EXECUTION_DATE='2024-01-15T08:18:58.465450+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-01-15T08:18:58.465450+00:00'
[2024-01-15, 08:18:59 UTC] {python.py:662} INFO - Python virtual environment will be cached in /tmp/venvs/venv-da1df057
[2024-01-15, 08:18:59 UTC] {process_utils.py:182} INFO - Executing cmd: /usr/local/bin/python -m virtualenv /tmp/venvs/venv-da1df057 --system-site-packages --python=python
[2024-01-15, 08:18:59 UTC] {process_utils.py:186} INFO - Output:
[2024-01-15, 08:19:00 UTC] {process_utils.py:190} INFO - created virtual environment CPython3.8.18.final.0-64 in 133ms
[2024-01-15, 08:19:00 UTC] {process_utils.py:190} INFO -   creator CPython3Posix(dest=/tmp/venvs/venv-da1df057, clear=False, no_vcs_ignore=False, global=True)
[2024-01-15, 08:19:00 UTC] {process_utils.py:190} INFO -   seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/***/.local/share/virtualenv)
[2024-01-15, 08:19:00 UTC] {process_utils.py:190} INFO -     added seed packages: pip==23.3.1, setuptools==69.0.2, wheel==0.42.0
[2024-01-15, 08:19:00 UTC] {process_utils.py:190} INFO -   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[2024-01-15, 08:19:00 UTC] {process_utils.py:182} INFO - Executing cmd: /tmp/venvs/venv-da1df057/bin/pip install -r /tmp/venvs/venv-da1df057/requirements.txt
[2024-01-15, 08:19:00 UTC] {process_utils.py:186} INFO - Output:
[2024-01-15, 08:19:00 UTC] {process_utils.py:190} INFO - Requirement already satisfied: requests in /home/***/.local/lib/python3.8/site-packages (from -r /tmp/venvs/venv-da1df057/requirements.txt (line 1)) (2.31.0)
[2024-01-15, 08:19:00 UTC] {process_utils.py:190} INFO - Requirement already satisfied: charset-normalizer<4,>=2 in /home/***/.local/lib/python3.8/site-packages (from requests->-r /tmp/venvs/venv-da1df057/requirements.txt (line 1)) (3.3.2)
[2024-01-15, 08:19:00 UTC] {process_utils.py:190} INFO - Requirement already satisfied: idna<4,>=2.5 in /home/***/.local/lib/python3.8/site-packages (from requests->-r /tmp/venvs/venv-da1df057/requirements.txt (line 1)) (3.6)
[2024-01-15, 08:19:00 UTC] {process_utils.py:190} INFO - Requirement already satisfied: urllib3<3,>=1.21.1 in /home/***/.local/lib/python3.8/site-packages (from requests->-r /tmp/venvs/venv-da1df057/requirements.txt (line 1)) (1.26.18)
[2024-01-15, 08:19:00 UTC] {process_utils.py:190} INFO - Requirement already satisfied: certifi>=2017.4.17 in /home/***/.local/lib/python3.8/site-packages (from requests->-r /tmp/venvs/venv-da1df057/requirements.txt (line 1)) (2023.11.17)
[2024-01-15, 08:19:03 UTC] {process_utils.py:190} INFO - 
[2024-01-15, 08:19:03 UTC] {process_utils.py:190} INFO - [notice] A new release of pip is available: 23.3.1 -> 23.3.2
[2024-01-15, 08:19:03 UTC] {process_utils.py:190} INFO - [notice] To update, run: python -m pip install --upgrade pip
[2024-01-15, 08:19:03 UTC] {python.py:700} INFO - New Python virtual environment created in /tmp/venvs/venv-da1df057
[2024-01-15, 08:19:03 UTC] {process_utils.py:182} INFO - Executing cmd: /tmp/venvs/venv-da1df057/bin/python /tmp/venv-callvale5a19/script.py /tmp/venv-callvale5a19/script.in /tmp/venv-callvale5a19/script.out /tmp/venv-callvale5a19/string_args.txt /tmp/venv-callvale5a19/termination.log
[2024-01-15, 08:19:03 UTC] {process_utils.py:186} INFO - Output:
[2024-01-15, 08:19:04 UTC] {python.py:201} INFO - Done. Returned value was: oof
[2024-01-15, 08:19:04 UTC] {taskinstance.py:1138} INFO - Marking task as SUCCESS. dag_id=issue_36775, task_id=my_task, execution_date=20240115T081858, start_date=20240115T081859, end_date=20240115T081904
[2024-01-15, 08:19:04 UTC] {local_task_job_runner.py:234} INFO - Task exited with return code 0
[2024-01-15, 08:19:04 UTC] {taskinstance.py:3281} INFO - 0 downstream tasks scheduled from follow-on schedule check

[themantalope]

@Taragolis

Thanks for your response. I checked the docker image I was using and I was on 2.7.1. Updating to 2.8.0 resolved the issue. Thank you!