Serialization issues with external_python decorator

[Snaptonis]

Hey everybody,

I started changing my dags to use @task.external_python().
I am still running into issues and I don't know what to do anymore.

A couple dags fail with:
PicklingError: Can't pickle <function get_connection at 0x7f105f947110>: it's not the same object as tasks.connections.get_connection.
Another fails with:
PicklingError: Can't pickle local object <function my_dag.<locals>.get_config at 0x7f105f94ba00>.

I have a mixture of normal tasks and external_python tasks.
I think the the problems start when I have expanded tasks where the result (so the xcom) is used in another task but I'm not certain.

Airflow Version: 3.3.0

If you need more infos I'm happy to provide.

[christianalberto]

Here is the straight-to-the-point explanation: when you use @task.external_python, Airflow tries to pickle your function to send it over to a separate Python environment. If the function tries to grab variables, helpers, or imports from outside its scope, pickling breaks.

To fix it, treat your task as a self-contained box. Everything it needs must live inside the function.

❌ Code that fails

from tasks.connections import get_connection  # ❌ Import outside task

def my_dag():

    def get_config():  # ❌ Local nested function outside task
        return {"env": "prod"}

    @task.external_python(python="/path/to/venv/bin/python")
    def my_task():
        cfg = get_config()  # 💥 Fails: cannot pickle local function
        conn = get_connection()  # 💥 Fails: module import mismatch across venvs

✅ Code fixed

def my_dag():

    @task.external_python(python="/path/to/venv/bin/python")
    def my_task():
        # ✅ Imports go INSIDE the task
        from tasks.connections import get_connection

        # ✅ Helper logic goes INSIDE the task
        cfg = {"env": "prod"}
        conn = get_connection()

3 golden rules to keep in mind:

1. Imports inside: Put any imports needed by the task directly inside the decorated function.
2. No nested functions: Don't reference helper functions defined inside my_dag().
3. Simple XComs: If you use .expand() or pass data between tasks, stick to plain data structures (strings, numbers, dicts, lists)—avoid passing complex custom objects or classes.

Moving the imports and helper logic inside the function will resolve both pickling errors right away.