Airflow cli response times

[static-moonlight]

The Airflow cli feels really slow. Is it normal that even the airflow command to print the help, takes a lot of time? Is there a way to speed that up?

$ time airflow -h
[...]
real    0m8.119s
user    0m7.744s
sys     0m0.270s

[starkmarkus]

I think the traceback shows something slightly different: it does not necessarily mean that airflow --help is already talking to the database. It looks like Airflow initializes its settings during import and tries to parse the configured sql_alchemy_conn in order to derive the async SQLAlchemy URI.

The important part seems to be this:

SQL_ALCHEMY_CONN_ASYNC = _get_async_conn_uri_from_sync(sync_uri=SQL_ALCHEMY_CONN)
...
scheme, rest = sync_uri.split(":", maxsplit=1)
ValueError: not enough values to unpack

So I would first check whether AIRFLOW__DATABASE__SQL_ALCHEMY_CONN or the corresponding airflow.cfg value is set to something invalid in that test environment. For example, an empty value, placeholder, secret name, or otherwise malformed value without a : would explain this error.

A quick check could be:

airflow config get-value database sql_alchemy_conn
env | grep AIRFLOW__DATABASE__SQL_ALCHEMY_CONN

Or, if the CLI crashes too early:

python - <<'PY'
import os
print(repr(os.environ.get("AIRFLOW__DATABASE__SQL_ALCHEMY_CONN")))
PY

For a simple local/test setup, you could also try setting it explicitly to a valid SQLite URL just to compare startup behavior:

export AIRFLOW__DATABASE__SQL_ALCHEMY_CONN='sqlite:////tmp/airflow.db'
time airflow --help

So my current guess would be:

• airflow --help does import and initialize quite a bit of Airflow, so it is not expected to be instant.
• This specific traceback is probably caused by an invalid database URI configuration, not by an actual failed database connection.
• The slow startup may still be related to the environment, filesystem, installed packages, or import overhead, but I would verify the DB URI first before drawing conclusions from this run.

[static-moonlight]

Based on your information I suspected the python environment itself and potentially the available computing resources (our test environment is not that good, but it's what I have available).

I'm using the off-the-shelf container image (docker.io/apache/airflow:3.2.2), which has a ton of providers already bundled. Roughly 30 of them, from which I might need a third or so. But since providers seems to be lazy loaded by default, and I didn't change that, I don't really think it's the providers.

As I said, our dedicated testing hardware is a bit slow, especially in the IO and networking area. On a hunch I tried to execute a simple Airflow command with no database configuration.

got ya ...

$ time airflow --help
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 3, in <module>
    from airflow.__main__ import main
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/__init__.py", line 81, in <module>
    settings.initialize()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/settings.py", line 801, in initialize
    configure_vars()
    ~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/settings.py", line 262, in configure_vars
    SQL_ALCHEMY_CONN_ASYNC = _get_async_conn_uri_from_sync(sync_uri=SQL_ALCHEMY_CONN)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/settings.py", line 243, in _get_async_conn_uri_from_sync
    scheme, rest = sync_uri.split(":", maxsplit=1)
    ^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)

real    0m4.578s
user    0m4.391s
sys     0m0.145

It looks like as if the airflow CLI command is always trying to talk to the database, even for a simple --help, and starts crying when it fails :) And with slow IO in our testing environment, I think the problem might be in that direction.

It would be interesting though why Airflow is ALWAYS trying to talk to the database, even for that. For everything else it kind of makes sense, since the database seems to be used as some kind of middleware and service registry as well.

[starkmarkus]

I think the traceback shows something slightly different: it does not necessarily mean that airflow --help is already talking to the database. It looks like Airflow initializes its settings during import and tries to parse the configured sql_alchemy_conn in order to derive the async SQLAlchemy URI.

The important part seems to be this:

SQL_ALCHEMY_CONN_ASYNC = _get_async_conn_uri_from_sync(sync_uri=SQL_ALCHEMY_CONN)
...
scheme, rest = sync_uri.split(":", maxsplit=1)
ValueError: not enough values to unpack

So I would first check whether AIRFLOW__DATABASE__SQL_ALCHEMY_CONN or the corresponding airflow.cfg value is set to something invalid in that test environment. For example, an empty value, placeholder, secret name, or otherwise malformed value without a : would explain this error.

A quick check could be:

airflow config get-value database sql_alchemy_conn
env | grep AIRFLOW__DATABASE__SQL_ALCHEMY_CONN

or, if the CLI crashes too early:

python - <<'PY'
import os
print(repr(os.environ.get("AIRFLOW__DATABASE__SQL_ALCHEMY_CONN")))
PY

For a simple local/test setup, you could also try setting it explicitly to a valid SQLite URL just to compare startup behavior:

export AIRFLOW__DATABASE__SQL_ALCHEMY_CONN='sqlite:////tmp/airflow.db'
time airflow --help

So my current guess would be:

1. airflow --help does import and initialize quite a bit of Airflow, so it is not expected to be instant.
2. But this specific traceback is probably caused by an invalid database URI configuration, not by an actual failed database connection.
3. The slow startup may still be related to the environment, filesystem, installed packages, or import overhead, but I would fix/verify the DB URI first before drawing conclusions from this run.

[GayathriSrividya]

Hi everyone, I looked into this and wanted to share a structured summary since this can be confusing at first glance.

What is happening

• The traceback with ValueError: not enough values to unpack is very likely happening during startup configuration parsing, not during an actual database connection attempt.
• Airflow initializes settings early, and while deriving the async SQLAlchemy URI from the configured sync URI, a malformed value can fail fast.
• That means a slow airflow --help and an invalid sql_alchemy_conn can overlap, making the root cause look like “database is slow” when the immediate failure is actually config parsing.

What to check first

• Confirm the effective value of AIRFLOW__DATABASE__SQL_ALCHEMY_CONN in the environment where the command is executed.
• Confirm database.sql_alchemy_conn in airflow.cfg for that same runtime context.
• Look for common bad values:

1. empty string
2. placeholder text
3. secret reference string instead of a URI
4. value missing a colon separator

Quick sanity test

• Set a known-good local value temporarily (for comparison only), then rerun airflow --help.
• If the unpacking traceback disappears, that strongly indicates malformed URI config was the direct issue.

Why this still feels slow even when config is valid

• airflow --help is not a “tiny CLI only” path; startup includes non-trivial imports and initialization work.
• On constrained environments (slow disk/network/CPU, heavy image, many installed packages), this startup overhead is much more noticeable.

Follow-up fix

I opened a PR to make this failure mode explicit and actionable instead of cryptic:

#68783

What that PR does:

1. validates malformed or empty sync SQLAlchemy URI input in startup path
2. raises a clearer error message that points directly to:

• AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
• sql_alchemy_conn in airflow.cfg

3. adds focused tests for:

• valid URI conversions
• malformed input behavior and messaging

If helpful, I can also share a small startup timing matrix (before/after config fix, and across two environments) to separate expected startup overhead from misconfiguration effects.