Variables API: non-string `value` returns masked 500 on PATCH and silently corrupts data on POST

[unixengineer]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.2.1 (code path still present on main as of this report)

What happened and how to reproduce it?

Summary

VariableBody.value is typed as JsonValue, so the Variables REST API accepts any JSON type (array, object, number, bool). Everything downstream, however, assumes a str. This produces two distinct failures depending on the verb:

Request	Result
POST /api/v2/variables with value as a JSON array	201 — silently stores the Python repr, i.e. ['a', 'b'], which is not valid JSON
PATCH /api/v2/variables/{key} with value as a JSON array	500 Internal Server Error, real exception masked

The POST case is the more dangerous of the two: it succeeds, so nobody notices until a DAG calls Variable.get(key, deserialize_json=True) and blows up on a value that can no longer be parsed.

Note the inconsistency: a non-string description is correctly rejected with a clean 422. Only value misbehaves.

Reproduction

# 1) POST — silently corrupts (HTTP 201)
curl -X POST "$HOST/api/v2/variables" \
  -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" \
  -d '{"key":"my_var","value":["a","b"],"description":"d"}'
# -> 201 {"key":"my_var","value":"['a', 'b']", ...}
#    json.loads("['a', 'b']") raises JSONDecodeError

# 2) PATCH — 500
curl -X PATCH "$HOST/api/v2/variables/my_var" \
  -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" \
  -d '{"key":"my_var","value":["a","b"],"description":"d"}'
# -> 500 Internal Server Error

For contrast, both verbs behave correctly when value is a JSON string ("[\"a\", \"b\"]") — 200/201 and the stored value round-trips as valid JSON.

Root cause

airflow-core/src/airflow/api_fastapi/core_api/datamodels/variables.py:

class VariableBody(StrictBaseModel):
    key: str = Field(max_length=ID_LEN)
    value: JsonValue = Field(serialization_alias="val")   # accepts any JSON type
    description: str | None = Field(default=None)
    team_name: str | None = Field(max_length=50, default=None)

The two verbs then diverge:

PATCH — update_orm_from_pydantic() applies the patch straight onto the ORM instance via BulkService.apply_patch_with_update_mask(). setattr hits the Variable.val setter in airflow-core/src/airflow/models/variable.py:

@val.setter
def set_val(self, value):
    if value is not None:
        ...
        self._val = fernet.encrypt(bytes(value, "utf-8")).decode()

bytes(<list>, "utf-8") raises. Verified in-process:

>>> from airflow.models import Variable
>>> v = Variable(key='k'); v.val = ['a', 'b']
Traceback (most recent call last):
  File "airflow/models/variable.py", line 103, in set_val
    self._val = fernet.encrypt(bytes(value, "utf-8")).decode()
TypeError: encoding without a string argument

POST — goes through Variable.set(), which coerces instead of validating:

if serialize_json:
    stored_value = json.dumps(value, indent=2)
else:
    stored_value = str(value)   # list -> "['a', 'b']"

str() on a list yields Python repr with single quotes, so the stored value is not valid JSON and cannot be recovered by deserialize_json=True.

Middleware masks the real error

The TypeError never reaches the logs. JWTRefreshMiddleware (a Starlette BaseHTTPMiddleware) surfaces only:

File "airflow/api_fastapi/auth/middlewares/refresh_token.py", line 61, in dispatch
    response = await call_next(request)
File "starlette/middleware/base.py", line 169, in call_next
    raise RuntimeError("No response returned.")
RuntimeError: No response returned.

This made the failure substantially harder to diagnose — the traceback names neither the field nor the underlying TypeError. Same masking behaviour as reported in #68868 (different trigger, team_name: "") and #66889.

Suggested fix

Tighten the schema so the API rejects non-string values with a 422, consistent with description:

value: str | None = Field(serialization_alias="val", default=None)

If JsonValue must be kept for backwards compatibility, add a field_validator that JSON-serialises non-string input (json.dumps) rather than letting str() produce a Python repr, and guard Variable.set_val so it raises a validation error rather than a bare TypeError.

Separately, it would be worth making JWTRefreshMiddleware propagate the underlying exception so these failures are diagnosable from the API server logs.

Operating System

Debian GNU/Linux 12 (official apache/airflow base image, Python 3.11)

Versions of Apache Airflow Providers

apache-airflow-providers-fab (FAB auth manager enabled)

Deployment

Official Apache Airflow Helm Chart

Deployment details

Airflow 3.2.1, Python 3.11, Kubernetes, Postgres metadata DB, FAB auth manager. Reproduced both directly against the api-server pod (localhost:8080) and through the ingress, so it is not proxy-related. Also reproduced with two different users (roles Admin and Op), so it is not permission-related.

Anything else?

Occurs every time. Originally hit via the Web UI when saving a variable whose value was a JSON array; the UI issues 4 retries with backoff, so a single click produces 4 x 500 in the api-server access log.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct