Connections panel inacessible after 3.2 migration

[DonHaul]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.2.0

What happened and how to reproduce it?

• After migrating from 3.1.7 to 3.2.0 the connections panel throws a 500 Internal Server Error.
• Checking the Logs we see it cant json serialize some extra fields properly ( see below Detail 1)
• Checking the DB we see some of these fields are empty string others NULL , both allowed in 3.1.7 ( see below Detail 2 )

Quick fix Updating the empty string to NULL solved the issue

Detail 1 - Logs:

[uvicorn.error] loc=httptools_impl.py:421
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1159, in __call__
    await super().__call__(scope, receive, send)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/applications.py", line 107, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/api_fastapi/common/http_access_log.py", line 83, in __call__
    await self.app(scope, receive, capture_send)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 29, in __call__
    await responder(scope, receive, send)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 130, in __call__
    await super().__call__(scope, receive, send)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 46, in __call__
    await self.app(scope, receive, self.send_with_compression)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
    with recv_stream, send_stream, collapse_excgroups():
  File "/usr/python/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
    raise exc
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 193, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 61, in dispatch
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 168, in call_next
    raise app_exc from app_exc.__cause__ or app_exc.__context__
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 716, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 736, in app
    await route.handle(scope, receive, send)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 290, in handle
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/routing.py", line 134, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/routing.py", line 120, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/routing.py", line 695, in app
    content = await serialize_response(
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/routing.py", line 306, in serialize_response
    return serializer(
           ^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/_compat/v2.py", line 218, in serialize_json
    return self._type_adapter.dump_json(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/pydantic/type_adapter.py", line 672, in dump_json
    return self.serializer.to_json(
           ^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.PydanticSerializationError: Error serializing to JSON: ValidationError: 1 validation error for ValidatorIterator
2.extra
  Value error, This code path should never happen as persisted Connections (DB layer) should always enforce `extra` as a JSON string. [type=value_error, input_value='', input_type=str]
    For further information visit https://errors.pydantic.dev/2.12/v/value_error
[] []

Detail 2 - Connections.extra state in DB after migration:

What you think should happen instead?

Connections should be visible after migration. My guess is either we are missing an alembic migration to make <empty_string> into NULL or simply loosen a bit the parsing to allow for this cases.

Operating System

No response

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

1.19.0

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[tirkarthi]

Related 09ed27a

cc: @pierrejeambrun

[tseruga]

You can use these steps to update old connections with the expected null extra field:

airflow connections export --file-format json connections_export.json

sed -E 's/"extra":[[:space:]]*""/"extra": null/g' connections_export.json > connections_fixed.json

airflow connections import --overwrite connections_fixed.json

Essentially, extra: "" is now invalid, but extra: null is valid.

[pierrejeambrun]

Thanks for reporting this.

_validate_extra states that Verify that ``extra`` is a JSON-encoded Python dict but for backward compatibility reason it appears it isn't called on empty string value "". It's an unintended side effect.

I'll open a PR to fix that.

[pierrejeambrun]

Fix PR #65014