Add --backend=custom option for external databases in Breeze

[andreahlert]

Summary

Adds a --backend=custom option to Breeze, allowing users to connect to an external
database without spinning up an unnecessary database container.

When custom is selected:

• No database container is started
• Users must provide their own connection string via AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
• A warning is displayed that only officially supported backends are tested
• start-airflow works normally (unlike --backend=none)

Use case

Users running database benchmarks or connecting to remote databases with custom
monitoring setups can now use Breeze without creating a dummy container or
working around the --backend=none limitation.

Changes

• global_constants.py - Added CUSTOM_BACKEND constant and included it in ALLOWED_BACKENDS
• common_options.py - Updated --backend help text to document the custom option
• shell_params.py - Added custom to the list of backends that skip port forwarding
• check_environment.sh - Added custom backend validation (requires AIRFLOW__DATABASE__SQL_ALCHEMY_CONN)
• entrypoint_ci.sh - Added custom backend display in initialization info
• backend-custom.yml - New docker-compose file that sets BACKEND=custom without overriding the DB connection

Closes: #55886

[potiuk]

I think if we do that, then there should be a parameter defined to provide the sqlalchemy URL to connect to - rather than relying on the env variable setting (and this new paraemeter should have an env fallback)

[andreahlert]

Good point! I'll add a --custom-db-url / -C parameter that accepts the SQLAlchemy connection string directly, with AIRFLOW__DATABASE__SQL_ALCHEMY_CONN as the environment variable fallback. When --backend=custom is used without either, Breeze will show a clear error.

I went with --custom-db-url for naming, but happy to rename if you prefer something different. Will update the PR shortly.

[jscheffl]

I Like the idea very much.

Due to a parallel PR the generated docs need to be re-generated. Then it is OK to be merged in my view

Just one thought before fixing all: Have you considered to add the DB URL directly in the place where --backend is defined? Then we would save passing an additional parameter. Only tradeoff would be that click can not validate the parameters from a fixed enum.

[andreahlert]

Thanks, glad you like it!

I thought about putting the URL directly in --backend, but it felt a bit tricky since that parameter drives other things too (docker image, test configs, CI matrix). Mixing a backend name with a raw connection string would need extra parsing to tell them apart, and we'd lose Click's enum validation for the known backends.

Keeping --custom-db-url separate also maps nicely to the AIRFLOW__DATABASE__SQL_ALCHEMY_CONN env fallback. But if you and @potiuk prefer the single-parameter route, I can refactor - no strong feelings either way.

Will regenerate the docs and address your inline comment too.

[jscheffl]

I thought about putting the URL directly in --backend, but it felt a bit tricky since that parameter drives other things too (docker image, test configs, CI matrix). Mixing a backend name with a raw connection string would need extra parsing to tell them apart, and we'd lose Click's enum validation for the known backends.

Keeping --custom-db-url separate also maps nicely to the AIRFLOW__DATABASE__SQL_ALCHEMY_CONN env fallback. But if you and @potiuk prefer the single-parameter route, I can refactor - no strong feelings either way.

That makes totally sense! So let's go with the current proposal in the PR. That is also OK.