Openlineage: read API key auth from Airflow connection

[VladaZakharova]

Adds OpenLineage HTTP API key authentication from Airflow connection.

Users now can configure connection like this:
{"type":"http","url":"https://openlineage.example.com","auth":{"type":"airflow_connection_api_key","conn_id":"openlineage_default"}}

The token now is read from the connection password, or from connection extra keys such as apiKey, api_key, token, or access_token. This token is used in constructing new valid connection under the hood.

This option will help creating connections without exposing tokens in Airflow connection configs.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[kacpermuda]

I see the value in using Airflow connections for OpenLineage configuration, but I'd suggest expanding the scope beyond just auth tokens for HTTP transport (that are solving a very specific use case).

Instead of a narrow solution, what if we allow storing the entire OpenLineage config dict (f.e. what can be put in the yaml file, or even just transport config) in an Airflow connection, that we can add to get_openlineage_config as yet another config source checked. (It should probably have precedence over yaml file or env vars, but that's something we can decide later)

This approach would:

• Work for any transport type, not just HTTP
• Support composite transports (e.g., two HTTP transports with different auth)
• Handle any OL config field, not just auth tokens
• Be more flexible for future use cases

The current auth-token-only solution would miss users with composite transports or other config needs. Could you expand the scope to cover the full OL config dict?

cc @mobuchowski

[VladaZakharova]

I see the value in using Airflow connections for OpenLineage configuration, but I'd suggest expanding the scope beyond just auth tokens for HTTP transport (that are solving a very specific use case).

Instead of a narrow solution, what if we allow storing the entire OpenLineage config dict (f.e. what can be put in the yaml file, or even just transport config) in an Airflow connection, that we can add to get_openlineage_config as yet another config source checked. (It should probably have precedence over yaml file or env vars, but that's something we can decide later)

This approach would:

• Work for any transport type, not just HTTP
• Support composite transports (e.g., two HTTP transports with different auth)
• Handle any OL config field, not just auth tokens
• Be more flexible for future use cases

The current auth-token-only solution would miss users with composite transports or other config needs. Could you expand the scope to cover the full OL config dict?

cc @mobuchowski

hi there!
okay, this sounds reasonable, i think it is worth trying
should i wait for @mobuchowski response? or i can go with implementation?

[kacpermuda]

should i wait for @mobuchowski response? or i can go with implementation?

Go ahead, I'll be happy to review the PR, then we can ask Maciej for merge as he has the power to do it.

[mobuchowski]

@VladaZakharova Sorry I was going to reply but totally lost track somewhere. Yeah, IMO it would be great if we could cover the whole transport config.

An alternative would be to cover the subset - for example HTTP transport - in a way that would not conflict later, so use the same JSON structure.

[VladaZakharova]

hi there!
I tried to address your ideas, please check them when possible :)

[kacpermuda]

Added few comments, the most important decision I think is "can we do openlineage conn type" and: should we?

[kacpermuda]

Should every resolve be guarded and actually fire only if there is an airflow connection defined? I think it fires every time now. We need to make sure that for users not setting up this conn, nothing changes.

[VladaZakharova]

I think we’re okay here. The resolver is called for each config source, but it only does anything when it finds an auth block with:

{"type": "airflow_connection_api_key"}

For normal OpenLineage config, like regular api_key auth, it does not read any Airflow connection and should behave the same as before. I added a test for that case to make sure we don’t accidentally change it later.

[kacpermuda]

Hmm, wondering if there is any way we can use OL client initialization as validation here to avoid duplicate check logic. If not it's fine, we'll extend this validation in the future.

[VladaZakharova]

I thought about using OpenLineageClient(config=...) for this, but I think it would be a bit too heavy for validation here. It would create the client/transport once just to check the config, and then we would create it again later in the adapter.

So for now I kept this check very small: the Airflow connection extra must be a JSON object with a transport object. The OpenLineage client still does the real transport/auth validation when it is created. If the OpenLineage client gets a dedicated validation method later, we can switch to that.

[kacpermuda]

Makes sense !

[kacpermuda]

Hey @VladaZakharova , I see some comments being resolved or marked as addressed, but see no recent commits. Is all the relevant code pushed as intended?

[VladaZakharova]

Hey @VladaZakharova , I see some comments being resolved or marked as addressed, but see no recent commits. Is all the relevant code pushed as intended?

not really, i forgot to push my changes :D

[kacpermuda]

Thanks, this looks good now ! Left two more nit comments about docstring improvement and one test we should add.

[kacpermuda]

Makes sense !

[potiuk]

@VladaZakharova — Removing the ready for maintainer review label and converting back to draft. CI has failed since the label was added (4 failing checks).

The label's contract is that the PR is ready for maintainer review — a regression like this means the PR temporarily isn't. Check the failing checks, fix the regression, push, then mark "Ready for review" again to re-enter the queue.

See the Pull Request quality criteria.

No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[kacpermuda]

I think this is ready to go out of draft, tests are passing, code looks good, one final review and merge from Maciej is needed.