[AIP-49] Airflow OpenTelemetry Provider #37989
Conversation
docs/apache-airflow-providers-opentelemetry/connections/opentelemetry.rst
Show resolved
Hide resolved
|
Finally made first pass. |
|
Thanks! :-)
…On Wed, Mar 13, 2024 at 7:37 PM Jarek Potiuk ***@***.***> wrote:
Finally made first pass.
—
Reply to this email directly, view it on GitHub
<#37989 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHZNLLXDQQEGNHKVIZ5UMX3YYDWNJAVCNFSM6AAAAABEMG6E32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJWGE4DQNZRHE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
howardyoo
requested review from
bolkedebruin,
kaxil,
pierrejeambrun,
eladkal,
o-nikolas,
hussein-awala,
uranusjr,
ryanahamilton,
bbovenzi,
ephraimbuddy,
dstandish and
XD-DENG
as code owners
hussein-awala
force-pushed
the
howardyoo/otel/provider
branch
from
88fe339
to
2c6ed8a
Compare
|
The PR was broken with around 100 unrelated commits, I just squashed all related commits into a single commit and rebased |
hussein-awala
removed request for
ephraimbuddy,
bolkedebruin,
bbovenzi,
kaxil,
XD-DENG,
pierrejeambrun,
dstandish,
eladkal and
o-nikolas
sure, willl do. Appreciate doing it. |
Just checked it, the merged commit looks good! |
…w into howardyoo/otel/provider
…w into howardyoo/otel/provider
| # Get all triggers that have no task instances depending on them... | ||
| ids = session.scalars( | ||
| # Get all triggers that have no task instances depending on them and delete them | ||
| ids = ( |
There was a problem hiding this comment.
I think that's unrelated ? (bad merge?)
| # | ||
| # END PROVIDER EXTRAS HERE | ||
|
|
||
| # PLEASE DO NOT MODIFY THIS SECTION MANUALLY. IT WILL BE OVERWRITTEN BY PRE-COMMIT !! |
There was a problem hiding this comment.
Bad merge here. It should be regenerated after the recent change for dynamic properties in pyproject toml. Copying the current version from main and re-running pre-commit should fix it.
| The `devel` extras are not available in the released packages. They are only available when you install | ||
| Airflow from sources in `editable` installation - i.e. one that you are usually using to contribute to | ||
| Airflow. They provide tools such as `pytest` and `mypy` for general purpose development and testing. | ||
| aiobotocore, airbyte, alibaba, all, all-core, all-dbs, amazon, apache-atlas, apache-beam, apache- |
There was a problem hiding this comment.
Bad merge here - should be replaced with the one from main and pre-commit run.
|
It looks good in general - I left a few comments where bad rebase/squash occured. Tests and Docs need to be fixed of course. I think we need to add more description (separate page in the "guides" section) explaining the usage of the provider - basically what you have described in the PR description here - but likely with some examples and explanation how the provider can be configured and used. It's very useful to be able to have traces automatically generated with listeners - this will enable a lot of opentelemetry configuration for earlier versions of airflow - particularly I think we need an example showing how to configure earlier version of Airflow (2.7.0+) and show few examples of how open-telemetry custom events can be emitted. |
| """Check whether otel listener is disabled.""" | ||
| return ( | ||
| is_otel_traces_enabled() is not True | ||
| and os.getenv("OTEL_LISTENER_DISABLED", "false").lower() == "false" |
There was a problem hiding this comment.
I think this should be a provider-specific configuration in provider.yaml rather than environment variable in [opentelemetry] section
|
One more comment here - I think we need to clearly describe what is the relation between the provider's traces that plugin interface provides - in the future when we merge the #37752 . Will traces replace those "plugin" events ? Or wil they be running side-by-side ? it's not clear for me at this moment what is the setup for:
|
So, in order to explain this a little bit more, the airflow's core otel trace will effectively 'replace' the provider's plugin events in case the provider detects existing OTEL tracing is currently enabled and running. yes, should be more clearer. |
closes #37628
Airflow OpenTelemetry Provider is a part of the Airflow OpenTelemetry feature that will allow DAG writers to emit opentelemetry metrics and traces conveniently within the framework of Airflow's provider. User who do not wish to use Airflow's core otel feature, or having Airflow version that does NOT have the otel feature available can use the provider package to send user specific metrics and traces.
The provider package also comes with listener plugin that will, when configured, automatically emit traces on DAG runs and TaskInstance runs. You can also combine the user-generated spans as part of the DAG run, giving users a complete control over how they would like to generate OpenTelemetry data while running DAGs.