add basic openlineage metrics to stats collector#31223
add basic openlineage metrics to stats collector#31223harels wants to merge 1 commit intoapache:mainfrom
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
|
harels
changed the title
harels
changed the title
| event = self._redacter.redact(event, max_depth=20) | ||
| try: | ||
| return self._client.emit(event) | ||
| with Stats.timer("ol.emit.attempts"): |
There was a problem hiding this comment.
@harels, this is great. It will help with the telemetry open lineage events.
Would it make sense for us to add or modify an existing test to assert we're sending these to Statsd?
There was a problem hiding this comment.
+1, yes a test would help here.
There was a problem hiding this comment.
+1. Test should also include scenario for exception
pankajkoti
left a comment
pankajkoti
left a comment
There was a problem hiding this comment.
Should we add some description about these metrics in the doc https://github.com/apache/airflow/blob/eaaa030ab4dc4d178dab8194fd4ef9450b6b67a5/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst ?
| event = self._redacter.redact(event, max_depth=20) | ||
| try: | ||
| return self._client.emit(event) | ||
| with Stats.timer("ol.emit.attempts"): |
There was a problem hiding this comment.
+1, yes a test would help here.
|
@harels AFAIK you wanted to add something here? |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
6 participants
We currently don't have a good way to understand the latency of emitting openlineage events from the openlineage provider.
Beyond logging, there also isn't an easy way to aggregate failed attempts of sending the events.
This PR uses the statsd collector and wraps the section that emits to event with a timer, as well as emit a counter for exceptions in sending the event.