You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There is an OpenLineage consumer that does not aggregate metadata across the lifecycle of a run, opting instead to ignore all but COMPLETE event types. This is reasonable for the use case of this consumer; they care about schemas and lineage but not operational status. However, it means that if lineage is provided during a START event, this consumer will miss it.
It would be ideal to accumulate metadata across the run on the consumer side, but this is currently not possible. For the time being, we can repeat input/output metadata and facet metadata in both START and COMPLETE events to accommodate.
The text was updated successfully, but these errors were encountered:
This is the cause of some fairly frequent issues where an Airflow task fails before a START event is able to be sent (there seem to be a few paths to the on_task_instance_failed hook being called). The OL events can be quite large, so I understand the urge to reduce duplication, but unfortunately, it's causing some data to be missed entirely.
This is already default BaseExtractor behavior, so we just have to go through the extractors and make sure that their extract_on_complete() do not just pass.
There is an OpenLineage consumer that does not aggregate metadata across the lifecycle of a run, opting instead to ignore all but
COMPLETE
event types. This is reasonable for the use case of this consumer; they care about schemas and lineage but not operational status. However, it means that if lineage is provided during aSTART
event, this consumer will miss it.It would be ideal to accumulate metadata across the run on the consumer side, but this is currently not possible. For the time being, we can repeat input/output metadata and facet metadata in both
START
andCOMPLETE
events to accommodate.The text was updated successfully, but these errors were encountered: