Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW] fix: Proceed without rendering templates if task_instance copy fails #2492

Merged
merged 1 commit into from Mar 5, 2024

Conversation

kacpermuda
Copy link
Contributor

Problem

In listener's on_task_instance_running method, we are working on task_instance copy when rendering templates. Sometimes making a copy can fail, if there are some objects that cannot be pickled and copied with copy.deepcopy().

Solution

I think we have two options if copying fails:

  1. Skip rendering templates
  2. Render templates on task_instance and not the copy.

This PR implements the first one, as the second one can lead to this issue.

One-line summary:

Proceed without rendering templates if task_instance copy fails in listener.on_task_instance_running.

Checklist

  • You've signed-off your work
  • Your pull request title follows our guidelines
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • Your comment includes a one-liner for the changelog about the specific purpose of the change (if necessary)
  • You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
  • You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

…in listener.on_task_instance_running

Signed-off-by: Kacper Muda <mudakacper@gmail.com>
@boring-cyborg boring-cyborg bot added the area:integration/airflow openlineage-airflow label Mar 5, 2024
@mobuchowski mobuchowski merged commit 5c032e2 into main Mar 5, 2024
29 checks passed
@mobuchowski mobuchowski deleted the fix/copy_ti_on_running branch March 5, 2024 15:35
Ruihua98 pushed a commit to Ruihua98/OpenLineage that referenced this pull request Mar 15, 2024
…in listener.on_task_instance_running (OpenLineage#2492)

Signed-off-by: Kacper Muda <mudakacper@gmail.com>
Signed-off-by: Ruihua Wang <ruihuawang@microsoft.com>
blacklight pushed a commit to blacklight/OpenLineage that referenced this pull request Apr 4, 2024
…in listener.on_task_instance_running (OpenLineage#2492)

Signed-off-by: Kacper Muda <mudakacper@gmail.com>
Signed-off-by: Fabio Manganiello <fabio@manganiello.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:integration/airflow openlineage-airflow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants