Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dbt-ol should transparently propagate the exit code of dbt run #2558

Closed
blacklight opened this issue Apr 3, 2024 · 0 comments · Fixed by #2560 or #2591
Closed

dbt-ol should transparently propagate the exit code of dbt run #2558

blacklight opened this issue Apr 3, 2024 · 0 comments · Fixed by #2560 or #2591

Comments

@blacklight
Copy link
Contributor

As of now, we use the return_code of the dbt process to run either dbt_run_event_end or dbt_run_event_failed.

We thus notify the downstream OpenLineage listener(s) about failures, but we don't do the same with the upstream process.

If dbt run failed, the dbt-ol process will still return zero. This is particularly relevant when a DBT workflow is launched by an Airflow DAG - in such case, in the event of failure, Airflow will still report the job as successful.
blacklight added a commit to blacklight/OpenLineage that referenced this issue Apr 3, 2024
@blacklight
[OpenLineage#2558] dbt-ol should transparently exit with the same exi…
bdf3450 
…t code as the underlying dbt process
blacklight added a commit to blacklight/OpenLineage that referenced this issue Apr 3, 2024
@blacklight
[OpenLineage#2558] dbt-ol should transparently exit with the same exi…
c1c5af9 
…t code as the underlying dbt process

Also in this commit:

- Use lazy template evaluation instead of f-strings for logging (see
  https://realpython.com/python-f-strings/#lazy-evaluation-in-logging)

- `subprocess.Popen` should always be run through a context manager.

Signed-off-by: Fabio Manganiello <fabio.manganiello@booking.com>
@blacklight blacklight mentioned this issue Apr 3, 2024
Merged
4 tasks
JDarDagran pushed a commit that referenced this issue Apr 3, 2024
@blacklight
[#2558] dbt-ol should transparently exit with the same exit code as t…
bbc9636 
…he underlying dbt process (#2560)

Also in this commit:

- Use lazy template evaluation instead of f-strings for logging (see
  https://realpython.com/python-f-strings/#lazy-evaluation-in-logging)

- `subprocess.Popen` should always be run through a context manager.

Signed-off-by: Fabio Manganiello <fabio.manganiello@booking.com>
Co-authored-by: Fabio Manganiello <fabio.manganiello@booking.com>
blacklight added a commit to blacklight/OpenLineage that referenced this issue Apr 4, 2024
@blacklight
[OpenLineage#2558] dbt-ol should transparently exit with the same exi…
fb3d180 
…t code as the underlying dbt process (OpenLineage#2560)

Also in this commit:

- Use lazy template evaluation instead of f-strings for logging (see
  https://realpython.com/python-f-strings/#lazy-evaluation-in-logging)

- `subprocess.Popen` should always be run through a context manager.

Signed-off-by: Fabio Manganiello <fabio.manganiello@booking.com>
Co-authored-by: Fabio Manganiello <fabio.manganiello@booking.com>
Signed-off-by: Fabio Manganiello <fabio@manganiello.tech>
@blacklight blacklight mentioned this issue Apr 9, 2024
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@blacklight