fix(glue): Fix GlueJobOperator verbose logs not showing in deferrable mode#64342
Draft
shivaam wants to merge 1 commit intoapache:mainfrom
Draft
fix(glue): Fix GlueJobOperator verbose logs not showing in deferrable mode#64342shivaam wants to merge 1 commit intoapache:mainfrom
shivaam wants to merge 1 commit intoapache:mainfrom
Conversation
…bOperator When using GlueJobOperator with deferrable=True and verbose=True, CloudWatch logs were silently ignored because the trigger inherited the base waiter's run() method which only polls job status. This adds a run() override and _forward_logs() helper to GlueJobCompleteTrigger that streams logs from both output and error CloudWatch log groups, matching the format used by the synchronous path. Key changes: - Extract get_glue_log_group_names() and format_glue_logs() as shared helpers in hooks/glue.py to eliminate sync/async duplication - Override run() in GlueJobCompleteTrigger for verbose log streaming - Yield TriggerEvent with error status on failure (matching base class pattern) instead of raising AirflowException in the triggerer process - Add tests for verbose success, failure, max attempts, pagination, ResourceNotFoundException, and no-new-events scenarios closes: apache#56535
shivaam
force-pushed
the
fix/glue-deferrable-verbose-56535
branch
from
85cb82e to
945c8a3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-land of #63086 (reverted by #64340) with corrected error handling.
Changes
When
GlueJobOperatorruns withdeferrable=Trueandverbose=True, CloudWatchlogs were silently ignored because the trigger inherited the base waiter's
run()method which only polls job status.
Improvements over the original PR:
TriggerEventwith error status instead of raisingAirflowException—matches the
AwsBaseWaiterTrigger.run()pattern soexecute_complete()handleserrors in the worker, not the triggerer. This eliminates the
AirflowExceptionimport from both the trigger and test files, which was the root cause of the revert.
get_glue_log_group_names()andformat_glue_logs()as shared helpers inhooks/glue.pyto eliminate sync/async duplication withGlueJobHook.print_job_logs.closes: #56535
related: #63086, #64340
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.6) following the guidelines