-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(glue): fix typo in reported warning, report with flow_urn #8138
Conversation
def get_dataflow_graph(self, script_path: str) -> Optional[Dict[str, Any]]: | ||
def get_dataflow_graph( | ||
self, script_path: str, flow_urn: str | ||
) -> Optional[Dict[str, Any]]: | ||
""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tracking is easier with key flow_urn. script_path is already present in reported message.
self.report.report_failure( | ||
script_path, | ||
self.report_warning( | ||
flow_urn, | ||
f"Unable to download DAG for Glue job from {script_path}, so job subtasks and lineage will be missing: {e}", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the glue script can not be accessed due to some reason (missing permission - 403 Or missing script - 404), treat this as warning than error.
Log warnings and failures. We lose some warnings/failures due to Lossy nature of report.failures and report.warnings, in case there are more than 10 elements.
Checklist