-
Notifications
You must be signed in to change notification settings - Fork 16.4k
task_type should be respected as it is supposed to be overridden.
#14821
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
kaxil
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, we use task_type to get the class name itself and was not intended to have a Human Readable name.
We also use it here for validation:
airflow/airflow/serialization/serialized_objects.py
Lines 434 to 440 in 0e13458
| for ope in plugins_manager.operator_extra_links: | |
| for operator in ope.operators: | |
| if ( | |
| operator.__name__ == encoded_op["_task_type"] | |
| and operator.__module__ == encoded_op["_task_module"] | |
| ): | |
| op_extra_links_from_plugin.update({ope.name: ope}) |
My use case isn't to place arbitrary sentences there, but create some fake-sub-classes (that my users will treat them as real class names). In 21a91c4 (2nd commit) I extended This won't affect existing code because
|
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*. |
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*. |
| operator.__name__ == encoded_op["_task_type"] | ||
| and operator.__module__ == encoded_op["_task_module"] | ||
| ): | ||
| ) if not isinstance(operator, str) else ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please find a cleaner way of writing this -- it is hard/impossible to grok what it is going.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Extracted this part to a function in 1fde244.
|
Code comments aside, I'm not sure if we want to support this behaviour - I'm trying to think through future implications where we might use the serialized representation for more (such as in the worker for actually creating Operator objects), and enabling this PR might stop us doing that. |
Supported regex `Pattern`. Match against full import path.
Do you mean
And either way, execute the deserialized operator without its DAG? |
|
I'm not thinking pickle, no, so something closer to the second point. |
|
Closing this for now as won't fix, sorry. |
The
task_typeproperty is designed to show customized name (for human to distinguish, not to import code)and is allowed to override in sub-class, if I understand correctly.
My Use Case
I need to distinguish operators by connection type, something likes:
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.