Resolve GCSFileTransformOperator destination fallbacks after rendering - #70488
Open
mitre88 wants to merge 3 commits into
Open
Resolve GCSFileTransformOperator destination fallbacks after rendering#70488mitre88 wants to merge 3 commits into
mitre88 wants to merge 3 commits into
Conversation
delimiter (GCSListObjectsOperator), objects/prefix (GCSDeleteObjectsOperator) and destination fallbacks (GCSFileTransformOperator) are template fields, but were validated or resolved in __init__, before Jinja rendering. The deprecation warning, the mutual-exclusion check and the destination defaulting now run at execute() time on rendered values.
mitre88
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jason810496,
jscheffl,
potiuk and
shahar1
as code owners
July 27, 2026 00:24
boring-cyborg
Bot
added
area:dev-tools
area:providers
backport-to-v3-3-test
uranusjr
reviewed
Jul 27, 2026
shahar1
requested changes
Jul 27, 2026
Comment on lines
-321
to
-331
| if objects is None and prefix is None: | ||
| err_message = "(Task {task_id}) Either objects or prefix should be set. Both are None.".format( | ||
| **kwargs | ||
| ) | ||
| raise ValueError(err_message) | ||
| if objects is not None and prefix is not None: | ||
| err_message = "(Task {task_id}) Objects or prefix should be set. Both provided.".format(**kwargs) | ||
| raise ValueError(err_message) | ||
|
|
||
| super().__init__(**kwargs) | ||
|
|
Contributor
There was a problem hiding this comment.
Please restore the changes here, as it will be flagged as false-positive (see ##70505 for more details)
| raise ValueError(f"(Task {self.task_id}) Either objects or prefix should be set. Both are None.") | ||
| if self.objects is not None and self.prefix is not None: | ||
| raise ValueError(f"(Task {self.task_id}) Objects or prefix should be set. Both provided.") | ||
|
|
Contributor
There was a problem hiding this comment.
Please revert these changes
| providers/google/src/airflow/providers/google/cloud/operators/dataproc.py::DataprocCreateClusterOperator | ||
| providers/google/src/airflow/providers/google/cloud/operators/dataproc.py::DataprocSubmitJobOperator | ||
| providers/google/src/airflow/providers/google/cloud/operators/functions.py::CloudFunctionDeployFunctionOperator | ||
| providers/google/src/airflow/providers/google/cloud/operators/gcs.py::GCSDeleteObjectsOperator |
Contributor
There was a problem hiding this comment.
Please restore this line
The provision checks in GCSDeleteObjectsOperator are false positives under the relaxed validator definition proposed in apache#70505, so they stay in the constructor and the class keeps its exemption entry.
mitre88
changed the title
Contributor
Author
|
Done — reverted the Drafted-by: Claude Code (Fable 5) (no human review before posting) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the template-field validation burn-down tracked in #70296.
GCSFileTransformOperatorresolvesdestination_bucket/destination_objectfallbacks (destination_bucket or source_bucket) in__init__, where the source fields still hold un-rendered Jinja expressions. This moves the fallback resolution toexecute()(and mirrors it inget_openlineage_facets_on_start, which runs beforeexecute), so the fallback applies to the rendered values.Scope notes from review:
delimiterdeprecation warning inGCSListObjectsOperatorstays in__init__— deprecation applies to any usage of the argument, not the rendered value.objects/prefixprovision checks inGCSDeleteObjectsOperatorstay in__init__— they are false positives under the relaxed validator definition proposed in Allow argument-provision checks of templated fields in operator __init__ #70505. Both classes keep their exemption entries; only theGCSFileTransformOperatorentry is removed.Note:
TestGCSDeleteObjectsOperator::test_get_openlineage_facets_on_start[objects]fails on a pristinemaincheckout as well — pre-existing and unrelated to this change.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Fable 5) following the guidelines