Mask list-shaped Variable values on deserialization - #70891
Open
potiuk wants to merge 1 commit into
Open
Conversation
_mask_and_deserialize_variable dispatched on the top-level type of the deserialized value and handled only str and dict, so a Variable whose JSON is a list was returned with no masking applied to anything inside it. add_mask walks iterables itself, so the same list nested one level inside a dict was already masked -- only a top-level one was skipped. The list is passed under the variable's key rather than anonymously: elements have no key names of their own, so they follow the variable key's sensitivity, and a list of ordinary values such as region names is not added to the global pattern set. Dicts inside the list are still masked by their own key names.
SameerMesiah97
left a comment
SameerMesiah97
left a comment
Contributor
There was a problem hiding this comment.
Looks good. Just a few stylistic nits (for e.g. I found one comment a bit hard to understand).
| # its own, so bare values inside follow the same rule as the plain-string case above. | ||
| # Masking them unconditionally would add every element to the global pattern set, so a | ||
| # list of ordinary values such as region names would be redacted everywhere it appeared. | ||
| # Dicts *inside* the list are still masked by their own key names. |
Contributor
There was a problem hiding this comment.
I had to read this a few times to understand what it was trying to convey. I think the comment can be more concise and clear. Please see below:
# Pass the Variable's key so list elements inherit the Variable's sensitivity
# instead of being added to the global mask patterns.
| @mock.patch("airflow.sdk.execution_time.context.mask_secret") | ||
| def test_var_json_masks_list_values(self, mock_mask_secret, mock_supervisor_comms): | ||
| """A JSON list is handed to the masker whole, exactly as a dict is. | ||
|
|
Contributor
There was a problem hiding this comment.
I would keep the first line for the docstring.
| this: bare values inside a list have no key names of their own, so they follow the | ||
| variable key's sensitivity. Passing the list anonymously instead would add every element | ||
| to the global pattern set, and an ordinary value such as a region name would then be | ||
| redacted everywhere it appeared in the logs. |
Contributor
There was a problem hiding this comment.
Same here. The first line of the docstring can be kept and extra context can be retained in the comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_mask_and_deserialize_variabledispatched on the top-level type of thedeserialized value and handled only
stranddict. A Variable whose JSON is alist therefore matched no branch and was returned with nothing inside it masked.
add_maskwalks dicts and iterables itself, so the same list nested one levelinside a dict was already masked -- the top-level dispatch was the only thing
deciding whether the nested values were reached at all.
Approach
Add the
listbranch, passing the value under the variable's key ratherthan anonymously.
That distinction is the whole design of the change. Elements of a list have no
key names of their own, so they follow the variable key's sensitivity -- the same
rule already applied to a plain string value. Passing the list anonymously would
mask every element unconditionally and add each to the global pattern set, so a
Variable holding
["us-east-1", "eu-west-1"]would causeus-east-1to beredacted everywhere it appeared in any log.
test_var_json_list_value_does_not_over_maskalready guarded that, and this keeps it true. Dicts inside the list are still
masked by their own key names.
Verified against the real masker:
[{"password": "…"}]db_configs[{"password": "***"}]["us-east-1", "eu-west-1"]aws_regions["…"]my_password***{"password": "…"}db_config{"password": "***"}(unchanged path)test_var_json_list_value_does_not_over_maskis updated: it assertedmask_secretwas called exactly once, and there are now two calls. The propertyit guards is unchanged and the assertion is now stated directly -- the list is
passed under the variable's key, and never anonymously.
Test plan
test_var_json_masks_list_values-- list handed to the masker under thevariable's key
test_var_json_list_value_does_not_over_mask-- updated to assert theno-over-masking property rather than the call count
test_var_json_scalar_values_pass_through-- int/bool/null/float returnedunchanged
test_context.py-- 170 passedruff check/ruff formatcleanWas generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 5 (1M context) following the guidelines at
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions