Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Only update crawler tags if present in config dict #32331

Merged
merged 2 commits into from
Jul 6, 2023
Merged

Only update crawler tags if present in config dict #32331

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
73e4f11
Only update crawler tags if present in config dict
luos-fc Jul 3, 2023
17ea779
Clean up uneeded config copy
luos-fc Jul 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 3 additions & 1 deletion airflow/providers/amazon/aws/hooks/glue_crawler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,9 @@ def update_crawler(self, **crawler_kwargs) -> bool:
crawler_name = crawler_kwargs["Name"]
current_crawler = self.get_crawler(crawler_name)

tags_updated = self.update_tags(crawler_name, crawler_kwargs.pop("Tags", {}))
tags_updated = (
self.update_tags(crawler_name, crawler_kwargs.pop("Tags")) if "Tags" in crawler_kwargs else False
)

update_config = {
key: value
Expand Down
14 changes: 12 additions & 2 deletions tests/providers/amazon/aws/hooks/test_glue_crawler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,14 +160,24 @@ def test_remove_all_tags(self, mock_get_conn):
ResourceArn=self.crawler_arn, TagsToRemove=["test", "bar"]
)

@mock_sts
@mock.patch.object(GlueCrawlerHook, "get_conn")
def test_update_missing_tags(self, mock_get_conn):
mock_config_missing_tags = deepcopy(mock_config)
mock_config_missing_tags.pop("Tags")
mock_get_conn.return_value.get_crawler.return_value = {"Crawler": mock_config_missing_tags}

assert self.hook.update_crawler(**mock_config_missing_tags) is False
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also add a testcase for when Tags key is in the config? Not sure if it's already covered.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think test_replace_tag function is testing the above case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah. Can you double check it @utkarsharma2 ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya, they are covered. I got confused by these lines, it seems like they don't add any value to the test. Test seems to work even after removing them.

airflow/tests/providers/amazon/aws/hooks/test_glue_crawler.py

Lines 169 to 170 in d012483

mock_config_two = deepcopy(mock_config)
mock_config_two.pop("Tags")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed those two lines

mock_get_conn.return_value.get_tags.assert_not_called()
mock_get_conn.return_value.tag_resource.assert_not_called()
mock_get_conn.return_value.untag_resource.assert_not_called()

@mock_sts
@mock.patch.object(GlueCrawlerHook, "get_conn")
def test_replace_tag(self, mock_get_conn):
mock_get_conn.return_value.get_crawler.return_value = {"Crawler": mock_config}
mock_get_conn.return_value.get_tags.return_value = {"Tags": mock_config["Tags"]}

mock_config_two = deepcopy(mock_config)
mock_config_two.pop("Tags")
assert self.hook.update_tags(mock_crawler_name, {"test": "bla", "bar": "test"}) is True
mock_get_conn.return_value.get_tags.assert_called_once_with(ResourceArn=self.crawler_arn)
mock_get_conn.return_value.untag_resource.assert_not_called()
Expand Down