Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AWS module_utils function to be able to check exceptions for a spec… #41202

Merged
merged 8 commits into from
Jun 12, 2018

Conversation

s-hertel
Copy link
Contributor

@s-hertel s-hertel commented Jun 6, 2018

…ific error code

WIP need to add this to the guidelines doc.

SUMMARY

Implementation for how we are using client.exceptions.from_code. from_code() is masking ClientErrors or ClientError subclasses.

ISSUE TYPE
  • Bugfix Pull Request
COMPONENT NAME

lib/ansible/module_utils/aws/core.py

ANSIBLE VERSION
ansible 2.7.0.dev0

@ansibot ansibot added WIP This issue/PR is a work in progress. Nevertheless it was shared for getting input from peers. affects_2.7 This issue/PR affects Ansible v2.7 feature This issue/PR relates to a feature request. needs_triage Needs a first human triage before being processed. support:core This issue/PR relates to code supported by the Ansible Engineering Team. labels Jun 6, 2018
@ansibot
Copy link
Contributor

ansibot commented Jun 6, 2018

The test ansible-test sanity --test pep8 [explain] failed with 1 error:

lib/ansible/module_utils/aws/core.py:223:5: E303 too many blank lines (2)

click here for bot help

@ryansb ryansb self-requested a review June 6, 2018 20:16
@jborean93 jborean93 removed the needs_triage Needs a first human triage before being processed. label Jun 7, 2018
@ryansb ryansb requested a review from willthames June 7, 2018 14:22
@ansibot
Copy link
Contributor

ansibot commented Jun 7, 2018

@ansibot ansibot added aws cloud module This issue/PR relates to a module. support:certified This issue/PR relates to certified code. support:community This issue/PR relates to code supported by the Ansible community. labels Jun 7, 2018
@ansibot
Copy link
Contributor

ansibot commented Jun 7, 2018

The test ansible-test sanity --test pylint [explain] failed with 11 errors:

lib/ansible/modules/cloud/amazon/aws_config_aggregator.py:101:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:93:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:109:11: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:112:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:134:15: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:137:16: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_recorder.py:102:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_rule.py:126:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/ec2_vpc_vgw.py:226:11: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/rds_instance_facts.py:368:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/rds_snapshot_facts.py:302:12: duplicate-except Catching previously caught exception type ClientError

The test ansible-test sanity --test no-underscore-variable [explain] failed with 1 error:

lib/ansible/module_utils/aws/core.py:271:9: use `dummy` instead of `_` for a variable name

click here for bot help

@ansibot ansibot added the bug This issue/PR relates to a bug. label Jun 7, 2018
@s-hertel
Copy link
Contributor Author

s-hertel commented Jun 7, 2018

thanks @ryansb, looks good

@s-hertel s-hertel changed the title [WIP] Add AnsibleAWSModule method to be able to check exceptions for a spec… Add AnsibleAWSModule method to be able to check exceptions for a spec… Jun 7, 2018
@s-hertel s-hertel changed the title Add AnsibleAWSModule method to be able to check exceptions for a spec… Add AWS module_utils function to be able to check exceptions for a spec… Jun 7, 2018
@ansibot ansibot added core_review In order to be merged, this PR must follow the core review workflow. and removed WIP This issue/PR is a work in progress. Nevertheless it was shared for getting input from peers. labels Jun 7, 2018
@ansibot
Copy link
Contributor

ansibot commented Jun 7, 2018

The test ansible-test sanity --test pylint [explain] failed with 11 errors:

lib/ansible/modules/cloud/amazon/aws_config_aggregator.py:101:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:93:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:109:11: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:112:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:134:15: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:137:16: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_recorder.py:102:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_rule.py:126:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/ec2_vpc_vgw.py:226:11: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/rds_instance_facts.py:368:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/rds_snapshot_facts.py:302:12: duplicate-except Catching previously caught exception type ClientError

click here for bot help

@ansibot ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed core_review In order to be merged, this PR must follow the core review workflow. labels Jun 7, 2018
@willthames
Copy link
Contributor

The duplicate-exception test doesn't seem to like this change :(

@willthames willthames force-pushed the AnsibleAWSModule-error-code branch from 3434a01 to f9228a1 Compare June 7, 2018 21:54
@willthames
Copy link
Contributor

@s-hertel I rebased your branch on latest devel and applied the same change to aws_eks_cluster so that if you work out how to appease the tests, aws_eks_cluster will get the fix too

@ansibot
Copy link
Contributor

ansibot commented Jun 7, 2018

The test ansible-test sanity --test pylint [explain] failed with 12 errors:

lib/ansible/modules/cloud/amazon/aws_config_aggregator.py:101:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:93:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:109:11: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:112:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:134:15: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_delivery_channel.py:137:16: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_recorder.py:102:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_config_rule.py:126:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/aws_eks_cluster.py:204:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/ec2_vpc_vgw.py:226:11: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/rds_instance_facts.py:368:12: duplicate-except Catching previously caught exception type ClientError
lib/ansible/modules/cloud/amazon/rds_snapshot_facts.py:302:12: duplicate-except Catching previously caught exception type ClientError

click here for bot help

@ansibot ansibot added the ci_verified Changes made in this PR are causing tests to fail. label Jun 7, 2018
@ryansb
Copy link
Contributor

ryansb commented Jun 8, 2018

This ignore should fix the tests

@ansibot
Copy link
Contributor

ansibot commented Jun 8, 2018

@ansibot ansibot added test This PR relates to tests. and removed ci_verified Changes made in this PR are causing tests to fail. labels Jun 8, 2018
lib/ansible/modules/cloud/amazon/ecs_ecr.py ansible-format-automatic-specification
lib/ansible/modules/cloud/amazon/rds_instance_facts.py duplicate-except
lib/ansible/modules/cloud/amazon/rds_snapshot_facts.py duplicate-except
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these ignores needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the exception being caught is being dynamically generated and may or may not be the same class caught below. Pylint doesn't differentiate between this and static duplicates yet. I've opened a ticket in hopes that we can ignore only dynamically generate exception possible-duplicates in the future pylint-dev/pylint#2174 (so we don't accidentally mask static duplicates for which there is no reason).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not add the error code check to the existing ClientError exception handlers instead of using dynamic types?

It looks like ClientError is already handled in most cases, so you'd end up with something like:

    except botocore.exceptions.ClientError as e:
        if is_boto3_error_code('NoSuchConfigRuleException'):
            # do stuff here
        else:
            # do other stuff here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, it just simplifies things and gives us the equivalent to what we thought from_code was (interestingly, from_code is also catching duplicate exceptions, pylint isn't catching it somehow).

Allows:

except is_boto3_error_code('ExceptionCodeWithSpecialHandling'):
    pass
except (BotoCoreError, ClientError) as e:
    module.fail_json_aws(e, msg="error message")

rather than

except ClientError as e:
    if is_boto3_error_code('NoSuchConfigRuleException'):
        pass
    else:
        module.fail_json_aws(e, msg="error message")
except BotoCoreError as e:
    module.fail_json_aws(e, msg="error message")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably use inline pylint ignore comments for these instead of ignore.txt, since these are not issues that need fixing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the correct ignore the # noqa comment on the line? Or is there an ignore where you can specify the error?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's # pylint: disable=duplicate-except

@ansibot ansibot added core_review In order to be merged, this PR must follow the core review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels Jun 8, 2018
@ryansb ryansb force-pushed the AnsibleAWSModule-error-code branch from bc5facd to 014f10b Compare June 12, 2018 13:48
@ansibot ansibot added needs_maintainer Ansibot is unable to identify maintainers for this PR. (Check `author` in docs or BOTMETA.yml) needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed core_review In order to be merged, this PR must follow the core review workflow. labels Jun 12, 2018
@ryansb ryansb merged commit 40d2df0 into ansible:devel Jun 12, 2018
jacum pushed a commit to jacum/ansible that referenced this pull request Jun 26, 2018
…ble#41202)

* Add aws/core.py function to check for specific AWS error codes

* Use sys.exc_info to get exception object if it isn't passed in

* Allow catching exceptions with is_boto3_error_code

* Replace from_code with is_boto3_error_code

* Return a type that will never be raised to support stricter type comparisons in Python 3+

* Use is_boto3_error_code in aws_eks_cluster

* Add duplicate-except to ignores when using is_boto3_error_code

* Add is_boto3_error_code to module development guideline docs
@ansible ansible locked and limited conversation to collaborators Jun 12, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
affects_2.7 This issue/PR affects Ansible v2.7 aws bug This issue/PR relates to a bug. cloud feature This issue/PR relates to a feature request. module This issue/PR relates to a module. needs_maintainer Ansibot is unable to identify maintainers for this PR. (Check `author` in docs or BOTMETA.yml) needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. support:certified This issue/PR relates to certified code. support:community This issue/PR relates to code supported by the Ansible community. support:core This issue/PR relates to code supported by the Ansible Engineering Team. test This PR relates to tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants