Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Feat/boto3 refactoring #58624

wants to merge 9 commits into from


Copy link

@major0 major0 commented Oct 4, 2020

What does this PR do?

Attempt to clearly de-couple AWS boto and boto3 and outline a clear path for removing the now deprecated AWS boto library.

This is a merger of two previous PR's as I was unable to cleanly separate them while still getting them to pass all the pre-commit and unit testing requirements.

Previous Behavior

  1. Numerous boto modules imported boto and boto3 even if they never used boto, and in a few cases they attempt to use both at the same time (even though they are generally incompatible).
        # Using conn.get_response is a bit of a hack, but it avoids having to
        # rewrite this whole module based on boto3
  1. boto3mod.get_connection() has a number of conditionals that will never fail (copy/pasted from botomod). Further more, these conditionals are in a try/except block which tries to catch a boto exception that is not defined in boto3.
  2. The salt.utils.version.check_boto_req() helper function is written in such a way that it defaults to assuming all modules need to use boto && boto3 (even though that is not desired).

New Behavior

  1. Remove check_boto/check_boto3 parameters from salt.utils.version.check_boto_reqs() and require explicit versions (and disallow specifying versions for boto and boto3 at the same time).
  2. Update various boto/boto3 mods to explicitly declare which version of boto they are checking for
  3. Remove unused boto imports from boto3 modules
  4. Rewrite boto3mod.get_connection() to work with boto3
  5. Initial unit testing for boto3mod (no tests previously existed for boto3mod).
  6. Add support to boto3mod.get_region() to attempt to use the local instance region by default, and if that fails to fall-back on us-east-1.
  7. Changes introduced by pre-commit scripts.

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

Commits signed with GPG?


Please review Salt's Contributing Guide for best practices.

See GitHub's page on GPG signing for more information about signing commits with GPG.

@major0 major0 requested a review from a team as a code owner October 4, 2020 01:09
@major0 major0 requested review from dwoz and removed request for a team October 4, 2020 01:09
@ghost ghost requested a review from krionbsd October 4, 2020 01:10
@major0 major0 changed the title Feat/boto3 refactoring WIP: Feat/boto3 refactoring Oct 4, 2020
@major0 major0 force-pushed the feat/boto3_refactoring branch 6 times, most recently from e78c9e5 to 04ad3af Compare October 4, 2020 16:23
krionbsd previously approved these changes Oct 5, 2020
A number of boto3 modules contain boto checks and imports, but do not
actually make any boto calls via the boto libraries.
The previous code looks to be a minor change of the boto
`get_connection()` code. In particular, `boto3.session.Session()`
generally does not perform any form of authentication or region
validation, and it certainly does not raise

Have rewritten the code to not attempt to use the old boto 2 module.
Add a simple http request to test to see if the saltmaster is running
within the AWS infrastructure and assume the instance's region as the
default region.  If this test fails then fall-back to the hard-coded
default of `us-east-1`.
This module did not previously have any unit tests defined.
Copy link
Collaborator Author

major0 commented Oct 13, 2020

@major0 major0 closed this Oct 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

3 participants