Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vpc_net check mode, IPV6 CIDR assoc/disassoc #631

Conversation

sebastien-rosset
Copy link
Contributor

@sebastien-rosset sebastien-rosset commented Jan 22, 2022

SUMMARY

Implement check mode correctly for the ec2_vpc_net module. The module was incorrectly making actual changes when executed in check mode.

  1. In check mode, do not change the configuration. Previously the module was making VPC changes in the following scenarios:
    1. Association with IPv4 CIDR or IPv6 CIDR.
    2. Disassociation from IPv4 CIDR or IPv6 CIDR.
  2. Handle case when Amazon-provided ipv6 block is enabled, then disabled, then enabled again.
  3. Do not disable IPv6 CIDR association (using Amazon pool) if ipv6_cidr property is not present in the task. If the VPC already exists and ipv6_cidr property, retain the current config.
  4. Add integration tests:
    1. Enable, disable, then re-enable Amazon-provided IPv6 CIDR.
    2. When VPC already exists and ipv6_cidr property is not specified, validate this does not disable IPv6 CIDR association.
ISSUE TYPE
  • Bugfix Pull Request
COMPONENT NAME

ec2_vpc_net

ADDITIONAL INFORMATION

@sebastien-rosset sebastien-rosset changed the title vpc_net check mode vpc_net check mode, IPV6 CIDR assoc/disassoc Jan 23, 2022
"""
start_time = time()
criteria_match = False
while time() < start_time + 300:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering if it would be feasible to define a waiter instead? @tremble

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to add support for opensearch module. To support custom endpoint, I discovered I need to fix issues in the aws_acm module to support tags (ansible-collections/community.aws#870), which depends on adding new IAM permissions (mattclay/aws-terminator#188). Along the way, I fixed some doc issues and use of deprecated tasks. Then I found out aws_acm needs to support certificate requests (ansible-collections/community.aws#869) so I can issue a certificate for the custom opensearch endpoint. Also, certificate requests would potentially require adding a new module for managing private CAs in AWS acm, otherwise we cannot test signing certificate requests with private CAs. Then because I'm testing opensearch with IPv6, I discovered I need to fix the IPv6 association in the ec2_vpc_net module (#631). The ec2_vpc_net was making changes even in check mode. Also, the ec2_route_table wasn't documented for IPv6 so #634 . While adding integration tests for #634, I found out community.aws relies on ansible.netcommons, but the ipsubnet filter has been moved to ansible.utils. That's one more problem to fix that I haven't done yet. Then it turns out both ansible.netcommons and ansible.utils have an issue that can cause up to 2 ^ 128 objects to be created: ansible-collections/ansible.utils#132. There is also a related 2 ^ N loop in netaddr: netaddr/netaddr#241.

Along the way, I was thinking of several other problems (such as ec2_vpc_net not supporting user-defined IPv6 subnets, missing doc). How about opening tracking issues for these issues that are being discovered along the way, otherwise I'm concerned there will be no end in sight.

@ansibullbot
Copy link

@ansibullbot ansibullbot added bug This issue/PR relates to a bug community_review integration tests/integration module module needs_triage new_contributor Help guide this first time contributor plugins plugin (any type) tests tests labels Jan 27, 2022
Copy link
Contributor

@alinabuzachis alinabuzachis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sebastien-rosset Thank you for taking time to work of this. Could you please add a changelog fragment?

@ansibullbot
Copy link

@jillr jillr removed the needs_triage label Feb 8, 2022
@ansibullbot ansibullbot removed the new_contributor Help guide this first time contributor label Feb 16, 2022
@sebastien-rosset
Copy link
Contributor Author

@alinabuzachis, @jillr , IMO this is a severe bug. Dry-run aka check-mode should never modify anything.

@alinabuzachis
Copy link
Contributor

@sebastien-rosset Thank you very much for your contributions overall. You're correct, there shouldn't be any change in check_mode. By the way, DryRun is not the same thing as check_mode. DryRun only checks permissions. I'm going to review this PR later on today and let you know. Sorry for the delay.

Copy link
Contributor

@alinabuzachis alinabuzachis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sebastien-rosset Thank you for working on this. Please reference the PR in the changelog. Other than that LGTM!

changelogs/fragments/631-ec2_vpc_net-check_mode.yml Outdated Show resolved Hide resolved
@softwarefactory-project-zuul
Copy link
Contributor

Build failed.

@sebastien-rosset
Copy link
Contributor Author

sebastien-rosset commented Mar 1, 2022

integration tests are failing because of /modules/ec2_vpc_igw.py", line 158, in get_igw_info TypeError: 'NoneType' object is not subscriptable:

#647

<testhost> ESTABLISH LOCAL CONNECTION FOR USER: zuul
<testhost> EXEC /bin/sh -c 'ANSIBLE_DEBUG_BOTOCORE_LOGS=True /home/zuul/venv/bin/python && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "<stdin>", line 121, in <module>
  File "<stdin>", line 113, in _ansiballz_main
  File "<stdin>", line 61, in invoke_module
  File "/usr/lib64/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib64/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_ec2_vpc_igw_payload_bg8zbtef/ansible_ec2_vpc_igw_payload.zip/ansible_collections/amazon/aws/plugins/modules/ec2_vpc_igw.py", line 254, in <module>
  File "/tmp/ansible_ec2_vpc_igw_payload_bg8zbtef/ansible_ec2_vpc_igw_payload.zip/ansible_collections/amazon/aws/plugins/modules/ec2_vpc_igw.py", line 248, in main
  File "/tmp/ansible_ec2_vpc_igw_payload_bg8zbtef/ansible_ec2_vpc_igw_payload.zip/ansible_collections/amazon/aws/plugins/modules/ec2_vpc_igw.py", line 132, in process
  File "/tmp/ansible_ec2_vpc_igw_payload_bg8zbtef/ansible_ec2_vpc_igw_payload.zip/ansible_collections/amazon/aws/plugins/modules/ec2_vpc_igw.py", line 226, in ensure_igw_present
  File "/tmp/ansible_ec2_vpc_igw_payload_bg8zbtef/ansible_ec2_vpc_igw_payload.zip/ansible_collections/amazon/aws/plugins/modules/ec2_vpc_igw.py", line 158, in get_igw_info
TypeError: 'NoneType' object is not subscriptable

@jatorcasso
Copy link
Contributor

recheck

@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded.

@alinabuzachis alinabuzachis added backport-3 PR should be backported to the stable-3 branch mergeit Merge the PR (SoftwareFactory) labels Mar 14, 2022
@softwarefactory-project-zuul
Copy link
Contributor

Build failed (gate pipeline). For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

@alinabuzachis alinabuzachis added backport-2 PR should be backported to the stable-2 branch mergeit Merge the PR (SoftwareFactory) and removed mergeit Merge the PR (SoftwareFactory) labels Mar 21, 2022
@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded (gate pipeline).

@softwarefactory-project-zuul softwarefactory-project-zuul bot merged commit 3e24a37 into ansible-collections:main Mar 21, 2022
@patchback
Copy link

patchback bot commented Mar 21, 2022

Backport to stable-2: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 3e24a37 on top of patchback/backports/stable-2/3e24a37b3f0c6678c3936a8a6cf228e02c64068c/pr-631

Backporting merged PR #631 into main

  1. Ensure you have a local repo clone of your fork. Unless you cloned it
    from the upstream, this would be your origin remote.
  2. Make sure you have an upstream repo added as a remote too. In these
    instructions you'll refer to it by the name upstream. If you don't
    have it, here's how you can add it:
    $ git remote add upstream https://github.com/ansible-collections/amazon.aws.git
  3. Ensure you have the latest copy of upstream and prepare a branch
    that will hold the backported code:
    $ git fetch upstream
    $ git checkout -b patchback/backports/stable-2/3e24a37b3f0c6678c3936a8a6cf228e02c64068c/pr-631 upstream/stable-2
  4. Now, cherry-pick PR vpc_net check mode, IPV6 CIDR assoc/disassoc #631 contents into that branch:
    $ git cherry-pick -x 3e24a37b3f0c6678c3936a8a6cf228e02c64068c
    If it'll yell at you with something like fatal: Commit 3e24a37b3f0c6678c3936a8a6cf228e02c64068c is a merge but no -m option was given., add -m 1 as follows intead:
    $ git cherry-pick -m1 -x 3e24a37b3f0c6678c3936a8a6cf228e02c64068c
  5. At this point, you'll probably encounter some merge conflicts. You must
    resolve them in to preserve the patch from PR vpc_net check mode, IPV6 CIDR assoc/disassoc #631 as close to the
    original as possible.
  6. Push this branch to your fork on GitHub:
    $ git push origin patchback/backports/stable-2/3e24a37b3f0c6678c3936a8a6cf228e02c64068c/pr-631
  7. Create a PR, ensure that the CI is green. If it's not — update it so that
    the tests and any other checks pass. This is it!
    Now relax and wait for the maintainers to process your pull request
    when they have some cycles to do reviews. Don't worry — they'll tell you if
    any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

@patchback
Copy link

patchback bot commented Mar 21, 2022

Backport to stable-3: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-3/3e24a37b3f0c6678c3936a8a6cf228e02c64068c/pr-631

Backported as #722

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback bot pushed a commit that referenced this pull request Mar 21, 2022
vpc_net check mode, IPV6 CIDR assoc/disassoc

SUMMARY

Implement check mode correctly for the ec2_vpc_net module. The module was incorrectly making actual changes when executed in check mode.

In check mode, do not change the configuration. Previously the module was making VPC changes in the following scenarios:

Association with IPv4 CIDR or IPv6 CIDR.
Disassociation from IPv4 CIDR or IPv6 CIDR.

Handle case when Amazon-provided ipv6 block is enabled, then disabled, then enabled again.
Do not disable IPv6 CIDR association (using Amazon pool) if ipv6_cidr property is not present in the task. If the VPC already exists and ipv6_cidr property, retain the current config.
Add integration tests:

Enable, disable, then re-enable Amazon-provided IPv6 CIDR.
When VPC already exists and ipv6_cidr property is not specified, validate this does not disable IPv6 CIDR association.

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

ec2_vpc_net
ADDITIONAL INFORMATION

Reviewed-by: Sebastien Rosset <None>
Reviewed-by: Alina Buzachis <None>
Reviewed-by: Joseph Torcasso <None>
Reviewed-by: Markus Bergholz <git@osuv.de>
(cherry picked from commit 3e24a37)
softwarefactory-project-zuul bot pushed a commit that referenced this pull request Mar 21, 2022
[PR #631/3e24a37b backport][stable-3] vpc_net check mode, IPV6 CIDR assoc/disassoc

This is a backport of PR #631 as merged into main (3e24a37).
SUMMARY

Implement check mode correctly for the ec2_vpc_net module. The module was incorrectly making actual changes when executed in check mode.

In check mode, do not change the configuration. Previously the module was making VPC changes in the following scenarios:

Association with IPv4 CIDR or IPv6 CIDR.
Disassociation from IPv4 CIDR or IPv6 CIDR.


Handle case when Amazon-provided ipv6 block is enabled, then disabled, then enabled again.
Do not disable IPv6 CIDR association (using Amazon pool) if ipv6_cidr property is not present in the task. If the VPC already exists and ipv6_cidr property, retain the current config.
Add integration tests:

Enable, disable, then re-enable Amazon-provided IPv6 CIDR.
When VPC already exists and ipv6_cidr property is not specified, validate this does not disable IPv6 CIDR association.




ISSUE TYPE


Bugfix Pull Request

COMPONENT NAME

ec2_vpc_net
ADDITIONAL INFORMATION
softwarefactory-project-zuul bot pushed a commit that referenced this pull request Mar 29, 2022
Improve doc to show support for IPv6 CIDR block

SUMMARY


Improve doc to show IPv6 CIDR blocks are supported.
Add example with IPv6 CIDR block.
Add missing attribute to return values.
Remove duplicate assertions in integration tests.
Add tests for IPv6 subnets in integration tests.


ISSUE TYPE


Bugfix Pull Request

COMPONENT NAME

ec2_vpc_route_table
ADDITIONAL INFORMATION



While testing IPv6 in this module, I discovered the ipsubnet filter does not work if the prefix length has a fairly high value such as /120.

I was running an integration test with a /120 subnet in this PR, but the ipsubnet call never returns and uses 100% CPU. I changed the value to a /64 subnet so this PR can pass.
The ipsubnet issue needs to be fixed in ansible.netcommon and the netaddr package. I've created a unit test to reproduce the problem: ansible-collections/ansible.netcommon#362


The integration tests in this PR depend on #631 for the VPC configuration.

Reviewed-by: Mark Chappell <None>
Reviewed-by: Sebastien Rosset <None>
Reviewed-by: Alina Buzachis <None>
Reviewed-by: Jill R <None>
patchback bot pushed a commit that referenced this pull request Mar 29, 2022
Improve doc to show support for IPv6 CIDR block

SUMMARY

Improve doc to show IPv6 CIDR blocks are supported.
Add example with IPv6 CIDR block.
Add missing attribute to return values.
Remove duplicate assertions in integration tests.
Add tests for IPv6 subnets in integration tests.

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

ec2_vpc_route_table
ADDITIONAL INFORMATION

While testing IPv6 in this module, I discovered the ipsubnet filter does not work if the prefix length has a fairly high value such as /120.

I was running an integration test with a /120 subnet in this PR, but the ipsubnet call never returns and uses 100% CPU. I changed the value to a /64 subnet so this PR can pass.
The ipsubnet issue needs to be fixed in ansible.netcommon and the netaddr package. I've created a unit test to reproduce the problem: ansible-collections/ansible.netcommon#362

The integration tests in this PR depend on #631 for the VPC configuration.

Reviewed-by: Mark Chappell <None>
Reviewed-by: Sebastien Rosset <None>
Reviewed-by: Alina Buzachis <None>
Reviewed-by: Jill R <None>
(cherry picked from commit c06d77c)
softwarefactory-project-zuul bot pushed a commit that referenced this pull request Mar 29, 2022
[PR #634/c06d77c9 backport][stable-3] Improve doc to show support for IPv6 CIDR block

This is a backport of PR #634 as merged into main (c06d77c).
SUMMARY


Improve doc to show IPv6 CIDR blocks are supported.
Add example with IPv6 CIDR block.
Add missing attribute to return values.
Remove duplicate assertions in integration tests.
Add tests for IPv6 subnets in integration tests.


ISSUE TYPE


Bugfix Pull Request

COMPONENT NAME

ec2_vpc_route_table
ADDITIONAL INFORMATION



While testing IPv6 in this module, I discovered the ipsubnet filter does not work if the prefix length has a fairly high value such as /120.

I was running an integration test with a /120 subnet in this PR, but the ipsubnet call never returns and uses 100% CPU. I changed the value to a /64 subnet so this PR can pass.
The ipsubnet issue needs to be fixed in ansible.netcommon and the netaddr package. I've created a unit test to reproduce the problem: ansible-collections/ansible.netcommon#362


The integration tests in this PR depend on #631 for the VPC configuration.
alinabuzachis pushed a commit to alinabuzachis/amazon.aws that referenced this pull request Mar 31, 2022
) (ansible-collections#751)

[PR ansible-collections#634/c06d77c9 backport][stable-3] Improve doc to show support for IPv6 CIDR block

This is a backport of PR ansible-collections#634 as merged into main (c06d77c).
SUMMARY


Improve doc to show IPv6 CIDR blocks are supported.
Add example with IPv6 CIDR block.
Add missing attribute to return values.
Remove duplicate assertions in integration tests.
Add tests for IPv6 subnets in integration tests.


ISSUE TYPE


Bugfix Pull Request

COMPONENT NAME

ec2_vpc_route_table
ADDITIONAL INFORMATION



While testing IPv6 in this module, I discovered the ipsubnet filter does not work if the prefix length has a fairly high value such as /120.

I was running an integration test with a /120 subnet in this PR, but the ipsubnet call never returns and uses 100% CPU. I changed the value to a /64 subnet so this PR can pass.
The ipsubnet issue needs to be fixed in ansible.netcommon and the netaddr package. I've created a unit test to reproduce the problem: ansible-collections/ansible.netcommon#362


The integration tests in this PR depend on ansible-collections#631 for the VPC configuration.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-2 PR should be backported to the stable-2 branch backport-3 PR should be backported to the stable-3 branch bug This issue/PR relates to a bug community_review has_issue integration tests/integration mergeit Merge the PR (SoftwareFactory) module module plugins plugin (any type) tests tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants