Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for automated integration testing of policies #455

Open
ocampocj opened this issue Sep 16, 2016 · 13 comments
Open

Allow for automated integration testing of policies #455

ocampocj opened this issue Sep 16, 2016 · 13 comments

Comments

@ocampocj
Copy link
Contributor

@ocampocj ocampocj commented Sep 16, 2016

@kapil, we should have the ability to automate policy testing using cloudformation (or other tool). AWS resources should be spun up, the policies should run against them, and then the resources can be terminated.

@kapilt

This comment has been minimized.

Copy link
Collaborator

@kapilt kapilt commented Nov 8, 2016

this sounds useful, i think its probably a standalone tool of some kind, its a little unclear what it would be doing

   policy.yml
   stack/       
   test.yml
   data

test.yml needs some notion of a stack, its format/runner, and then some notion of validating the output.

@chadwhitacre

This comment has been minimized.

Copy link
Contributor

@chadwhitacre chadwhitacre commented Jan 7, 2017

From #62 (comment):

Ideally we'd actually run the policies automatically to make sure the examples still work, but that would require probably another yaml block specifying a fixture policy to put AWS into the state assumed by the target yaml itself.


a few users expressed interest in having a policy test runner, which would actually do something like that, by giving some fixtures (terraform or cfn) and then being able to run the policy and assert actions.

@chadwhitacre

This comment has been minimized.

Copy link
Contributor

@chadwhitacre chadwhitacre commented Jan 7, 2017

@chadwhitacre

This comment has been minimized.

Copy link
Contributor

@chadwhitacre chadwhitacre commented Jan 7, 2017

Started messing around with this. Setting up fixture via CloudFormation and running Custodian against it is easy enough. The missing piece for me is comprehending resulting state so we can diff our expectation against it. Can one export an AWS state back to a CloudFormation json/yaml description?

@kapilt

This comment has been minimized.

Copy link
Collaborator

@kapilt kapilt commented Jan 7, 2017

i was thinking the validation would be another custodian policy ;-)

@chadwhitacre

This comment has been minimized.

Copy link
Contributor

@chadwhitacre chadwhitacre commented Jan 9, 2017

Starting working this in https://github.com/whit537/cloud-custodian/commit/5f2e8ff7d3868f27b826679e5d8d971a64a66b45, but I'm running up against IAM perms again:

(cloud-custodian) $ nosetests -s tests/integration/test_something.py 

[…]

----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/nose/suite.py", line 209, in run
    self.setUp()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
    self.setupContext(ancestor)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext
    try_run(context, names)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/nose/util.py", line 471, in try_run
    return func()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tests/integration/test_something.py", line 46, in setUpClass
    TemplateURL='https://s3.amazonaws.com/cloudformation-examples/AWSCloudFormer.template'
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tests/integration/test_something.py", line 29, in create_stack
    response = cls.cfn.create_stack(StackName=name, **kw)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/botocore/client.py", line 251, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/botocore/client.py", line 537, in _make_api_call
    raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (InsufficientCapabilitiesException) when calling the CreateStack operation: Requires capabilities : [CAPABILITY_IAM]
-------------------- >> begin captured logging << --------------------
@kapilt

This comment has been minimized.

Copy link
Collaborator

@kapilt kapilt commented Jan 9, 2017

this is more about aws cfn basics.. you have to pass --capabilities CAPABILITY_IAM on the cli, or equivalent in the api when creating the stack. lmgtfy ;-)

@chadwhitacre chadwhitacre mentioned this issue Jan 10, 2017
0 of 2 tasks complete
@chadwhitacre

This comment has been minimized.

Copy link
Contributor

@chadwhitacre chadwhitacre commented Jan 10, 2017

PR in #836.

@chadwhitacre chadwhitacre removed their assignment Jan 11, 2017
@chadwhitacre

This comment has been minimized.

Copy link
Contributor

@chadwhitacre chadwhitacre commented Feb 4, 2017

@chadwhitacre

This comment has been minimized.

Copy link
Contributor

@chadwhitacre chadwhitacre commented Feb 4, 2017

And cf. #903.

@chadwhitacre

This comment has been minimized.

Copy link
Contributor

@chadwhitacre chadwhitacre commented Apr 5, 2017

Organizations hit GA on Feb 27.

@kiwiz

This comment has been minimized.

Copy link
Contributor

@kiwiz kiwiz commented Jan 8, 2020

I'm interested in this functionality from a slightly different angle. When developing policies, it's helpful to have a static dump of API calls to test against. This insulates us from changes due to actions on cloud resources & speeds up iteration (no need to wait on API calls & you can inspect the response).

Looking at the docs, the record_flight_data functionality appears to do almost everything we'd need. Is there any interest in extract that functionality out and making it accessible via the main custodian run command? If not, I'd still like to submit a PR to add assume_role support to the PillTest class.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
blue team
Interested
4 participants
You can’t perform that action at this time.