Flag support. (Implementation 2) #30

scriptsrc · 2017-04-18T00:21:29Z

Just get WEBSITE and POLICY:

from cloudaux.orchestration.aws.s3 import get_bucket, FLAGS as FLAGS
bucket = get_bucket("zsh.sh", flags=FLAGS.WEBSITE | FLAGS.POLICY)

Result:

>>> print json.dumps(bucket, indent=2, sort_keys=True)
{
  "Policy": {
    "Statement": [
      {
        "Action": "s3:GetObject", 
        "Effect": "Allow", 
        "Principal": {
          "AWS": "*"
        }, 
        "Resource": "arn:aws:s3:::zsh.sh/*", 
        "Sid": "AddPerm"
      }
    ], 
    "Version": "2008-10-17"
  }, 
  "Website": {
    "ErrorDocument": {
      "Key": "404.html"
    }, 
    "IndexDocument": {
      "Suffix": "index.html"
    }
  }
}

Get the BASE as well:

bucket = get_bucket("zsh.sh", flags=FLAGS.BASE | FLAGS.WEBSITE | FLAGS.POLICY)                                                                                                                                  
print json.dumps(bucket, indent=2, sort_keys=True)

Result:

{
  "Arn": "arn:aws:s3:::zsh.sh", 
  "Policy": {
    "Statement": [
      {
        "Action": "s3:GetObject", 
        "Effect": "Allow", 
        "Principal": {
          "AWS": "*"
        }, 
        "Resource": "arn:aws:s3:::zsh.sh/*", 
        "Sid": "AddPerm"
      }
    ], 
    "Version": "2008-10-17"
  }, 
  "Region": "us-west-1", 
  "Website": {
    "ErrorDocument": {
      "Key": "404.html"
    }, 
    "IndexDocument": {
      "Suffix": "index.html"
    }
  }, 
  "_version": 5
}

from cloudaux.orchestration.aws.iam.role import get_role, FLAGS as ROLEFLAGS
role = get_role({'Arn': 'arn:aws:iam::xxxxxxxxxxxx:role/SecurityMonkey'}, flags=ROLEFLAGS.INSTANCE_PROFILES | ROLEFLAGS.MANAGED_POLICIES)

scriptsrc · 2017-04-18T00:22:53Z

Alternate implementation to #29.

mcpeak

Really nice enhancement! After staring at both for a while I definitely prefer this to the chain of if's.

mcpeak · 2017-04-18T04:09:20Z

cloudaux/orchestration/aws/s3.py

+        if flags & entry['flag']:
+            result.update({entry['key']: entry['method'](bucket_name, **conn)})
+
+    if flags & FLAGS.GRANTS:


Any way to get this to use the same approach as rest of flags?

See the comment for the rtv_ix. It would be nice to use the same approach. Maybe you can help figure out how to do it.

mcpeak · 2017-04-18T04:10:01Z

cloudaux/orchestration/aws/s3.py

+            if type(key) not in [list, tuple]: 
+                key_list = [key]
+            for idx in xrange(len(flag_list)):
+                cls.r.append(dict(flag=flag_list[idx], key=key_list[idx], method=fn, rtv_ix=idx))


Maybe add a comment as simple as an example line here just as a shortcut to future reviewers?

What's rtv_ix used for here?

good point - I initially wrote the FlagRegistry class to try and handle the get_grants() method, which is unique in that it has multiple return values that should get mapped to different fields in the return dictionary.

+# @FlagRegistry.register( +# flag=(FLAGS.GRANTS, FLAGS.GRANT_REFERENCES, FLAGS.OWNER), +# key=('grants', 'grant_references', 'owner')) +def get_grants(bucket_name, include_owner=True, **conn):

The rtv_ix was suppose to indicate the index of the return value. However, I kind of abandoned the idea of having FlagRegistry support these more complex interactions. I don't want to call the method three times.

scriptsrc · 2017-04-18T19:49:59Z

cloudaux/orchestration/aws/s3.py

 import logging
 import json


 logger = logging.getLogger('cloudaux')


-def get_grants(bucket_name, include_owner=False, **conn):
+FLAGS=Bunch(


Is Bunch okay or should we use something else.
Can IDE's autocomplete on bunch?

scriptsrc · 2017-04-21T16:57:33Z

Things to consider:

s3 GRANT flags share a code right now. (I'll push a change to fix this.)
Should FLAGS.BASE be a thing? Can you use get_* without BASE?
- What if you just want a user's access keys, but don't want the data returned by boto's get_user...
Should I force flags on the other two GCP technologies?
- If we make a FLAGS.BASE, then yes.
Bunch okay or should we use something else.
- Can IDE's autocomplete on bunch?

…ch from requirements

scriptsrc · 2017-04-21T20:30:33Z

S3 GRANT flags fixed.
FLAGS.BASE is now a thing. Yay.
I put FLAGS on all the techs.
I got rid of Bunch.

scriptsrc · 2017-04-21T21:23:42Z

@supertom - I think this is about ready.

I added flags to all three GCP technologies.

scriptsrc · 2017-04-21T23:05:07Z

FlagRegistry.build_out calls each method regardless of whether the corresponding flag is set.

Fix that before merge.

Run the security_monkey tests and watchers with this version and ensure things work as expected.

scriptsrc · 2017-04-21T23:07:56Z

include_created default should be False, but the flag default is FLAG.ALL.

Need to preserve this behavior.

…eturn values is required.

…rrecting include_created behavior for backwards compatibility.

scriptsrc · 2017-04-22T20:05:52Z

I fixed the build_out() method and the include_created argument/defaults.

Test on SM before merging.

scriptsrc · 2017-04-24T05:58:59Z

Tested on SM. Works great.

Spend some time making sure the GCP services work as expected and then we're good to go.

scriptsrc added 2 commits April 17, 2017 20:47

Initial commit of s3 flag support.

8b42a80

Adding flag registry

a8eaf3e

mcpeak reviewed Apr 18, 2017

View reviewed changes

scriptsrc commented Apr 18, 2017

View reviewed changes

scriptsrc added 7 commits April 19, 2017 17:08

Support for multiple return values.

1f17007

Adding FlagRegistry to IAM Role

94b6bad

Fixing naming for the FlagRegistry class.

bb617fb

Adding flag feature to IAM user

1efadb6

Adding flags to gcp.serviceaccount

acc5119

Documenting the flags.

d3da007

Bumping version to 1.2.0

a8bb586

scriptsrc requested a review from mikegrima April 21, 2017 00:18

scriptsrc added 2 commits April 21, 2017 17:40

Making S3 GRANTS, GRANT_REFERENCES, and OWNER flags separate values.

dd4a0a6

Adding Flag class. Adding BASE flag to all technologies. Removing bun…

2b4cf81

…ch from requirements

scriptsrc changed the title ~~[WIP] Initial commit of s3 flag support. (Implementation 2)~~ Flag support. (Implementation 2) Apr 21, 2017

Updating Flag section in README's

0580c08

scriptsrc added 4 commits April 22, 2017 19:39

Updating FlagRegistry.build_out to only call a method if one of the r…

c369f7e

…eturn values is required.

whitespace

eae66d7

Removing FLAGS.CREATED_DATE from default S3 flags in get_bucket(). Co…

9b33358

…rrecting include_created behavior for backwards compatibility.

Removing include_created flag from GCS Bucket get_bucket method

30e0d0c

scriptsrc mentioned this pull request Apr 24, 2017

[WIP] Initial commit of s3 flag support. #29

Closed

Merge branch 'master' into issue_7_flags_2

943d0b5

scriptsrc merged commit fe0ef9f into master Apr 27, 2017

scriptsrc mentioned this pull request Apr 27, 2017

optional fields param in orchestration #7

Closed

scriptsrc deleted the issue_7_flags_2 branch November 25, 2017 07:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flag support. (Implementation 2) #30

Flag support. (Implementation 2) #30

scriptsrc commented Apr 18, 2017 •

edited

Loading

scriptsrc commented Apr 18, 2017

mcpeak left a comment

mcpeak Apr 18, 2017

scriptsrc Apr 18, 2017

mcpeak Apr 18, 2017

mcpeak Apr 18, 2017

scriptsrc Apr 18, 2017

scriptsrc Apr 18, 2017 •

edited

Loading

scriptsrc commented Apr 21, 2017

scriptsrc commented Apr 21, 2017

scriptsrc commented Apr 21, 2017

scriptsrc commented Apr 21, 2017

scriptsrc commented Apr 21, 2017

scriptsrc commented Apr 22, 2017

scriptsrc commented Apr 24, 2017

Flag support. (Implementation 2) #30

Flag support. (Implementation 2) #30

Conversation

scriptsrc commented Apr 18, 2017 • edited Loading

scriptsrc commented Apr 18, 2017

mcpeak left a comment

Choose a reason for hiding this comment

mcpeak Apr 18, 2017

Choose a reason for hiding this comment

scriptsrc Apr 18, 2017

Choose a reason for hiding this comment

mcpeak Apr 18, 2017

Choose a reason for hiding this comment

mcpeak Apr 18, 2017

Choose a reason for hiding this comment

scriptsrc Apr 18, 2017

Choose a reason for hiding this comment

scriptsrc Apr 18, 2017 • edited Loading

Choose a reason for hiding this comment

scriptsrc commented Apr 21, 2017

scriptsrc commented Apr 21, 2017

scriptsrc commented Apr 21, 2017

scriptsrc commented Apr 21, 2017

scriptsrc commented Apr 21, 2017

scriptsrc commented Apr 22, 2017

scriptsrc commented Apr 24, 2017

scriptsrc commented Apr 18, 2017 •

edited

Loading

scriptsrc Apr 18, 2017 •

edited

Loading