SecurityMonkey for GovClod region #247

skdubey · 2015-11-09T02:07:31Z

Hi,

I am trying to setup SecurityMonkey for GovCloud region, but later found that non of the region is being listed in Dashboard. Is there any change require to make it work on GovCloud ?

mikegrima · 2015-11-09T02:27:56Z

Hi @skdubey, we would love to do some additional testing to get it to work with GovCloud.

Can you provide additional details on what you are seeing?

mikegrima · 2015-11-09T02:29:30Z

One major challenge is that only a small subset of AWS technologies are available in GovCloud, so you may see some missing functionality as a result.

mikegrima · 2015-11-09T02:33:29Z

You may need to do some refactoring...

For example:

security_monkey/security_monkey/constants.py

Line 25 in 03c1594

TROUBLE_REGIONS = ['cn-north-1', 'us-gov-west-1']

You would need to remove the govcloud region from the TROUBLED_REGIONS constant.

skdubey · 2015-11-09T02:38:11Z

Thanks for the quick reply,

But i am not able to see anything in Dashboard, nor any server log which should say that it is attempting to do something on system. Services are running fine.

skdubey · 2015-11-09T02:40:37Z

Hi Mikegrima,

What is ROUBLED_REGIONS ? Does it means, all the regions mentioned in this list, system will not check for those regions ?

mikegrima · 2015-11-09T02:47:23Z

@skdubey : Take, for example the code here:

security_monkey/security_monkey/watchers/sqs.py

Lines 58 to 67 in 03c1594

    
           try: 
        
               sqs = connect(account, 'sqs', region=region) 
        
               all_queues = self.wrap_aws_rate_limited_call( 
        
                   sqs.get_all_queues 
        
               ) 
        
           except Exception as e: 
        
               if region.name not in TROUBLE_REGIONS: 
        
                   exc = BotoConnectionIssue(str(e), 'sqs', account, region.name) 
        
                   self.slurp_exception((self.index, account, region.name), exc, exception_map) 
        
               continue

If an Exception is raised, it will skip over the regions defined in TROUBLED_REGIONS, which by default, we have the GovCloud region in there.

What happens if you remove the GovCloud region from that constant? I would try that first, and post the results.

skdubey · 2015-11-09T02:50:22Z

@mikegrima : Thanks, i will update you the result after making changes shortly.

skdubey · 2015-11-09T07:10:47Z

Hello Mike,

Which file is this? I tried to find this string globally in security_monkey
directory and even in source, but couldn't find it.

On Mon, Nov 9, 2015 at 8:17 AM, Mike Grima notifications@github.com wrote:

@skdubey https://github.com/skdubey : Take, for example the code here:

security_monkey/security_monkey/watchers/sqs.py

Lines 58 to 67 in 03c1594

try:

sqs = connect(account, 'sqs', region=region)

all_queues = self.wrap_aws_rate_limited_call(

sqs.get_all_queues

)

except Exception as e:

if region.name not in TROUBLE_REGIONS:

exc = BotoConnectionIssue(str(e), 'sqs', account, region.name)

self.slurp_exception((self.index, account, region.name), exc, exception_map)

continue

If an Exception is raised, it will skip over the regions defined in
TROUBLED_REGIONS, which by default, we have the GovCloud region in there.

What happens if you remove the GovCloud region from that constant? I would
try that first, and post the results.

—
Reply to this email directly or view it on GitHub
#247 (comment)
.

Regards,
Sandeep

skdubey · 2015-11-09T07:13:38Z

Sorry got it "constants.py". Should i restart the service to take effect or
rebuild it from source after making this change ?

On Mon, Nov 9, 2015 at 12:40 PM, sandeep dubey sandeep.sanash@gmail.com
wrote:

Hello Mike,

Which file is this? I tried to find this string globally in
security_monkey directory and even in source, but couldn't find it.

On Mon, Nov 9, 2015 at 8:17 AM, Mike Grima notifications@github.com
wrote:

@skdubey https://github.com/skdubey : Take, for example the code here:

security_monkey/security_monkey/watchers/sqs.py

Lines 58 to 67 in 03c1594

try:

sqs = connect(account, 'sqs', region=region)

all_queues = self.wrap_aws_rate_limited_call(

sqs.get_all_queues

)

except Exception as e:

if region.name not in TROUBLE_REGIONS:

exc = BotoConnectionIssue(str(e), 'sqs', account, region.name)

self.slurp_exception((self.index, account, region.name), exc, exception_map)

continue

If an Exception is raised, it will skip over the regions defined in
TROUBLED_REGIONS, which by default, we have the GovCloud region in there.

What happens if you remove the GovCloud region from that constant? I
would try that first, and post the results.

—
Reply to this email directly or view it on GitHub
#247 (comment)
.

Regards,
Sandeep

Regards,
Sandeep

skdubey · 2015-11-09T07:31:49Z

Hello Mike,

After removing the govcloud region and restarting the supervisor, i am
getting "Error loading resource from API. Error: " error in console but no log in access/error.log file.

On Mon, Nov 9, 2015 at 12:43 PM, sandeep dubey sandeep.sanash@gmail.com
wrote:

Sorry got it "constants.py". Should i restart the service to take effect
or rebuild it from source after making this change ?

On Mon, Nov 9, 2015 at 12:40 PM, sandeep dubey sandeep.sanash@gmail.com
wrote:

Hello Mike,

Which file is this? I tried to find this string globally in
security_monkey directory and even in source, but couldn't find it.

On Mon, Nov 9, 2015 at 8:17 AM, Mike Grima notifications@github.com
wrote:

@skdubey https://github.com/skdubey : Take, for example the code here:

security_monkey/security_monkey/watchers/sqs.py

Lines 58 to 67 in 03c1594

try:

sqs = connect(account, 'sqs', region=region)

all_queues = self.wrap_aws_rate_limited_call(

sqs.get_all_queues

)

except Exception as e:

if region.name not in TROUBLE_REGIONS:

exc = BotoConnectionIssue(str(e), 'sqs', account, region.name)

self.slurp_exception((self.index, account, region.name), exc, exception_map)

continue

If an Exception is raised, it will skip over the regions defined in
TROUBLED_REGIONS, which by default, we have the GovCloud region in
there.

What happens if you remove the GovCloud region from that constant? I
would try that first, and post the results.

—
Reply to this email directly or view it on GitHub
#247 (comment)
.

Regards,
Sandeep

Regards,
Sandeep

Regards,
Sandeep

scriptsrc · 2015-11-09T18:48:50Z

Hey @skdubey - We'd love to be able to get security_monkey to work in govcloud. Unfortunately, we don't have access to it. If you'd feel comfortable doing a google hangout sometime so we can try and work through the issues, that would be great for the project.

Do you know if GovCloud is going to support Lambda in the future? I don't see it on the current list:

https://aws.amazon.com/govcloud-us/details/

dischord01 · 2015-11-16T19:24:15Z

Im at the same point working with Security Monkey in GovCloud. I would like to try a Google Hangout to try to resolve this. I have the web interface up and running but nothing is being returned. I removed the GovCloud entry from troubled regions and still SM returns nothing.

For Lambda, not sure if they will or not. Its kind of driven from user demand what they put in GovCloud.

skdubey · 2015-11-17T03:58:44Z

Sorry Patrick for late reply, I was traveling.

Let me know what is the right time to skype/hangout ?

Regarding Lambda, not sure when AWS will introduce this service in
Govcloud, as there are other services would be in their priority list.

On Tue, Nov 10, 2015 at 12:19 AM, Patrick Kelley notifications@github.com
wrote:

Hey @skdubey https://github.com/skdubey - We'd love to be able to get
security_monkey to work in govcloud. Unfortunately, we don't have access to
it. If you'd feel comfortable doing a google hangout sometime so we can try
and work through the issues, that would be great for the project.

Do you know if GovCloud is going to support Lambda in the future? I don't
see it on the current list:

https://aws.amazon.com/govcloud-us/details/

—
Reply to this email directly or view it on GitHub
#247 (comment)
.

Regards,
Sandeep

hellovadi · 2015-11-17T19:34:59Z

@skdubey the error message: "Error loading resource from API. Error: "
suggests the Security Monkey is trying to reach an endpoint that GovCloud has no idea of. In Security Monkey code, I see comments like:

--# If we can't get the region, default to us-east-1 so we can fillout

For example in your case, Security Monkey is trying to connect to ec2.us-east-1.amazonaws.com when it is supposed to connect to ec2.us-gov-west-1.amazonaws.com. GovCloud knows nothing about the commercial regions. A possible fix is to query the metadata server for availability-zone and then set the default region.

skdubey · 2015-11-25T08:55:50Z

@hellovadi, I replaced the default region in s3.py and restarted the
supervisor service but still not able to get this region. I am not a
developer so understanding this is little bit tough for me.

@patrick, Should we go on remote session to work on this if you have time.

On Wed, Nov 18, 2015 at 1:05 AM, hellovadi notifications@github.com wrote:

@skdubey https://github.com/skdubey the error message: "Error loading
resource from API. Error: reachable.>"
suggests the Security Monkey is trying to reach an endpoint that GovCloud
has no idea of. In Security Monkey code, I see comments like:
If we can't get the region, default to us-east-1 so we can fillout

For example in your case, Security Monkey is trying to connect to
ec2.us-east-1.amazonaws.com when it is supposed to connect to
ec2.us-gov-west-1.amazonaws.com. GovCloud knows nothing about the
commercial regions. A possible fix is to query the metadata server for
availability-zone and then set the default region.

—
Reply to this email directly or view it on GitHub
#247 (comment)
.

Regards,
Sandeep

skdubey · 2015-11-25T09:01:51Z

Just noticed below error. Seems that i should replace all the connection
url to govcloud url ?

2015-11-25 08:48:50,173 INFO: 0 deleted Security Groups in
[u'inSyncGovCloud'] [in
/usr/local/src/security_monkey/security_monkey/watcher.py:347]
2015-11-25 08:48:50,173 INFO: 0 created Security Groups in
[u'inSyncGovCloud'] [in
/usr/local/src/security_monkey/security_monkey/watcher.py:348]
2015-11-25 08:48:50,174 INFO: 0 changed Security Groups in
[u'inSyncGovCloud'] [in
/usr/local/src/security_monkey/security_monkey/watcher.py:358]
2015-11-25 08:48:50,174 INFO: Account inSyncGovCloud is done with Security
Group [in /usr/local/src/security_monkey/security_monkey/reporter.py:67]
2015-11-25 08:48:50,174 INFO: Running ELB for inSyncGovCloud (15 minutes
interval) [in /usr/local/src/security_monkey/security_monkey/reporter.py:55]
2015-11-25 08:48:50,520 ERROR: Job "run_change_reporter (trigger:
interval[0:15:00], next run at: 2015-11-25 09:03:34.330395)" raised an
exception [in build/bdist.linux-x86_64/egg/apscheduler/scheduler.py:520]
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/apscheduler/scheduler.py", line 512,
in _run_job
retval = job.func(_job.args, *_job.kwargs)
File "/usr/local/src/security_monkey/security_monkey/scheduler.py", line
46, in run_change_reporter
reporter.run(account, interval)
File "/usr/local/src/security_monkey/security_monkey/reporter.py", line
56, in run
(items, exception_map) = watcher.slurp()
File "/usr/local/src/security_monkey/security_monkey/watchers/elb.py",
line 127, in slurp
self._setup_botocore(account)
File "/usr/local/src/security_monkey/security_monkey/watchers/elb.py",
line 81, in _setup_botocore
self.botocore_session = connect(account, 'botocore')
File
"/usr/local/src/security_monkey/security_monkey/common/sts_connect.py",
line 61, in connect
role = sts.assume_role('arn:aws:iam::' + account.number + ':role/' +
role_name, 'secmonkey')
File
"/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/sts/connection.py",
line 384, in assume_role
return self.get_object('AssumeRole', params, AssumedRole, verb='POST')
File
"/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/connection.py",
line 1207, in get_object
raise self.ResponseError(response.status, response.reason, body)
BotoServerError: BotoServerError: 403 Forbidden

Sender
InvalidClientTokenId
The security token included in the request is invalid

430679e9-9351-11e5-bf63-5b75a543508e

On Wed, Nov 25, 2015 at 2:25 PM, sandeep dubey sandeep.sanash@gmail.com
wrote:

@hellovadi, I replaced the default region in s3.py and restarted the
supervisor service but still not able to get this region. I am not a
developer so understanding this is little bit tough for me.

@patrick, Should we go on remote session to work on this if you have
time.

On Wed, Nov 18, 2015 at 1:05 AM, hellovadi notifications@github.com
wrote:

@skdubey https://github.com/skdubey the error message: "Error loading
resource from API. Error: reachable.>"
suggests the Security Monkey is trying to reach an endpoint that GovCloud
has no idea of. In Security Monkey code, I see comments like:
If we can't get the region, default to us-east-1 so we can fillout

For example in your case, Security Monkey is trying to connect to
ec2.us-east-1.amazonaws.com when it is supposed to connect to
ec2.us-gov-west-1.amazonaws.com. GovCloud knows nothing about the
commercial regions. A possible fix is to query the metadata server for
availability-zone and then set the default region.

—
Reply to this email directly or view it on GitHub
#247 (comment)
.

Regards,
Sandeep

Regards,
Sandeep

hellovadi · 2015-11-25T23:33:19Z

@skdubey you need to work with a contributor to fix this. When I run:

find src/security_monkey -type f | xargs grep us-east-1

there are many hits. In many cases SM just assumes us-east-1 as the default region. If you want a quick turnaround: clone SM repo, edit the source to handle every line of the output of the command I have given above. Sorry I used to have GovCloud access, now I do not.

scriptsrc · 2016-08-04T23:27:29Z

Closing as "wontfix".

Really, I'd love for it to run in govcloud. I'd love to have a PR for this. I don't have the ability to do so myself, and Netflix doesn't need it.

scriptsrc added the wontfix label Aug 4, 2016

scriptsrc closed this as completed Aug 4, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SecurityMonkey for GovClod region #247

SecurityMonkey for GovClod region #247

skdubey commented Nov 9, 2015

mikegrima commented Nov 9, 2015

mikegrima commented Nov 9, 2015

mikegrima commented Nov 9, 2015

skdubey commented Nov 9, 2015

skdubey commented Nov 9, 2015

mikegrima commented Nov 9, 2015

skdubey commented Nov 9, 2015

skdubey commented Nov 9, 2015

skdubey commented Nov 9, 2015

skdubey commented Nov 9, 2015

scriptsrc commented Nov 9, 2015

dischord01 commented Nov 16, 2015

skdubey commented Nov 17, 2015

hellovadi commented Nov 17, 2015

skdubey commented Nov 25, 2015

skdubey commented Nov 25, 2015

hellovadi commented Nov 25, 2015

scriptsrc commented Aug 4, 2016

SecurityMonkey for GovClod region #247

SecurityMonkey for GovClod region #247

Comments

skdubey commented Nov 9, 2015

mikegrima commented Nov 9, 2015

mikegrima commented Nov 9, 2015

mikegrima commented Nov 9, 2015

skdubey commented Nov 9, 2015

skdubey commented Nov 9, 2015

mikegrima commented Nov 9, 2015

skdubey commented Nov 9, 2015

skdubey commented Nov 9, 2015

skdubey commented Nov 9, 2015

skdubey commented Nov 9, 2015

scriptsrc commented Nov 9, 2015

dischord01 commented Nov 16, 2015

skdubey commented Nov 17, 2015

hellovadi commented Nov 17, 2015

skdubey commented Nov 25, 2015

skdubey commented Nov 25, 2015

hellovadi commented Nov 25, 2015

scriptsrc commented Aug 4, 2016