Cannot reach web service

[westlifezs]

Hello,

I followed the instruction but I cannot reach the webservice.
I checked the security group and set it to public.
but it still says "this page is not available".
Could anyone let me know what could be the reason?

Here is part of the process list that relevant to security monkey within in box:

postgres 13821     1  0 00:23 ?        00:00:05 /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
postgres 13823 13821  0 00:23 ?        00:00:00 postgres: checkpointer process
postgres 13824 13821  0 00:23 ?        00:00:00 postgres: writer process
postgres 13825 13821  0 00:23 ?        00:00:00 postgres: wal writer process
postgres 13826 13821  0 00:23 ?        00:00:02 postgres: autovacuum launcher process
postgres 13827 13821  0 00:23 ?        00:00:00 postgres: stats collector process
root     14575     2  0 00:27 ?        00:00:00 [kworker/u30:2]
root     14947     2  0 00:47 ?        00:00:00 [kworker/u30:1]
root     15190     1  0 01:48 ?        00:00:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
www-data 15192 15190  0 01:48 ?        00:00:01 python /usr/local/src/security_monkey/manage.py run_api_server
www-data 15203 15192  0 01:48 ?        00:00:00 python /usr/local/src/security_monkey/manage.py run_api_server
www-data 15205 15192  0 01:48 ?        00:00:00 python /usr/local/src/security_monkey/manage.py run_api_server
www-data 15206 15192  0 01:48 ?        00:00:00 python /usr/local/src/security_monkey/manage.py run_api_server
www-data 15208 15192  0 01:48 ?        00:00:00 python /usr/local/src/security_monkey/manage.py run_api_server
www-data 15209 15192  0 01:48 ?        00:00:00 python /usr/local/src/security_monkey/manage.py run_api_server
www-data 15212 15192  0 01:48 ?        00:00:00 python /usr/local/src/security_monkey/manage.py run_api_server
root     15322     1  0 02:00 ?        00:00:00 nginx: master process /usr/sbin/nginx
www-data 15325 15322  0 02:00 ?        00:00:00 nginx: worker process
www-data 15326 15322  0 02:00 ?        00:00:00 nginx: worker process
www-data 15327 15322  0 02:00 ?        00:00:00 nginx: worker process
www-data 15328 15322  0 02:00 ?        00:00:00 nginx: worker process
root     15424  1075  0 02:33 ?        00:00:00 sshd: ubuntu [priv]

Thanks and look forward to hearing from you,

[mikegrima]

Hello @westlifezs

Can you ssh onto your instance and run sudo netstat -tlpn | grep nginx? Then, while on that instance, try to curl localhost on the listed port numbers. You should get some HTML output representing Security Monkey.

If that works, then check if you have firewall rules on your instance blocking inbound access externally to that port (use sudo iptables -L).

Lastly, please verify that your security group is ingressing the proper TCP port that you have enabled in your NGINX configuration.

Please let us know if you are still having issues.

Thank You

[westlifezs]

Mike,

thanks for your response. I figured out the issue eventually. I should use https rather than http. However, I came across another issue. Now I can see the web interface. I could not see anything by clicking "search". However, I have already setup the account by following the wiki instructions. is there any other approach that I can verify where is the root cause for this issue?

Thanks,

Su

[mikegrima]

Hello @westlifezs

When you say that you set up an account, are you referring to the creation of a username/password for you to log in as, or are you referring to an AWS account to watch and audit?

Whenever you add or remove AWS accounts to audit, the scheduler must be restarted.

Thank You

[westlifezs]

I am talking about AWS account. I did restart the scheduler but I still can see nothing under the search button. is there anything I can do to diagnose this issue?

thanks.

[mikegrima]

Hello @westlifezs

Can you check out the logs? Assuming you are using the sample configuration in the quickstart, the file is /var/log/security_monkey/security_monkey-deploy.log.

If it is working properly, you should see something along the lines of: Slurping ....

[westlifezs]

Hi Mike,

Thanks for pointing me to the log.
I checked the log and found the following error. any hints? thx

2016-02-11 06:48:03,299 WARNING: Failed to connect to ses using boto. Check your boto credentials. <type 'exceptions.Exception'> 'AccessKeyId' [in /usr/local/src/security_monkey/security_monkey/common/ut\
ils.py:87]
2016-02-11 06:48:03,305 WARNING: Traceback (most recent call last):
  File "/usr/local/src/security_monkey/security_monkey/common/utils.py", line 84, in send_email
    ses = boto.ses.connect_to_region(ses_region)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/ses/__init__.py", line 51, in connect_to_region
    return region.connect(**kw_params)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/regioninfo.py", line 187, in connect
    return self.connection_cls(region=self, **kw_params)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/ses/connection.py", line 57, in __init__
    profile_name=profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/connection.py", line 555, in __init__
    profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/provider.py", line 200, in __init__
    self.get_credentials(access_key, secret_key, security_token, profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/provider.py", line 376, in get_credentials
    self._populate_keys_from_metadata_server()
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/provider.py", line 395, in _populate_keys_from_metadata_server
    self._access_key = security['AccessKeyId']
KeyError: 'AccessKeyId'
 [in /usr/local/src/security_monkey/security_monkey/common/utils.py:88]
2016-02-11 06:49:03,658 WARNING: Failed to connect to ses using boto. Check your boto credentials. <type 'exceptions.Exception'> 'AccessKeyId' [in /usr/local/src/security_monkey/security_monkey/common/ut\
ils.py:87]
2016-02-11 06:49:03,659 WARNING: Traceback (most recent call last):
  File "/usr/local/src/security_monkey/security_monkey/common/utils.py", line 84, in send_email
    ses = boto.ses.connect_to_region(ses_region)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/ses/__init__.py", line 51, in connect_to_region
    return region.connect(**kw_params)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/regioninfo.py", line 187, in connect
    return self.connection_cls(region=self, **kw_params)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/ses/connection.py", line 57, in __init__
    profile_name=profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/connection.py", line 555, in __init__
    profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/provider.py", line 200, in __init__
    self.get_credentials(access_key, secret_key, security_token, profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/provider.py", line 376, in get_credentials
    self._populate_keys_from_metadata_server()
  File "/usr/local/lib/python2.7/dist-packages/boto-2.36.0-py2.7.egg/boto/provider.py", line 395, in _populate_keys_from_metadata_server
self._access_key = security['AccessKeyId']
KeyError: 'AccessKeyId'

[westlifezs]

I did not find anywhere mentioned AccessKeyId in http://securitymonkey.readthedocs.org/en/latest/index.html
Shall I specify it somewhere?

[mikegrima]

Security Monkey utilizes AWS on-instance credentials that are provided automatically to your instance. The Boto API knows how to obtain them.

Did you create the IAM roles and launch your instance with the Instance Profile specified in the Quickstart document?

You can find out which IAM role your instance is running with by executing the command:
curl http://169.254.169.254/latest/meta-data/iam/info/

(Please do not paste in the output from that command in here!)

[westlifezs]

curl http://PUBLIC_IP/latest/meta-data/iam/info/
curl: (7) Failed to connect to PUBLIC_IP port 80: Connection refused

I am showing the error above. I think this is because that I am using https.
Is there other ways to get this issue around?

thx

[westlifezs]

I checked through web UI and it shows:

IAM role SecurityMonkeyInstanceProfile

Am I right? I strictly followed the instruction while setting this up.

[mikegrima]

The curl command needs to be executed on the instance itself. The IP in the command: 169.254.169.254 gets information from the AWS on-instance metadata service.

Can you verify the launch configuration of your instance and check that it is launched into the SecurityMonkeyInstanceProfile?

[westlifezs]

I got the same error while running inside the box. either through sudo or not.

what do you mean by launch configuration? is it "SecurityMonkeyLaunchPerms" mentioned in the documentation? if so, then it will be the same as the documentation.

thx

[westlifezs]

by the way, shall I have a bogo config file in my box? I did not find the file. also security monkey does not mention anything about bogo. I checked bogo doc and it seems like such a config is needed. Could you let me know what the bogo config looks like in a healthy box? under which directory and what's the format? I am running ubuntu 1404 LTS. thx

[mikegrima]

Hi @westlifezs

Open your AWS account console, then go to the EC2 Console. In there, select "instances" and search for the instance that is running Security Monkey. When you select the instance, there should be some information about the instance on the bottom of the page.

What is listed under "IAM Role"?

Boto is the AWS Python API that is used by Security Monkey to fetch information from AWS. It uses the on-instance metadata service to obtain the credentials -- you do not need to place any files on the instance with credentials. If the instance is not launched with the IAM role, then it will not be able to communicate with the on-instance metadata service and will thus fail to obtain AWS credentials to make API calls.

[westlifezs]

Sorry I misunderstood your explanation in my previous post.
I can successfully retrieve my credential from the curl call.

ubuntu@:~$ curl http://169.254.169.254/latest/meta-data/iam/info/
{
  "Code" : "Success",
  "LastUpdated" : "2016-02-12T00:32:15Z",
  "InstanceProfileArn" : "arn:aws:iam::ACCOUNT_ID:instance-profile/SecurityMonkeyInstanceProfile",
  "InstanceProfileId" : "LONG_STRING"
}

Does this look right?

Thx

[westlifezs]

as I mentioned earlier already, the IAM role is shown "SecurityMonkeyInstanceProfile" through web console. anywhere else I should look into?

thx

[mikegrima]

That looks correct.

The next question is: are your roles configured properly?

Did you add the SecurityMonkeyInstanceProfile to the Trust Relationship of the SecurityMonkey role in the destination account? The SecurityMonkeyInstanceProfile also needs to have sts:AssumeRole access the destination account's SecurityMonkey IAM Role.

[westlifezs]

I think so.
Here is the trust policy of securityMonkey role.

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT_ID:role/SecurityMonkeyInstanceProfile"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Anything else I can check?

thx!

[mikegrima]

That looks correct. Can you check the SecurityMonkeyInstanceProfile, and check that it has sts:AssumeRole on the SecurityMonkey role ARN?

[westlifezs]

Here is it. Shall I put securitymonkey instead of "security_monkey_docker_user" in the file?

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT_ID:user/security_monkey_docker_user"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

[mikegrima]

Can you explain your setup a bit?

For example, are you running this in a Docker container?

[westlifezs]

no. it is running on an instance. I think this is probably the root cause. thanks for helping me to identify the issue.
However, while I am trying to update the trust policy by replacing "security_monkey_docker_user" with "SecurityMonkey", it cannot succeed. It will not update at all. also no error message is given after that. Do you have any suggestions to fix this issue?

Thanks,

[mikegrima]

Do you have an IAM role named SecurityMonkey in that account?

[westlifezs]

I think so. I only have one account. while clicking "roles", I can see both "SecurityMonkey" and "SecurityMonkeyInstanceProfile" in the list. does it imply SecurityMonkey role belongs to my account? Or is there another way to check it? Sorry I have not been using AWS for a long while.

[mikegrima]

Quick question: in the output above, you have ACCOUNT_ID. Did you copy and paste that directly in here, or did you remove the actual account id from there?

[mikegrima]

Another thing: the ARN for this role should be:
arn:aws:iam::PUT_YOUR_ACCOUNT_ID_HERE_WITHOUT_DASHES:role/SecurityMonkey

[westlifezs]

I purposely hided our account_id. it should be a numeric value.

[westlifezs]

it seems like the error triggered by the following function within /usr/local/src/security_monkey/security_monkey/common/utils.py

def send_email(subject=None, recipients=[], html=""):
"""
Given a message, will send that message over SES or SMTP, depending upon how the app is configured.
"""
plain_txt_email = "Please view in a mail client that supports HTML."
if app.config.get('EMAILS_USE_SMTP'):
try:
with app.app_context():
msg = Message(subject, recipients=recipients)
msg.body = plain_txt_email
msg.html = html
mail.send(msg)
app.logger.debug("Emailed {} - {} ".format(recipients, subject))
except Exception, e:
m = "Failed to send failure message with subject: {}\n{} {}".format(subject, Exception, e)
app.logger.warn(m)
app.logger.warn(traceback.format_exc())

else:
    try:
        ses_region = app.config.get('SES_REGION', 'us-east-1')
        ses = boto.ses.connect_to_region(ses_region)
    except Exception, e:
        m = "Failed to connect to ses using boto. Check your boto credentials. {} {}".format(Exception, e)
        app.logger.warn(m)
        app.logger.warn(traceback.format_exc())
        return

    for email in recipients:
        try:
            ses.send_email(app.config.get('MAIL_DEFAULT_SENDER'), subject, html, email, format="html")
            app.logger.debug("Emailed {} - {} ".format(email, subject))
        except Exception, e:
            m = "Failed to send failure message with subject: {}\n{} {}".format(subject, Exception, e)
            app.logger.warn(m)
            app.logger.warn(traceback.format_exc())

My question is, is there a way to turn off SES or I have to configure it?

[westlifezs]

also may I know which way is the easiest way to restart security monkey?
Shall it be just "sudo service supervisor restart" or I need to also restart web service?
I want to hardcode access key id as a workaround if it is not configurable.

thanks,

[mikegrima]

I'm interested in knowing why you are unable to fetch credentials from the instance.

[westlifezs]

is there a way that I can test at lower level regarding fetching such credentials?
For example by using curl calls or client?

curl http://169.254.169.254/latest/meta-data/iam/info/ cannot provide access key id, right?

[mikegrima]

Hi @westlifezs :

Can you open the python console on the instance, and paste in the following (add in your account number):

import boto3

client = boto3.client('sts')
ar = client.assume_role(RoleArn="arn:aws:iam::INSERT_YOUR_ACCOUNT_ID_HERE:role/SecurityMonkey", RoleSessionName="test")

print(ar)

Do not paste the output in here, but it should get back a dict-like structure. If that raises an exception, please let me know.

[westlifezs]

the exception is the same as we have seen in the log:

Traceback (most recent call last):
  File "netflix.py", line 3, in <module>
    client = boto3.client('sts')
  File "/usr/local/lib/python2.7/dist-packages/boto3-1.2.1-py2.7.egg/boto3/__init__.py", line 79, in client
    return _get_default_session().client(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/boto3-1.2.1-py2.7.egg/boto3/session.py", line 199, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python2.7/dist-packages/botocore-1.3.7-py2.7.egg/botocore/session.py", line 754, in create_client
    credentials = self.get_credentials()
  File "/usr/local/lib/python2.7/dist-packages/botocore-1.3.7-py2.7.egg/botocore/session.py", line 413, in get_credentials
    'credential_provider').load_credentials()
  File "/usr/local/lib/python2.7/dist-packages/botocore-1.3.7-py2.7.egg/botocore/credentials.py", line 621, in load_credentials
    creds = provider.load()
  File "/usr/local/lib/python2.7/dist-packages/botocore-1.3.7-py2.7.egg/botocore/credentials.py", line 300, in load
    metadata = fetcher.retrieve_iam_role_credentials()
  File "/usr/local/lib/python2.7/dist-packages/botocore-1.3.7-py2.7.egg/botocore/utils.py", line 211, in retrieve_iam_role_credentials
    'access_key': data[role_name]['AccessKeyId'],
KeyError: 'AccessKeyId'

[westlifezs]

I only replaced "INSERT_YOUR_ACCOUNT_ID_HERE" with our account id. anything else shall I replace within the script?

[mikegrima]

It seems that you have an issue with your setup, as the AWS SDK's are unable to fetch the on-instance credentials.

Are you using an AWS provided instance image, or are you using your own?

[westlifezs]

also is this region independent? we only have us-east1 region enabled but nothing else.

[westlifezs]

I think I used AWS provide image.
here is the details:
AMI ID ubuntu-trusty-14.04-amd64-server-20160114.5

Does it look right?

[mikegrima]

Do you have the aws CLI installed? Can you run aws help?

[westlifezs]

I just installed. However, it still does not work after I installed.

[westlifezs]

I booted another fresh ubuntu image
ubuntu-trusty-14.04-amd64-server-20151117-ecd5575e-d805-450e-843e-f2a9872b8c80-ami-0a8acf60.3 (ami-86e0ffe7)

The error info is the same after I run your test python script.
I installed both bob3 and aws cli.

[westlifezs]

which boto version is supported by security monkey?
Here is our version
python -c "import boto; print boto.Version"
2.36.0

[mikegrima]

What happens when you:
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/SecurityMonkeyInstanceProfile ?

Does that print out credentials?

Security Monkey uses both Boto 2 and 3.

[westlifezs]

sudo curl http://169.254.169.254/latest/meta-data/iam/security-credentials/SecurityMonkeyInstanceProfile
{
  "Code" : "AssumeRoleUnauthorizedAccess",
  "Message" : "EC2 cannot assume the role SecurityMonkeyInstanceProfile.  Please see documentation at http://docs.amazonwebservices.com/IAM/latest/UserGuide/RolesTroubleshooting.html.",
  "LastUpdated" : "2016-02-12T23:49:55Z"
}

[mikegrima]

In the IAM console, open the Trust Policy for SecurityMonkeyInstanceProfile. It should allow the principal: ec2.amazonaws.com in.

[westlifezs]

this is the trust policy, is it right?

{

  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::XXXXXXXX:role/SecurityMonkey"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

[mikegrima]

That is a trust policy, but it is incorrect. -- That is what the trust policy should be for SecurityMonkey.

SecurityMonkeyInstanceProfile needs the following Trust Policy:

      {
        "Action": "sts:AssumeRole",
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
          "Service": "ec2.amazonaws.com"
        }
      }

That will allow the instance to get the credentials for the role.

[westlifezs]

so my SecurityMonkey trust policy is:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::OUR_NUMERIC_ACCOUNT_ID:role/SecurityMonkeyInstanceProfile"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

and my securityMonkeyProfile trust policy is

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Is this right?

[mikegrima]

Yes, that is correct.

[westlifezs]

cool thanks a lot. I got the credential. But the server is still hitting the error. I think I should restart the server to make this effective. Shall I install restart just scheduler or supervisor or something else? thanks

[mikegrima]

Restart the scheduler.

[westlifezs]

I did
sudo supervisorctl
restart securitymonkeyscheduler
ctrl+d
tail -f /var/log/security_monkey/security_monkey-deploy.log

and it still shows the same error.
anything else I need to do to restart the scheduler?
thx

[mikegrima]

Reboot the instance.

[westlifezs]

it seems like the previous error has gone as there is nothing shown in the deploy log file in the last 30 minutes. However, I see another error in security_monkey.error.log and it shows:

2016/02/13 00:17:20 [error] 1170#0: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 198.6.50.15, server: , request: "GET /api/1/items?accounts=&count=25&names=&page=1&regions=&technologies= HTTP/1.1", upstream: "http://127.0.0.1:5000/api/1/items?accounts=&count=25&names=&page=1&regions=&technologies=", host: "XXXXXXXX", referrer: "XXXXXXXX"
2016/02/13 00:22:26 [error] 1170#0: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 198.6.50.15, server: , request: "GET /api/1/issues?accounts=&active=True&count=25&enabledonly=true&names=&page=1&regions=&searchconfig=active+accesskey&technologies=iamuser HTTP/1.1", upstream: "http://127.0.0.1:5000/api/1/issues?accounts=&active=True&count=25&enabledonly=true&names=&page=1&regions=&searchconfig=active+accesskey&technologies=iamuser", host: "XXXXXXXX", referrer: "XXXXXXXX/"

Do I need to change some settings or configurations to resolve this? my security group is open.

[westlifezs]

By the way I think it would be nice if the trust policy for securityMonkeyProfile can be added in the documentation. Newcomers need it.

[westlifezs]

forget about the previous error. I fixed it by reconfiguring the web server.
Thank you very much for the help during the process!

[mikegrima]

Glad it works!