Not able to create account

[skdubey]

Hello,

I have followed Quick Start Guide and facing below issues -

1. When i first time open the url/page, as per doc, i should see the 'Login' page, instead i see all options available after login, and user is Anonymous.
2. When i tried to add account, it does nothing. Nginx log says " [error] 17478#0: *99 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "POST /api/1/account HTTP/1.1", upstream: "http://127.0.0.1:5000/api/1/account", host: "ec2-x-x-x-x.us-west-2.compute.amazonaws.com", referrer: "https://ec2-x-x-x-x.us-west-2.compute.amazonaws.com/"
3. I also noticed that port 5000 is not running.

Please guide me to right direction.

[skdubey]

In browser i also see below error :

API unaccessible. HTTP 502:

<title>502 Bad Gateway</title>

502 Bad Gateway

---

nginx/1.4.6 (Ubuntu)

[scriptsrc]

Go into the supervisor directory and run:

sudo supervisorctl -c security_monkey.ini

When you get the supervisor prompt, run "status" to see what is running. There should be two things running: "securitymonkey" and "securitymonkeyscheduler".

The "securitymonkey" entry is the Flask application that should be listening on port 5000. You can restart that process and then run status a few times to see if it worked.

> restart securitymonkey
> status

Most likely, the flask application is having difficulty starting. There should be an error message somewhere in /tmp/securitymonkey.log

I've seen problems where the flask application can't find the SECURITY_MONKEY_SETTINGS environment variable that is suppose to point to env-config/config-deploy.py

[skdubey]

Thanks for pointing to right direction. Now i see the error in /tmp/securitymonkey.log saying :
DEBG 'securitymonkey' stderr output: variable_name)
RuntimeError: The environment variable 'SECURITY_MONKEY_SETTINGS' is not set and as such configuration could not be loaded. Set this variable and make it point to a configuration file

What i did is -
$ export SECURITY_MONKEY_SETTINGS=/srv/security_monkey/env-config/config-deploy.py
$ python manage.py db upgrade

and restarted the service, But again getting same error. Should i require to set this in any config file ?
I can also confirm that variable is set as per below output -
$ echo $SECURITY_MONKEY_SETTINGS
/srv/security_monkey/env-config/config-deploy.py

What else am i missing ?

[skdubey]

Same path is mentioned in "security_monkey.ini" file.

directory=/srv/security_monkey/
environment=PYTHONPATH='/srv/security_monkey/',SECURITY_MONKEY_SETTINGS="/srv/security_monkey/env-config/config-deploy.py"

[skdubey]

Solved -
I couldn't figured out the issue, but tried an workaround by creating a symlink of security_monkey in /home/ubuntu/ and restarted the services, It worked.

[scriptsrc]

I'm glad you were able to fix it. I'll keep tracking this as I've seen it a few times.

If you have the variable in your environment, maybe you should start supervisor with sudo -E instead of just sudo:

sudo -E supervisorctl -c security_monkey.ini

That doesn't make a lot of sense though since the variable should be getting set in security_monkey.ini.

[scriptsrc]

FYI - I just merged a pull request that may help with this issue: #25

[skdubey]

Hi,
I am trying to setup SecurityMonkey for production environment, but facing more issue since last time.

1. Email not working. SES is enabled for my account. Getting below error -
   2014-07-29 07:27:38,212 INFO: Failed to send failure message: <type 'exceptions.Exception'> SESAddressNotVerifiedError: 400 Email address is not verified.
   
   
   Sender
   MessageRejected
   Email address is not verified.
   
   cc923280-16f1-11e4-9f40-135437038bc3
   
   [in /secmonkey/security_monkey/security_monkey/init.py:83]

Should it require to use the ses endpoint and credentials somewhere ?

1. securitymonkeyscheduler is not running.
   supervisor> status
   securitymonkey RUNNING pid 3883, uptime 0:00:26
   securitymonkeyscheduler FATAL Exited too quickly (process log may have details)
   2014-07-29 08:08:12,356 INFO exited: securitymonkeyscheduler (exit status 0; not expected)
   2014-07-29 08:08:12,356 DEBG received SIGCLD indicating a child quit
   2014-07-29 08:08:13,357 INFO gave up: securitymonkeyscheduler entered FATAL state, too many start retries too quickly
2. Forgot password opens 404 page.

[scriptsrc]

Hey skdubey,

SES can be a pain to setup. I think you'll need to head into the console to setup/verify your email address.
http://docs.aws.amazon.com/ses/latest/DeveloperGuide/ses-errors.html

For the scheduler, could you backup your configurations try the develop branch?

$ git checkout develop

Is the 404 an HTTP vs HTTPS issue?

Thanks for the feedback. I want to get these things fixed soon so others don't run into them.

[skdubey]

Hello Patrick,

I tried develop branch and still facing same issues.

1. Scheduler is not starting
   ubuntu@security-monkey:~/security_monkey/supervisor$ sudo -E supervisorctl -c security_monkey.ini
   securitymonkey RUNNING pid 11567, uptime 0:06:19
   securitymonkeyscheduler FATAL Exited too quickly (process log may have details)
   supervisor> status
   securitymonkey RUNNING pid 11689, uptime 0:00:16
   securitymonkeyscheduler BACKOFF Exited too quickly (process log may have details)
   supervisor> status
   securitymonkey RUNNING pid 11689, uptime 0:00:22
   securitymonkeyscheduler FATAL Exited too quickly (process log may have details)
2. It was https vs http but after using develop branch it still says 404 but on https only and doesn't change to http.
3. While registering a new user it redirects to http page and shows 404 error, after entering the manual url including https it opens fine. I have attached the screen shot for same.
   
   
4. I am able to send email through aws console, DKIM etc. is setup and running fine. How can i send you the system logs for debugging.

[skdubey]

Hi,

Any update on this ?

[skdubey]

I tried by re-installing everything, but facing same issue.

[airforceboricua]

Hello,

I am running into the same issue. Is there any fix on this?

[skdubey]

Finally i terminated the existing instance and started a new one with fresh setup and it worked for me this time except emails are not going through. My SES setup & configuration is correct. Can someone help me in this case ?
Where can i change the SES related configurations or does it requires ? I am in us-west-2 region, i hope it doesn't matter but just wanted to check if by default it assumes us-east-1.

[oba11]

I think SES got broken since the last merge to master, couldnt use SES since commit where SMTP was introduced here.

[skdubey]

Thanks for update. I just tried by using user/pass section, restarted 'securitymonkey' service but noticed that it still using SES to send mail and not second provider. Am i missing something here ?
Mail Setting -

These are only required if using SMTP instead of SES

EMAILS_USE_SMTP = True # Otherwise, Use SES
MAIL_SERVER = 'mail.authsmtp.com'
MAIL_PORT = 2525
MAIL_USE_SSL = True
MAIL_USERNAME = 'xxxxxxxxx'
MAIL_PASSWORD = 'yyyyyyyy'

[skdubey]

Is it fixed ? btw, what is the process to get new updates ? Is it - git pull > restart services ?

[scriptsrc]

Hey skdubey. The region shouldn't matter as long as you're not in cn-north-1 or the gov region. I have instances running in us-west-1 as well. I also have this deployed in both ec2 and vpc.

I just brought up a new instance yesterday and SES is working correctly.
The mail settings that you pasted in are telling Security Monkey to use SMTP:

EMAILS_USE_SMTP = True

If you would prefer SES, that should be set to False.

Security Monkey does need a better upgrade mechanism. The Netflix way is to have the environment build and bake a new AMI for each version and replace the existing instance. (Instances at Netflix are intentionally somewhat immutable.) To do an in place upgrade, you would want to backup your custom configurations. These are probably in:

env-config/config-deploy.py
possibly supervisor/security_monkey.ini

You can run a "git status" to have git tell you which files have been modified.

Then do a git pull. Before you copy your config files back, compare them to the new config files that were just pulled down to see if you need to merge in any new settings. Then, restart the services.

I'd love to receive a PULL request for easier upgrades. It could be as simple as renaming all the config files to "config_filename"+".default" in the git repo, so that a git pull won't overwrite their settings. Could also add "*.default" to the .gitignore. The installation instructions would need to be modified slightly to instruct users to copy the ".default" files into place and then modify them.

[oba11]

SES works for me now, didnt notice that DEFAULT_MAIL_SENDER has been changed to MAIL_DEFAULT_SENDER

To update is kind of simple for us, I have saltstack pillar with fixed commit hash, once I change the commit hash, all the installation process.

[scriptsrc]

Yeah, that's confusing. I believe Flask-Mail uses one of those and the Flask_Mail built into Flask-Security uses the other.

[vgomoiu]

Hi. I've spend lots of time to create a new account ad fix the other issues, but it seems I have to address this here. I get confirmation mail using SMTP, but:

Internal Server Error

2014-08-29 15:28:48 [18285] [ERROR] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn-18.0-py2.7.egg/gunicorn/workers/sync.py", line 131, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1836, in call
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/local/lib/python2.7/dist-packages/Flask_RESTful-0.2.5-py2.7.egg/flask_restful/init.py", line 145, in error_router
return original_handler(e)
File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/Flask_RESTful-0.2.5-py2.7.egg/flask_restful/init.py", line 145, in error_router
return original_handler(e)
File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/local/lib/python2.7/dist-packages/Flask_Security-1.7.3-py2.7.egg/flask_security/decorators.py", line 205, in wrapper
return f(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/Flask_Security-1.7.3-py2.7.egg/flask_security/views.py", line 122, in register
login_user(user)
File "/usr/local/lib/python2.7/dist-packages/Flask_Security-1.7.3-py2.7.egg/flask_security/utils.py", line 82, in login_user
identity=Identity(user.id))
File "build/bdist.linux-x86_64/egg/blinker/base.py", line 267, in send
for receiver in self.receivers_for(sender)]
File "/usr/local/lib/python2.7/dist-packages/Flask_Principal-0.4.0-py2.7.egg/flask_principal.py", line 469, in _on_identity_changed
self.set_identity(identity)
File "/usr/local/lib/python2.7/dist-packages/Flask_Principal-0.4.0-py2.7.egg/flask_principal.py", line 418, in set_identity
self._set_thread_identity(identity)
File "/usr/local/lib/python2.7/dist-packages/Flask_Principal-0.4.0-py2.7.egg/flask_principal.py", line 463, in _set_thread_identity
identity=identity)
File "build/bdist.linux-x86_64/egg/blinker/base.py", line 267, in send
for receiver in self.receivers_for(sender)]
File "/usr/local/lib/python2.7/dist-packages/Flask_Security-1.7.3-py2.7.egg/flask_security/core.py", line 214, in on_identity_loaded
for role in current_user.roles:
File "/usr/local/lib/python2.7/dist-packages/Werkzeug-0.9.4-py2.7.egg/werkzeug/local.py", line 338, in getattr
return getattr(self.get_current_object(), name)
File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/attributes.py", line 233, in get
return self.impl.get(instance_state(instance), dict)
File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/attributes.py", line 579, in get
value = self.callable(state, passive)
File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/strategies.py", line 478, in _load_for_state
(orm_util.state_str(state), self.key)
DetachedInstanceError: Parent instance <User at 0x7f9039903910> is not bound to a Session; lazy load operation of attribute 'roles' cannot proceed

What is wrong here?
Thanks in advance

[skdubey]

1. Despite of all the verifications SES still didn't worked for me. Error is as below :
   2014-08-26 12:23:04,831 DEBUG: Failed to send failure message with subject: Welcome
   <type 'exceptions.Exception'> SESAddressNotVerifiedError: 400 Email address is not verified.
   
   
   Sender
   MessageRejected
   Email address is not verified.
   
   b7dfc63f-2d1b-11e4-99b7-3bf0a7e96a8e
   
   [in /home/ubuntu/security_monkey/security_monkey/common/utils/utils.py:88]
2. Apart from above i am also observing some strang behaviour on console. For example :
   a) If i try to login to console i still see 'Anonymous' in console and no data on page, But if i click on 'Setting', I can see the added account. Not sure if it is browser cache (Firefox) issue or something else.
   b) If i register a new user it redirects to /register with error page, but if i try to login with same created user, i am able to do so.
   c). SMTP setting is working for me, but if set SECURITY_CONFIRMABLE & SECURITY_RECOVERABLE as True, i am not able to login to console, setting this to False works for me.

Need help on points below -
a) I don't want to register anyone just by knowing the url of console. How can i protect this ?
b) Where does all the created users for console stored? I looked at postgres db but couldb't found any.

Occassionally, i also noticed attached error.

[scriptsrc]

Hey skdubey,

Try to use Chrome for the time being. I'm working on fixing an issue with Safari/Firefox.

The users are stored in a database table called "user". You should see these tables in your database:

account
alembic_version
association
item
itemaudit
itemcomment
itemrevision
itemrevisioncomment
role
role_users
technology
user

If you don't see any tables, you'll need to run python manage.py db upgrade to have alembic/Flask-Migrate create the tables for you.

You will need to be able to receive the Welcome email and confirm the account using the link provided in the email before you can login to your account.

You can disable user registration by setting SECURITY_REGISTERABLE to false. You could then create individual users manually by adding a method to manage.py similar to the create_user() method listed here: https://pythonhosted.org/Flask-Security/quickstart.html

def create_user():
    user_datastore.create_user(email='matt@nobien.net', password='password')
    db.session.commit()

That way, user registration is a process that involves the security team. I'll add a method for manual user creation to manage.py for the next release.

Your SES problems may be answered here:
http://aws.amazon.com/ses/faqs/

"Also, while all users must verify the email addresses or domains they will send from, sandbox users must also verify the email addresses or domains they will send to. The purpose of verification is to confirm that you own these email addresses or domains, and to prevent others from using them. See the Amazon SES Developer Guide for details on verifying email addresses and domains."

[skdubey]

Hi,

Thanks a lot for your time. Please bear with me for some more queires.

1. Chrome worked but it requires 2-3 browser refresh to load the data, some times it take more than that. But better than Firefox.
2. Setting SECURITY_REGISTERABLE to false worked.
3. I tried that but not able to see anything except db user. There are 5 users who are able to access console but in db it doesn't shows anything.
   
4. Yes, i received a welcome email which doesn't have url to confirm but email id as below.
   

[gbarton]

skdubey: just ran into that postgres thing today, try:
\c secmonkey;
select * from public.user;

I think user is a postgres table as well and that gets hit without the schema name.

[skdubey]

Thanks gbarton,

Now i see the list of users, but couldn't able to find the meaning of
column 'active = t', 'confirmed_at = ', 'daily_audit_email = t/f ' and
'change_reports = ALL'. Where can i find these information and will it
affect if i change the value in db directly ?

On Tue, Sep 9, 2014 at 1:02 AM, gbarton notifications@github.com wrote:

skdubey: just ran into that postgres thing today, try:
\c secmonkey;
select * from public.user;

I think user is a postgres table as well and that gets hit without the
schema name.

—
Reply to this email directly or view it on GitHub
#18 (comment)
.

[skdubey]

Ah!, sorry
Very simple, i realized it later : t = True and f = False and "change_reports = ALL" means all the changes should be notified to the particular users.

[scriptsrc]

Thanks gbarton for helping out. I was stumped on the user table problem.

Skdubey, how are you doing with your install now? Can this issue be closed or are you still running into issues?

[skdubey]

Hello Patrick,

I still didn't get the welcome email part, where confirmation is required.
Can some send me the sample email ? I created multiple users but couldn't
get the confirmation link to click on, but i do get the welcome email.

Also curious to know the status of enhancements like, report should be
downloadable in .csv or .pdf format from console.

On Tue, Sep 9, 2014 at 11:36 AM, Patrick Kelley notifications@github.com
wrote:

Thanks gbarton for helping out. I was stumped on the user table problem.

Skdubey, how are you doing with your install now? Can this issue be closed
or are you still running into issues?

—
Reply to this email directly or view it on GitHub
#18 (comment)
.

[vgomoiu]

Hi skdubey

I managed to have it all running although no confirmation mail.

I had the same issue with the confirmation mail, as you had, which now is fixed. Once more, make sure that you have full production for SES or confirm in SES email addresses/domains.

What happened for me: I am running in eu-west, and the defaults in boto ses are for us-east. After changing the defaults in

/usr/local/lib/python2.7/dist-packages/boto-2.32.1-py2.7.egg/boto/ses/connection.py

class SESConnection(AWSAuthConnection):

ResponseError = BotoServerError
DefaultRegionName = 'eu-west-1'
DefaultRegionEndpoint = 'email.eu-west-1.amazonaws.com'

everything worked and I could see mails comming.

Also, after getting the welcome mail form registering, if I look in the USER table, all the users are marked as ACTIVE (True) .

This is my 2 cents

[skdubey]

Thanks vgomolu,

You gave me some hope, because i am running in us-west-2 region while the
Default is us-east-1. As per your update i changed the same file update it
to us-west-2 and restarted the both the services (securitymonkey &
schedular) with no luck, But found below error which id different than
earlier. Earlier i was getting that "your email is not verified" and now i
am getting this : "<type 'exceptions.Exception'> :1:0: syntax
error". Below is the actual output -

2014-09-09 12:30:38,888 DEBG 'securitymonkeyscheduler' stderr output:
Failed to send failure message with subject: [Accountname] Changes in
securitygroup
<type 'exceptions.Exception'> :1:0: syntax error

One more observation i have is, in my system there are two python instances
at /usr/local/lib/{python2.7 & python3.4}. I made changes in 2.7 as there
is nothing under "/usr/local/lib/python3.4/dist-packages/".

On Tue, Sep 9, 2014 at 5:34 PM, vgomoiu notifications@github.com wrote:

Hi skdubey

I managed to have it all running although no confirmation mail.

I had the same issue with the confirmation mail, as you had, which now is
fixed. Once more, make sure that you have full production for SES or
confirm in SES email addresses/domains.

What happened for me: I am running in eu-west, and the defaults in boto
ses are for us-east. After changing the defaults in

/usr/local/lib/python2.7/dist-packages/boto-2.32.1-py2.7.egg/boto/ses/connection.py

class SESConnection(AWSAuthConnection):

ResponseError = BotoServerError
DefaultRegionName = 'eu-west-1'
DefaultRegionEndpoint = 'email.eu-west-1.amazonaws.com'

everything worked and I could see mails comming.

Also, after getting the welcome mail form registering, if I look in the
USER table, all the users are marked as ACTIVE (True) .

This is my 2 cents

—
Reply to this email directly or view it on GitHub
#18 (comment)
.

[vgomoiu]

skdubey, I had exactly that error wehn first time I've updated the connection.py with the SES SMTP endpoint instead of API endpoint; so for you should be

DefaultRegionName = 'us-west-2'
DefaultRegionEndpoint = 'email.us-west-2.amazonaws.com'

Please make sure it looks like above.

Also, you are correct, the update is only for python 2.7
Also check config-deploy.py and make sure you have to correct format.

[skdubey]

Thanks a lot vgomoiu to make this working. It is working with SES now.

I have one more point - What should be the lowest check interval (default
is 10 min.) and how can i change that value ?

On Tue, Sep 9, 2014 at 6:27 PM, vgomoiu notifications@github.com wrote:

skdubey, I had exactly that error wehn first time I've updated the
connection.py with the SES SMTP endpoint instead of API endpoint; so for
you should be

DefaultRegionName = 'us-west-2'
DefaultRegionEndpoint = 'email.us-west-2.amazonaws.com'

Please make sure it looks like above.

Also, you are correct, the update is only for python 2.7
Also check config-deploy.py and make sure you have to correct format.

—
Reply to this email directly or view it on GitHub
#18 (comment)
.

[skdubey]

Also, the "Forgot Password" is throwing 404 error and redirecting to url #/reset.

[vgomoiu]

Glad to hear it's working.

For the check interval:
/home/ubuntu/security_monkey/security_monkey/init.py
and find the "def setup_scheduler"

The SECURITY_CONFIRMABLE = False and SECURITY_RECOVERABLE = False don't seem to be implemented yet.

[skdubey]

Again, thanks a lot vgomoiu.

You have been very helpful to me.

On Wed, Sep 10, 2014 at 12:59 PM, vgomoiu notifications@github.com wrote:

Glad to hear it's working.

For the check interval:
/home/ubuntu/security_monkey/security_monkey/init.py
and find the "def setup_scheduler"

The SECURITY_CONFIRMABLE = False and SECURITY_RECOVERABLE = False don't
seem to be implemented yet.

—
Reply to this email directly or view it on GitHub
#18 (comment)
.