Merge 60dbc14 into 94eb032

Netflix · Feb 9, 2018 · b02488e · b02488e
2 parents 94eb032 + 60dbc14
commit b02488e
Show file tree

Hide file tree

Showing 8 changed files with 153 additions and 45 deletions.
diff --git a/README.md b/README.md
@@ -3,20 +3,28 @@ Security Monkey
 
 <img align="right" alt="Security Monkey Logo 2017" src="docs/images/Security_Monkey.png" width="50%">
 
-Security Monkey monitors your [AWS and GCP accounts](https://medium.com/@Netflix_Techblog/netflix-security-monkey-on-google-cloud-platform-gcp-f221604c0cc7) for policy changes and alerts on insecure configurations.  Support is available for OpenStack public and private clouds.  It provides a single UI to browse and search through all of your accounts, regions, and cloud services.  The monkey remembers previous states and can show you exactly what changed, and when.
+Security Monkey monitors your [AWS and GCP accounts](https://medium.com/@Netflix_Techblog/netflix-security-monkey-on-google-cloud-platform-gcp-f221604c0cc7) for policy changes and alerts on insecure configurations.  Support is available for OpenStack public and private clouds.  Security Monkey can also watch and monitor your GitHub organizations, teams, and repositories.
+
+It provides a single UI to browse and search through all of your accounts, regions, and cloud services.  The monkey remembers previous states and can show you exactly what changed, and when.
 
 Security Monkey can be extended with [custom account types](docs/plugins.md), [custom watchers](docs/development.md#adding-a-watcher), [custom auditors](docs/development.md#adding-an-auditor), and [custom alerters](docs/misc.md#custom-alerters).
 
 It works on CPython 2.7. It is known to work on Ubuntu Linux and OS X.
 
-[![Stories in Ready](https://badge.waffle.io/Netflix/security_monkey.svg?label=ready&title=Ready)](http://waffle.io/Netflix/security_monkey) [![Gitter chat](https://badges.gitter.im/gitterHQ/gitter.png)](https://gitter.im/Netflix/security_monkey)
+[![Gitter chat](https://badges.gitter.im/gitterHQ/gitter.png)](https://gitter.im/Netflix/security_monkey)
 
 | Develop Branch  | Master Branch |
 | ------------- | ------------- |
 | [![Build Status](https://travis-ci.org/Netflix/security_monkey.svg?branch=develop)](https://travis-ci.org/Netflix/security_monkey)  | [![Build Status](https://travis-ci.org/Netflix/security_monkey.svg?branch=master)](https://travis-ci.org/Netflix/security_monkey)  |
 | [![Coverage Status](https://coveralls.io/repos/github/Netflix/security_monkey/badge.svg?branch=develop)](https://coveralls.io/github/Netflix/security_monkey?branch=develop)  | [![Coverage Status](https://coveralls.io/repos/github/Netflix/security_monkey/badge.svg?branch=master)](https://coveralls.io/github/Netflix/security_monkey?branch=master) |
 
 
+🚨⚠️🥁🎺 PLEASE READ: BREAKING CHANGES FOR 1.0 🎺🥁⚠️🚨
+--------------
+If you are upgrading to 1.0 for the first time, please review the [Quickstart](quickstart.md) and the [Autostarting](autostarting.md)
+documents as there is a new deployment pattern for Security Monkey. Also, new IAM permissions have been added.
+
+
 Project resources
 -----------------
 
@@ -28,6 +36,7 @@ Project resources
 - [Gitter.im Chat Room](https://gitter.im/Netflix/security_monkey)
 - [CloudAux](https://github.com/Netflix-Skunkworks/cloudaux)
 - [PolicyUniverse](https://github.com/Netflix-Skunkworks/policyuniverse)
+- [Troubleshooting](docs/troubleshooting.md)
 
 
 Instance Diagram

diff --git a/docs/changelog.md b/docs/changelog.md
@@ -1,12 +1,61 @@
 Changelog
 =========
 
+v1.0.0 (2018-02-09)
+--------------------
+Major Milestone release.
+
+There are many, many changes that have been made.  Below are some of the most important items to keep note of:
+
+1. **BREAKING CHANGES -- ALL NEW DEPLOYMENT MODEL** (Please review the [Quickstart](quickstart.md) and [Autostarting](autostarting.md) docs for details)
+    - We swapped out APScheduler in favor of Celery. This allows us to actually scale Security Monkey with multiple UI instances, and many, many workers so
+      you can get data into Security Monkey much faster!
+1. Lots, and lots of bug fixes and documentation updates
+1. New features:
+    - OpenStack watching and auditing support
+    - GitHub Organization, Repos, and Teams watching and auditing
+    - AWS GovCloud Support
+    - Azure AD SSO provider support
+    - AWS Glacier support
+    - Support for [SWAG account syncing](https://github.com/Netflix-Skunkworks/swag-client).
+    - Auditor improvements
+
+1. Many IAM changes. [Please review the IAM docs](https://github.com/Netflix/security_monkey/blob/develop/docs/quickstart.md#account-types) and update your permissions accordingly.
+
+Too many PRs to list... Special thanks to the following contributors:
+    - @mikegrima
+    - @monkeysecurity
+    - @mstair
+    - @kevgliss
+    - @mcpeak
+    - @zpritcha
+    - @mark-ignacio
+    - @falcoris
+    - @vishbhalla
+    - @frohoff
+    - @tabletcorry
+    - @shrikant0013
+    - @pjbgf
+    - @billy-lechtenberg
+    - @Qmando
+    - @jleaniz
+    - @wozz
+    - @markofu
+    - @cxmcc
+    - @jpohjolainen
+    - @PyScott
+    - @sysboy
+    - @gellerb
+    - @fabiop
+    - @joaquin386
+    - @oba11
+    - @castrapel
+    - @NunoPinheiro
+
 
 v0.9.3 (2017-07-31)
 ----------------------------------------
 
-TODO: Write this.
-
 Important Notes:
 - Additional Permissions Requried:
     - "lambda:getfunctionconfiguration",

diff --git a/docs/faq.md b/docs/faq.md
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
@@ -0,0 +1,65 @@
+Troubleshooting Guide
+==========
+
+This is a doc to outline some of the most common issues that users have encountered and their solutions. We very much welcome
+Pull-Requests to this doc for items to help other Security Monkey users.
+
+Existing Resources
+------------
+In general, if you are encountering issues, please review the GitHub issues (open and closed) to see if anyone else has
+already experienced the issue. We often find that most issues users experience have already been solved.
+
+Also, please review the [quickstart guide](https://github.com/Netflix/security_monkey/blob/develop/docs/quickstart.md). This will likely help uncover the issue you may be experiencing.
+
+
+Enable Debug Logging
+---------------
+Logs are very useful for debugging issues. Enabling debug logging will help provide additional details on what may be breaking.
+To do this, you need to modify the configuration Python file that is in use by Security Monkey. Namely, you need to modify the `LOG_CFG` section.
+You need to set all `level` settings to `DEBUG`. Save the file, and then reload Security Monkey.
+
+
+Common Issues
+-----------
+This is a list of common issues and their resolutions.  
+
+1. **No data is loading**
+
+    This is perhaps the number 1 issue users encounter. This can be caused for a number of reasons:
+      - Insufficient permissions for the Security Monkey IAM Roles.
+        **Solution**: [Follow the IAM instructions](https://github.com/Netflix/security_monkey/blob/develop/docs/quickstart.md#account-types) for the given technology in question and ensure that the proper permissions are in place.
+
+      - Scheduler and workers are not functioning properly.
+        **Solution**: Follow the [autostarting guide](https://github.com/Netflix/security_monkey/blob/develop/docs/autostarting.md), and ensure that the following is true:
+          - Remember, there should only ever be exactly one scheduler instance running (only 1 celery scheduler process that should ever be running)
+          - Security Monkey and the workers have network connectivity to the Redis queue.
+          - To track down issues with the scheduler, try running the `monkey find_changes -a ACCOUNT` command to see if items can be fetched. This will help uncover other
+            issues that may be relevant.
+
+1. **I'm seeing "Access Denied" errors.**
+
+    This is caused by insufficient permissions. **Solution**: [Follow the IAM instructions](https://github.com/Netflix/security_monkey/blob/develop/docs/quickstart.md#account-types) for the given technology in question and ensure that the proper permissions are in place.
+
+1. **Error: Too many open files.** (This is not likely to be as much of a problem in v1.0+ but if you encounter it, then follow the instructions below)
+
+    You might see an error along the lines of: `Too many open files' [in /usr/local/src/security_monkey/security_monkey/exceptions.py:68]`
+
+    **Solution:** Try increasing the limit for open file handlers
+
+    ```bash
+    /etc/security/limits.conf
+    *    soft nofile 100000
+    *    hard nofile 100000
+
+    /etc/pam.d/common-session
+    session required pam_limits.so
+
+    /etc/pam.d/common-session-noninteractive 
+    session required pam_limits.so
+
+    /etc/supervisor/supervisord.conf, in the [supervisord] section:
+    minfds=100000
+    ```
+
+    Reference: [Raising the maximum number of file descriptors](https://underyx.me/2015/05/18/raising-the-maximum-number-of-file-descriptors)
+
diff --git a/docs/update.md b/docs/update.md
@@ -22,6 +22,8 @@ Security Monkey now has 5 primary components:
 1. PostgreSQL Database (for storage)
 1. Redis (message broker for workers)
 
+Also, (for AWS) please review the [IAM documentation](https://github.com/Netflix/security_monkey/blob/develop/docs/iam_aws.md) as there are new permissions required.
+
 
 General Deployment Guidance:
 ------------------
@@ -38,6 +40,7 @@ Performing the steps in this order will ensure:
 - Proper DB upgrades occur without possibly impacting workers mutating the database
 - The schedulers and workers are working properly together
 
+
 Update Steps:
 -----------
 -   Prerequisites
@@ -57,6 +60,11 @@ This doc assumes you already have installed and are running a Security Monkey en
 1. Redis
 1. NGINX
 
+## Update the Security Monkey IAM permissions (if applicable):
+
+As new features come out, Security Monkey may require new IAM permissions. Always follow the [respective IAM doc for the given technology](https://github.com/Netflix/security_monkey/blob/develop/docs/quickstart.md#account-types)
+to see if you need to update your Security Monkey permissions. Failure to do this will result in Access Denied errors and items not appearing in Security Monkey.
+
 ### Backup config and installation files
 
 Backup your `/usr/local/src/security_monkey/env-config/config.py` and move your existing installation to backup directory

diff --git a/security_monkey/__init__.py b/security_monkey/__init__.py
@@ -23,7 +23,7 @@
 import stat
 
 ### VERSION ###
-__version__ = '0.9.3'   # TODO update this to 0.9.4!
+__version__ = '1.0.0'
 
 ### FLASK ###
 from flask import Flask

diff --git a/security_monkey/task_scheduler/beat.py b/security_monkey/task_scheduler/beat.py
@@ -53,19 +53,24 @@ def setup_the_tasks(sender, **kwargs):
                     sender.add_periodic_task(interval, task_account_tech.s(account.name, monitor.watcher.index))
                     app.logger.debug("[+] Scheduled task to occur every {} minutes".format(interval))
 
+                    # TODO: Due to a bug with Celery (https://github.com/celery/celery/issues/4041) we temporarily
+                    #       disabled this to avoid many duplicate events from getting added.
                     # Also schedule a manual audit changer just in case it doesn't properly
                     # audit (only for non-batched):
-                    if not monitor.batch_support:
-                        sender.add_periodic_task(
-                            crontab(hour=10, day_of_week="mon-fri"), task_audit.s(account.name, monitor.watcher.index))
-                        app.logger.debug("[+] Scheduled task for tech: {} for audit".format(monitor.watcher.index))
+                    # if not monitor.batch_support:
+                    #     sender.add_periodic_task(
+                    #         crontab(hour=10, day_of_week="mon-fri"), task_audit.s(account.name, monitor.watcher.index))
+                    #     app.logger.debug("[+] Scheduled task for tech: {} for audit".format(monitor.watcher.index))
+                    #
+                    # app.logger.debug("[{}] Completed scheduling for technology: {}".format(account.name,
+                    #                                                                        monitor.watcher.index))
 
-                    app.logger.debug("[{}] Completed scheduling for technology: {}".format(account.name,
-                                                                                           monitor.watcher.index))
             app.logger.debug("[+] Completed scheduling tasks for account: {}".format(account.name))
 
         # Schedule the task for clearing out old exceptions:
         app.logger.info("Scheduling task to clear out old exceptions.")
+
+        # TODO: Investigate if this creates many duplicate tasks RE: Celery bug mentioned above
         sender.add_periodic_task(crontab(hour=3, minute=0), clear_expired_exceptions.s())
 
     except Exception as e:

diff --git a/security_monkey/tests/scheduling/test_celery_scheduler.py b/security_monkey/tests/scheduling/test_celery_scheduler.py
@@ -382,9 +382,9 @@ def test_fix_orphaned_deletions(self):
     @patch("security_monkey.task_scheduler.beat.setup")
     @patch("security_monkey.task_scheduler.beat.purge_it")
     @patch("security_monkey.task_scheduler.tasks.task_account_tech")
-    @patch("security_monkey.task_scheduler.tasks.task_audit")
+    # @patch("security_monkey.task_scheduler.tasks.task_audit")
     @patch("security_monkey.task_scheduler.tasks.clear_expired_exceptions")
-    def test_celery_beat(self, mock_expired_exceptions, mock_task_audit, mock_account_tech, mock_purge, mock_setup):
+    def test_celery_beat(self, mock_expired_exceptions, mock_account_tech, mock_purge, mock_setup):
         from security_monkey.task_scheduler.beat import setup_the_tasks
         from security_monkey.watchers.iam.iam_role import IAMRole
         from security_monkey.auditors.iam.iam_role import IAMRoleAuditor
@@ -411,12 +411,12 @@ def test_celery_beat(self, mock_expired_exceptions, mock_task_audit, mock_accoun
         # The ".s" are the scheduled tasks. Too lazy to grab the intervals out.
         assert mock_account_tech.s.called
         assert mock_expired_exceptions.s.called
-        assert mock_task_audit.s.called
+        #assert mock_task_audit.s.called
 
         # Build the expected mock results:
         scheduled_tech_result_list = []
         async_result_list = []
-        audit_result_list = []
+        # audit_result_list = []
 
         import security_monkey.watcher
         import security_monkey.auditor
@@ -427,11 +427,11 @@ def test_celery_beat(self, mock_expired_exceptions, mock_task_audit, mock_accoun
                 async_result_list.append((((account.name, w),),))
 
             # It's just policy for IAM:
-            audit_result_list.append(((account.name, "policy"),))
+            # audit_result_list.append(((account.name, "policy"),))
 
         assert mock_account_tech.s.call_args_list == scheduled_tech_result_list
         assert async_result_list == mock_account_tech.apply_async.call_args_list
-        assert audit_result_list == mock_task_audit.s.call_args_list
+        # assert audit_result_list == mock_task_audit.s.call_args_list
 
         security_monkey.task_scheduler.tasks.get_monitors = old_get_monitors