Adding OpenStack cloud support (#837)

* Adding OpenStack support to Security Monkey

* Removing user watcher and tests, need to consider best approach to identity (user, projects, domains) as it does not map to other cloud IAM

* Removing glance images and load balancers from initial checkin. Limited value and complications with restricting roles, need to revisit

* Updating README and docs to include OpenStack instance and IAM instructions

* Fixing OpenStack doc markdown formatting errors

* Correct import statements for cloudaux mocks

* Adding OpenStack artifacts to travis build

* Fix cloudaux test import statement

* Update import statements for network and object_store unit tests

* Simplifying quickstart documentation for packages needed for OpenStack

* Adding missing punctuation to quickstart

* Updates based on @MonkeySecurity PR review feedback

.travis.yml

@@ -25,7 +25,9 @@ matrix:
       - pip install pip --upgrade
       - pip install setuptools --upgrade
       - pip install google-compute-engine
+      - pip install openstacksdk
       - pip install cloudaux\[gcp\]
+      - pip install cloudaux\[openstack\]
       - python setup.py develop
       - pip install .[tests]
       - pip install coveralls

README.md

@@ -3,7 +3,7 @@ Security Monkey
 
 <img align="right" alt="Security Monkey Logo 2017" src="docs/images/Security_Monkey.png" width="50%">
 
-Security Monkey monitors your [AWS and GCP accounts](https://medium.com/@Netflix_Techblog/netflix-security-monkey-on-google-cloud-platform-gcp-f221604c0cc7) for policy changes and alerts on insecure configurations.  It provides a single UI to browse and search through all of your accounts, regions, and cloud services.  The monkey remembers previous states and can show you exactly what changed, and when.
+Security Monkey monitors your [AWS and GCP accounts](https://medium.com/@Netflix_Techblog/netflix-security-monkey-on-google-cloud-platform-gcp-f221604c0cc7) for policy changes and alerts on insecure configurations.  Support is available for OpenStack public and private clouds.  It provides a single UI to browse and search through all of your accounts, regions, and cloud services.  The monkey remembers previous states and can show you exactly what changed, and when.
 
 Security Monkey can be extended with [custom account types](docs/plugins.md), [custom watchers](docs/development.md#adding-a-watcher), [custom auditors](docs/development.md#adding-an-auditor), and [custom alerters](docs/misc.md#custom-alerters).
 

docs/iam_openstack.md

@@ -0,0 +1,121 @@
+IAM Role Setup on OpenStack
+===========================
+
+Security Monkey currently only supports the Keystone [password](https://docs.openstack.org/keystoneauth/latest/authentication-plugins.html) identity plugin. 
+This allows for support for both V2/V3 identity services. It is anticipated that additional identity plugins will be added.
+
+
+Credentials
+-----------
+
+OpenStack clients have migrated to utilizing the [os-client-config](https://docs.openstack.org/os-client-config/latest/) library for configuation. This supports definition of multiple cloud configs in a single yaml file. The Security Monkey OpenStack integration utilizes os-client-config, where legacy OpenRC files can be formatted into clouds.yaml entries.
+
+    $ cat ~/.config/openstack/clouds.yaml
+    clouds:
+        openstack:
+             auth:
+                 auth_url: http://192.168.1.1/identity/v3
+                 username: "secmonkey"
+                 project_id: 3bb7e8e8e01247ea9401dced0a642093
+                 project_name: "demo"
+                 password: "XXXXXXXXX"
+                 user_domain_name: "Default"
+             region_name: "RegionOne"
+             interface: "public"
+             identity_api_version: 3
+
+
+Account Setup
+-------------
+
+By default, regular users created in a project are considered project owners and have both read and write access to project configurations (i.e., you can view current security groups and also create/delete). If you do not have access to setup or request a read-only role added (below) and plan to use an existing user or create a dedicated security monkey user, understand that user will have orchestration rights.
+
+Example commands below are using the unified openstack client. User creation requires "admin" user rights (can also be performed in Horizon).
+
+    openstack user create secmonkey --password XXXXXXXXX --project demo
+    +---------------------+----------------------------------+
+    | Field               | Value                            |
+    +---------------------+----------------------------------+
+    | default_project_id  | 3bb7e8e8e01247ea9401dced0a642093 |
+    | domain_id           | default                          |
+    | enabled             | True                             |
+    | id                  | d206369150994984ad1926821b5ce39b |
+    | name                | secmonkey                        |
+    | options             | {}                               |
+    | password_expires_at | None                             |
+    +---------------------+----------------------------------+
+
+Role Setup
+----------
+
+OpenStack's support for IAM [roles](https://docs.openstack.org/keystone/latest/admin/cli-manage-projects-users-and-roles.html) continues to evolve. 
+
+Currently, role configuration is an involved/manual process. It requires admin account access to create and assign roles and also the ability to directly edit the OpenStack service policy files (or the ability to set via automated workflow). In addition, roles are managed per service (/etc/PROJECT/policy.json). Modify the user/project names below as appropriate.
+
+We will create a dedicated read-only role and modify service policy to restrict to get APIs. 
+
+    $ openstack role create read-only
+    +-----------+----------------------------------+
+    | Field     | Value                            |
+    +-----------+----------------------------------+
+    | domain_id | None                             |
+    | id        | 58c2bffb10174664aef7707c0fe6885a |
+    | name      | read-only                        |
+    +-----------+----------------------------------+
+
+Add the secmonkey user (created above) to the read-only role (this command has no output)
+
+    $ openstack role add --user secmonkey --project demo read-only
+
+Verify the user was added to the role
+
+    $ openstack role assignment list --user secmonkey --project demo --names 
+    +-----------+--------------------+-------+--------------+--------+-----------+
+    | Role      | User               | Group | Project      | Domain | Inherited |
+    +-----------+--------------------+-------+--------------+--------+-----------+
+    | read-only | secmonkey@Default  |       | demo@Default |        | False     |
+    +-----------+--------------------+-------+--------------+--------+-----------+
+
+
+These instructions were written based on the Pike release, syntax may look different depending on the release. Let's start where most of our watchers reference, the neutron (networking service) policy file (by default in /etc/neutron/policy.json). There are many ways we could setup the policies to restrict our role to only get APIs. 
+One of the least intrusive is to exclude the read_only role from the owner rule and create a read_only rule that we can apply to the get APIs. Then we update each get with an or clause for our read_only rule.
+
+    {
+        "context_is_admin":  "role:admin or user_name:neutron",
+        "owner": "tenant_id:%(tenant_id)s and not role:read_only",
+        "admin_or_owner": "rule:context_is_admin or rule:owner",
+        "context_is_advsvc":  "role:advsvc",
+        "admin_or_network_owner": "rule:context_is_admin or tenant_id:%(network:tenant_id)s",
+        "admin_owner_or_network_owner": "rule:owner or rule:admin_or_network_owner",
+        "admin_only": "rule:context_is_admin",
+        "read_only": "role:read_only",
+        "regular_user": "",
+        ...
+        "get_network": "rule:admin_or_owner or rule:shared or rule:external or rule:context_is_advsvc or rule:read_only",
+        ...
+        "get_port": "rule:context_is_advsvc or rule:admin_owner_or_network_owner or rule:read_only",
+        ...
+        "get_router": "rule:admin_or_owner or rule:read_only",
+        ...
+        "get_floatingip": "rule:admin_or_owner or rule:read_only",
+    }
+
+Changes are effectively immediately, requires no service restart. We have to repeat for each service policy that is supported.
+
+Edit nova (compute service) policy to restrict to only the instance show APIs (by default, /etc/nova/policy.json).
+
+    {
+        "context_is_admin":  "role:admin",
+        "admin_or_owner":  "is_admin:True or project_id:%(project_id)s and not role:read_only",
+        "read_only": "role:read_only",
+        "default": "rule:admin_or_owner",
+        "admin_api": "is_admin:True",
+        ...
+        "os_compute_api:servers:show": "rule:admin_or_owner or rule:read_only",
+        ...
+    }
+
+Next:
+-----
+
+- [Back to the Quickstart](quickstart.md#database)

docs/instance_launch_openstack.md

@@ -0,0 +1,25 @@
+Launch an OpenStack instance
+============================
+
+Orchestrate an instance (via Horizon/Heat/APIs). Below are some guidelines. Your cloud setup, conventions, and policies will dictate specifics.
+
+-   **Source**: Ubuntu LTS (14.04 or 16.04) source
+-   **Machine Type**: m1.medium or equivalent (3.75GB RAM minimum)
+-   **Security Group**: Allow appropriately restricted ingress HTTPS and SSH. For egress, allow access to the OpenStack service API endpoints (in Horizon, found under API Access details) and for the initial Security Monkey installation you will access to the Internet/proxy.
+
+
+Connecting to your new instance:
+--------------------------------
+
+We will connect to the new instance over ssh:
+
+    $ ssh -i <PRIVATE_KEY> -l ubuntu <IP_ADDRESS>
+
+Replace the PRIVATE_KEY parameter with the private key for your keypair assigned in instance creation
+Replace the IP_ADDRESS with the IP address of your instance (public or floating ip, depending on network assignment)
+
+
+Next:
+-----
+
+- [Back to the Quickstart](quickstart.md#install-security-monkey-on-your-instance)

docs/quickstart.md

@@ -1,10 +1,10 @@
 Quick Start Guide
 =================
 
-Setup on AWS or GCP
--------------------
+Setup on AWS, GCP, or OpenStack
+-------------------------------
 
-Security Monkey can run on an Amazon EC2 (AWS) instance or a Google Cloud Platform (GCP) instance (Google Cloud Platform). The only real difference in the installation is the IAM configuration and the bringup of the Virtual Machine that runs Security Monkey.
+Security Monkey can run on an Amazon EC2 (AWS) instance, Google Cloud Platform (GCP) instance (Google Cloud Platform), or OpenStack (public or private cloud) instance. The only real difference in the installation is the IAM configuration and the bringup of the Virtual Machine that runs Security Monkey.
 
 GitHub Organization Monitoring
 --------------
@@ -15,22 +15,25 @@ IAM Permissions
 
 -   [AWS IAM instructions](iam_aws.md).
 -   [GCP IAM instructions](iam_gcp.md).
+-   [OpenStack IAM instructions](iam_openstack.md).
 
 Database
 --------
 
 Security Monkey needs a postgres database. Select one of the following:
 
--   Local Postgres (You'll set this up later once you have an instance up.)
+-   Local Postgres (You'll set this up later once you have an instance up.).
 -   [Postgres on AWS RDS](postgres_aws.md).
 -   [Postgres on GCP's Cloud SQL](postgres_gcp.md).
+-   Currently OpenStack utilizes a local postgres database.
 
 Launch an Instance:
 -------------------
 
 -   [docker instructions](docker.md).
 -   [Launch an AWS instance](instance_launch_aws.md).
 -   [Launch a GCP instance](instance_launch_gcp.md).
+-   [Launch an OpenStack instance](instance_launch_openstack.md).
 
 Install Security Monkey on your Instance
 ----------------------------------------
@@ -91,6 +94,7 @@ Releases are on the master branch and are updated about every three months. Blee
     pip install --upgrade urllib3[secure]   # to prevent InsecurePlatformWarning
     pip install google-compute-engine  # Only required on GCP
     pip install cloudaux\[gcp\]
+    pip install cloudaux\[openstack\] # Only required on OpenStack
     python setup.py develop
 
 ### 🚨⚠️🥁🎺 ULTRA SUPER IMPORTANT SPECIAL NOTE PLEASE READ THIS 🎺🥁⚠️🚨 ###
@@ -181,11 +185,17 @@ You'll need to add at least one account before starting the scheduler. It's easi
                                   [--notes NOTES] --id IDENTIFIER
                                   [--update-existing] [--creds_file CREDS_FILE]
 
+    monkey add_account_openstack
+    usage: monkey add_account_openstack [-h] -n NAME [--thirdparty] [--active]
+                                  [--notes NOTES] --id IDENTIFIER
+                                  [--update-existing]
+                                  [--cloudsyaml_file CLOUDSYAML_FILE]
+
 For clarity: the `-n NAME` refers to the name that you want Security Monkey to use to associate with the account.
 A common example would be "test" for your testing AWS account or "prod" for your main production AWS account. These names are unique.
 
 The `--id IDENTIFIER` is the back-end cloud service identifier for a given provider. For AWS, it's the 12 digit account number, 
-and for GCP, it's the project ID.
+and for GCP, it's the project ID. For OpenStack, it's the cloud configuration to load from the clouds.yaml file.
 
 ### Syncing With SWAG
 

security_monkey/account_managers/openstack_account.py

@@ -0,0 +1,40 @@
+#     Copyright (c) 2017 AT&T Intellectual Property. All rights reserved.
+#
+#     Licensed under the Apache License, Version 2.0 (the "License");
+#     you may not use this file except in compliance with the License.
+#     You may obtain a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#     Unless required by applicable law or agreed to in writing, software
+#     distributed under the License is distributed on an "AS IS" BASIS,
+#     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     See the License for the specific language governing permissions and
+#     limitations under the License.
+"""
+.. module: security_monkey.accounts.openstack_account
+    :platform: Unix
+    :synopsis: Manages generic OpenStack account.
+
+
+.. version:: $$VERSION$$
+.. moduleauthor:: Michael Stair <mstair@att.com>
+
+
+"""
+from security_monkey.account_manager import AccountManager, CustomFieldConfig
+from security_monkey.datastore import Account
+
+
+class OpenStackAccountManager(AccountManager):
+    account_type = 'OpenStack'
+    identifier_label = 'Cloud Name'
+    identifier_tool_tip = 'OpenStack Cloud Name. Cloud configuration to load from clouds.yaml file'
+
+    cloudsyaml_tool_tip = ('Path on disk to clouds.yaml file')
+    custom_field_configs = [        
+        CustomFieldConfig('cloudsyaml_file', 'OpenStack clouds.yaml file', True, cloudsyaml_tool_tip)
+    ]
+
+    def __init__(self):
+        super(OpenStackAccountManager, self).__init__()

security_monkey/auditors/openstack/object_container.py

@@ -0,0 +1,52 @@
+#     Copyright (c) 2017 AT&T Intellectual Property. All rights reserved.
+#
+#     Licensed under the Apache License, Version 2.0 (the "License");
+#     you may not use this file except in compliance with the License.
+#     You may obtain a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#     Unless required by applicable law or agreed to in writing, software
+#     distributed under the License is distributed on an "AS IS" BASIS,
+#     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     See the License for the specific language governing permissions and
+#     limitations under the License.
+"""
+.. module: security_monkey.openstack.auditors.object_container
+    :platform: Unix
+
+.. version:: $$VERSION$$
+.. moduleauthor:: Michael Stair <mstair@att.com>
+
+"""
+
+from security_monkey.auditor import Auditor
+from security_monkey.watchers.openstack.object_store.object_container import OpenStackObjectContainer
+
+
+class OpenStackObjectContainerAuditor(Auditor):
+    index = OpenStackObjectContainer.index
+    i_am_singular = OpenStackObjectContainer.i_am_singular
+    i_am_plural = OpenStackObjectContainer.i_am_plural
+
+    def __init__(self, accounts=None, debug=False):
+        super(OpenStackObjectContainerAuditor, self).__init__(accounts=accounts, debug=debug)
+
+    def check_acls(self, container_item):
+        read_acl = container_item.config.get("read_ACL")
+        write_acl = container_item.config.get("write_ACL")
+
+        if read_acl:
+            for acl in read_acl.split(","):
+                if acl == ".r:*":
+                    message = "ACL - World Readable"
+                    self.add_issue(30, message, container_item)
+                elif acl == ".rlistings":
+                    message = "ACL - World Listable"
+                    self.add_issue(10, message, container_item)
+
+        if write_acl:
+            for acl in write_acl.split(","):
+                if acl == "*:*":
+                    message = "ACL - World Writable"
+                    self.add_issue(20, message, container_item)

security_monkey/auditors/openstack/security_group.py

@@ -0,0 +1,33 @@
+#     Copyright (c) 2017 AT&T Intellectual Property. All rights reserved.
+#
+#     Licensed under the Apache License, Version 2.0 (the "License");
+#     you may not use this file except in compliance with the License.
+#     You may obtain a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#     Unless required by applicable law or agreed to in writing, software
+#     distributed under the License is distributed on an "AS IS" BASIS,
+#     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     See the License for the specific language governing permissions and
+#     limitations under the License.
+"""
+.. module: security_monkey.openstack.auditors.security_group
+    :platform: Unix
+
+.. version:: $$VERSION$$
+.. moduleauthor:: Michael Stair <mstair@att.com>
+
+"""
+
+from security_monkey.auditors.security_group import SecurityGroupAuditor
+from security_monkey.watchers.openstack.network.security_group import OpenStackSecurityGroup
+
+class OpenStackSecurityGroupAuditor(SecurityGroupAuditor):
+    index = OpenStackSecurityGroup.index
+    i_am_singular = OpenStackSecurityGroup.i_am_singular
+    i_am_plural = OpenStackSecurityGroup.i_am_plural
+    network_whitelist = []
+
+    def __init__(self, accounts=None, debug=False):
+        super(OpenStackSecurityGroupAuditor, self).__init__(accounts=accounts, debug=debug)