Merge Development Branch to Master in Preparation for v0.3.0 (#61)

Features Added Plugin System for Community Plugins Pep8 all code base Parallel host acquisition Support ip address or instance id for targeting Separate plugin code to additional python module Handle GPG key installation if not present Support custom incident plans Development Enhancements Flake8 in CI Pipeline Moto Mocking for Outside-In Test Coverage * update exception type in connection.py * flake8 tests to satisfy travis * freeze version of logutils to align with marsho * freeze version of logutils to align with marsho * update script tag in travis * change script in travis to run only cli test * now supports boto profiles as optional arg * suppress pep8 E402 * update tests * various pep8 fixes * add cli.py updates to support profile * various and sundry pep8 fixes * fix typo in readme closes #31 * update to skip memory acquisition if CIDR range not provided closes #35 * flake8 updates to tests * fix under/over indent * add examiner acl plugin for allowing access during acquire * remove unnecessary code from isolate * run new examiner acl plugin if cidr range provided closes #34 * update isolate to fix bug in isolate sg creation * update case lib to use moto instead of live calls * fix pep8 errors * redact failing connection test in lieu of moto mocking * update libraries to handle no boto configuration * fix pep8 error * fixes to gpg insertion and network isolation * add gnupg req * update gitignore * pep8 fixes * log metadata and console output in valid json instead of blob closes #4 * move metadata gather to before isolation preserving SG data. Closes #3 * Feature/plugin loading (#49) * update travis to stub config * support plugin loading for core and community * update travis file * update travis file * update travis file * update travis file * update travis file * pep8 fixes * no need to remove bucket when mocking * freeze logutils at 0.3.3 * fix requirements syntax * Move plugin handling to dedicated repository (#51) * * Move core plugins to dedicated repository. * Update plugin loader to use relative path to aws_ir_plugins module * Add git repository to requirements.txt as stop gap until pypy * Remove unnecessary files * pep8 and test fixes related to bad imports in plans * update travis * Feature/custom plugins (#52) * * Move core plugins to dedicated repository. * Update plugin loader to use relative path to aws_ir_plugins module * Add git repository to requirements.txt as stop gap until pypy * Remove unnecessary files * pep8 and test fixes related to bad imports in plans * support list of plugins as optional arguments * pep8 fixes * update travis.yml * Update key.py * support targeting by instance id * move marsho to git pull vs pypy because not up to date * update test to match targeting format * support targeting by instance id (#56) * support targeting by instance id * move marsho to git pull vs pypy because not up to date * update test to match targeting format * aws_ir takes batch of instances * update cli.py fix bad merge * update to use boto session instead of client with plugin * fix bad merge * update key to have steps to list * fix pep8 and session name * fix pep8 and session name * dynamic plugin loading and end use of klass vs object * pep8 fixes * more pep8 fixes * remove double underscores for extra code style * misc fixes and documentation updates * fix test rename method reference * pep8 fixes * update doc configs and add developer guide * add responder role cloudformation stack * update quickstart to include cloudformation info * udate requirements.txt and setup to point at packages
ThreatResponse · Jul 21, 2017 · 181792e · 181792e
1 parent f80989a
commit 181792e
Show file tree

Hide file tree

Showing 42 changed files with 1,576 additions and 1,318 deletions.
diff --git a/.gitignore b/.gitignore
@@ -3,6 +3,8 @@ __pycache__/
 *.py[cod]
 *$py.class
 
+.idea/
+
 # C extensions
 *.so
 

diff --git a/.travis.yml b/.travis.yml
@@ -2,6 +2,14 @@ language: python
 python:
   - "3.6"
 before_install:
+  - "mkdir -p /home/travis/.aws/"
+  - "echo [default] > /home/travis/.aws/credentials"
+  - "echo aws_access_key_id = sdfhdsfsadfdsfj >> /home/travis/.aws/credentials"
+  - "echo aws_secret_access_key = sdfhdsfsadfdsfj >> /home/travis/.aws/credentials"
+  - "echo [default] > /home/travis/.aws/config"
+  - "echo region = us-west-2 >> /home/travis/.aws/config"
+  - "echo output = json >> /home/travis/.aws/config"
+
   - "pip install flake8"
   - "flake8 aws_ir"
   - "flake8 tests"
@@ -10,3 +18,6 @@ install:
   - "pytest tests/"
   - "python setup.py build"
   - "python setup.py install"
+script:
+  - "pytest tests/"
+
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -10,3 +10,22 @@ All notable changes to this project will be documented in this file.
 
 - Rename `host-compromise` subcommand to `instance-compromise` [#25](https://github.com/ThreatResponse/aws_ir/issues/25).
 - Replace `timesketch_logger` class with python `logging` library [#28](https://github.com/ThreatResponse/aws_ir/issues/28)
+
+## [0.3.0] - 2017-07-20
+### Features Added
+
+* Plugin System for Community Plugins
+* Pep8 all code base
+* Parallel host acquisition
+* Support ip address or instance id for targeting
+* Separate plugin code to additional python module
+* Handle GPG key installation if not present
+* Support custom incident plans
+
+### Development Enhancements
+
+* Flake8 in CI Pipeline
+* Moto Mocking for Outside-In Test Coverage
+
+
+
diff --git a/README.rst b/README.rst
@@ -32,24 +32,28 @@ Optional Arguments
 
 .. code-block:: bash
 
-   $ aws_ir -h
-   usage: aws_ir [-h] [--case-number CASE_NUMBER]
+   usage: aws_ir [-h] [--version] [--verbose] [--profile PROFILE]
+                 [--case-number CASE_NUMBER]
                  [--examiner-cidr-range EXAMINER_CIDR_RANGE]
                  [--bucket-name BUCKET_NAME] [--dry-run]
                  {instance-compromise,key-compromise} ...
-   
+
    Incident Response command line for Amazon Web Services. This command line
-   interface is designed to process instance and key based incursions without delay
+   interface is designed to process host and key based incursions without delay
    or error.
-   
+
    positional arguments:
      {instance-compromise,key-compromise}
        instance-compromise
        key-compromise
-   
+
    optional arguments:
      -h, --help            show this help message and exit
-   
+
+     --version             show program's version number and exit
+     --verbose             log debug messages
+     --profile PROFILE     A named boto profile to use instead of the default
+                           profile.
      --case-number CASE_NUMBER
                            The case number to use., usually of the form
                            "cr-16-053018-2d2d"
@@ -71,25 +75,27 @@ It's single argument is the access key id, he compromised key is disabled via th
 
 .. code-block:: bash
 
-   $ aws_ir key-compromise -h
    usage: aws_ir key-compromise [-h] --access-key-id ACCESS_KEY_ID
-   
+                                [--plugins PLUGINS]
+
    optional arguments:
      -h, --help            show this help message and exit
      --access-key-id ACCESS_KEY_ID
+     --plugins PLUGINS     Run some or all of the plugins in a custom order.
+                           Provided as a comma separated listSupported plugins:
+                           disableaccess_key,revokests_key
 
 Below is the output of running the ``key-compromise`` subcommand.
 
 .. code-block:: bash
 
-   $ aws_ir key-compromise --access-key-id AKIAJGOVL2FIYOG6YFIA
-   2016-07-29 16:53:35,458 - aws_ir.cli - INFO - Initial connection to AmazonWebServices made.
-   2016-07-29 16:53:42,772 - aws_ir.cli - INFO - Inventory AWS Regions Complete 11 found.
-   2016-07-29 16:53:42,772 - aws_ir.cli - INFO - Inventory Availability Zones Complete 27 found.
-   2016-07-29 16:53:42,773 - aws_ir.cli - INFO - Beginning inventory of instances world wide.  This might take a minute...
-   2016-07-29 16:53:49,527 - aws_ir.cli - INFO - Inventory complete.  Proceeding to resource identification.
-   2016-07-29 16:53:54,839 - aws_ir.cli - INFO - Set satus of access key AKIAJGOVL2FIYOG6YFIA to Inactive
-   Processing complete
+   $ aws_ir key-compromise --access-key-id AKIAINLHPIG64YJXPK5A
+   2017-07-20T21:04:01 - aws_ir.cli - INFO - Initialization successful proceeding to incident plan.
+   2017-07-20T21:04:01 - aws_ir.plans.key - INFO - Attempting key disable.
+   2017-07-20T21:04:03 - aws_ir.plans.key - INFO - STS Tokens revoked issued prior to NOW.
+   2017-07-20T21:04:03 - aws_ir.plans.key - INFO - Disable complete.  Uploading results.
+   Processing complete for cr-17-072104-7d5f
+   Artifacts stored in s3://cloud-response-9cabd252416b4e5a893395c533f340b7
 
 Instance Compromise
 *******************
@@ -101,53 +107,110 @@ The ``instance-compromise`` subcommand takes three arguments, the ``instance-ip`
 Currently ``user`` must be capable of passwordless sudo for memory capture to complete.  If ``user`` does not have passwordless sudo capabilities all artifiacts save for the memory capture will be gathered.
 
 .. code-block:: bash
-
    $ aws_ir instance-compromise -h
-   usage: aws_ir instance-compromise [-h] --instance-ip INSTANCE_IP --user USER
-                                 --ssh-key SSH_KEY
-   
+   usage: aws_ir instance-compromise [-h] [--target TARGET] [--targets TARGETS]
+                                     [--user USER] [--ssh-key SSH_KEY]
+                                     [--plugins PLUGINS]
+
    optional arguments:
-     -h, --help            show this help message and exit
-     --instance-ip INSTANCE_IP
-     --user USER           this is the privileged ssh user for acquiring memory
-                           from the instance.
-     --ssh-key SSH_KEY     provide the path to the ssh private key for the user.
+     -h, --help         show this help message and exit
+     --target TARGET    instance-id|instance-ip
+     --targets TARGETS  File of resources to process instance-id or ip-address.
+     --user USER        this is the privileged ssh user for acquiring memory from
+                        the instance. Required for memory only.
+     --ssh-key SSH_KEY  provide the path to the ssh private key for the user.
+                        Required for memory only.
+     --plugins PLUGINS  Run some or all of the plugins in a custom order.
+                        Provided as a comma separated list of supported plugins:
+                        examineracl_host,gather_host,isolate_host,snapsh
+                        otdisks_host,stop_host,tag_host,get_memory
 
 AWS IR saves all forensic artifacts except for disk snapshots in an s3 bucket created for each case.  Disk snapshots are tagged with the same case number as the rest of the rest of the artifacts.
 
 Below is the output of running the ``instance-compromise`` subcommand.
 
 .. code-block:: bash
 
-   $ aws_ir instance-compromise --instance-ip 52.42.254.41 --user ec2-user --ssh-key key.pem
-   2016-07-28 16:02:17,104 - aws_ir.cli - INFO - Initial connection to AmazonWebServices made.
-   2016-07-28 16:02:23,741 - aws_ir.cli - INFO - Inventory AWS Regions Complete 11 found.
-   2016-07-28 16:02:23,742 - aws_ir.cli - INFO - Inventory Availability Zones Complete 27 found.
-   2016-07-28 16:02:23,742 - aws_ir.cli - INFO - Beginning inventory of instances world wide.  This might take a minute...
-   2016-07-28 16:02:30,398 - aws_ir.cli - INFO - Inventory complete.  Proceeding to resource identification.
-   2016-07-28 16:02:35,608 - aws_ir.cli - INFO - Security Group Created sg-a25e0fc4
-   2016-07-28 16:02:35,895 - aws_ir.cli - INFO - Security Group Egress Access Revoked for sg-a25e0fc4
-   2016-07-28 16:02:36,206 - aws_ir.cli - INFO - Access Ingress Added for proto=tcp from=22 to=22 cidr_range=0.0.0.0/0 for sg=sg-a25e0fc4
-   2016-07-28 16:02:36,475 - aws_ir.cli - INFO - Shifted instance into isolate security group.
-   2016-07-28 16:02:37,975 - aws_ir.cli - INFO - Took a snapshot of volume vol-68accce1 to snapshot snap-d5c4e32f
-   2016-07-28 16:02:38,078 - aws_ir.cli - INFO - Attempting run margarita shotgun for ec2-user on 52.42.254.41 with key.pem
-   2016-07-28 16:02:38,592 margaritashotgun.repository [INFO] downloading https://threatresponse-lime-modules.s3.amazonaws.com/lime-4.4.11-23.53.amzn1.x86_64.ko as lime-2016-07-28T16:02:38.591954-4.4.11-23.53.amzn1.x86_64.ko
-   2016-07-28 16:02:39,817 margaritashotgun.memory [INFO] 52.42.254.41: dumping memory to s3://cloud-response-38c5c23e79e24bc8a5d5d79103b312ff/52.42.254.41-mem.lime
-   2016-07-28 16:03:06,466 margaritashotgun.memory [INFO] 52.42.254.41: capture 10% complete
-   2016-07-28 16:03:20,368 margaritashotgun.memory [INFO] 52.42.254.41: capture 20% complete
-   2016-07-28 16:03:35,419 margaritashotgun.memory [INFO] 52.42.254.41: capture 30% complete
-   2016-07-28 16:03:49,523 margaritashotgun.memory [INFO] 52.42.254.41: capture 40% complete
-   2016-07-28 16:04:03,385 margaritashotgun.memory [INFO] 52.42.254.41: capture 50% complete
-   2016-07-28 16:04:18,561 margaritashotgun.memory [INFO] 52.42.254.41: capture 60% complete
-   2016-07-28 16:04:32,104 margaritashotgun.memory [INFO] 52.42.254.41: capture 70% complete
-   2016-07-28 16:04:45,952 margaritashotgun.memory [INFO] 52.42.254.41: capture 80% complete
-   2016-07-28 16:05:05,152 margaritashotgun.memory [INFO] 52.42.254.41: capture 90% complete
-   2016-07-28 16:05:18,778 margaritashotgun.memory [INFO] 52.42.254.41: capture complete: s3://cloud-response-38c5c23e79e24bc8a5d5d79103b312ff/52.42.254.41-mem.lime
-   2016-07-28 16:05:19,306 - aws_ir.cli - INFO - memory capture completed for: ['52.42.254.41'], failed for: []
-   2016-07-28 16:05:19,454 - aws_ir.cli - INFO - Stopping instance: instance_id=i-ef048f40
-   Processing complete : Launch an analysis workstation with the command 
-   
-                   aws_ir -n cr-16-072816-a4d6 create_workstation us-west-2
+   $  aws_ir --examiner-cidr-range '4.4.4.4/32' instance-compromise --target 52.40.162.126 --user ec2-user --ssh-key ~/Downloads/testing-041.pem
+      2017-07-20T21:10:50 - aws_ir.cli - INFO - Initialization successful proceeding to incident plan.
+      2017-07-20T21:10:50 - aws_ir.libs.case - INFO - Initial connection to AmazonWebServices made.
+      2017-07-20T21:11:03 - aws_ir.libs.case - INFO - Inventory AWS Regions Complete 14 found.
+      2017-07-20T21:11:03 - aws_ir.libs.case - INFO - Inventory Availability Zones Complete 37 found.
+      2017-07-20T21:11:03 - aws_ir.libs.case - INFO - Beginning inventory of resources world wide.  This might take a minute...
+      2017-07-20T21:11:03 - aws_ir.libs.inventory - INFO - Searching ap-south-1 for instance.
+      2017-07-20T21:11:05 - aws_ir.libs.inventory - INFO - Searching eu-west-2 for instance.
+      2017-07-20T21:11:05 - aws_ir.libs.inventory - INFO - Searching eu-west-1 for instance.
+      2017-07-20T21:11:06 - aws_ir.libs.inventory - INFO - Searching ap-northeast-2 for instance.
+      2017-07-20T21:11:07 - aws_ir.libs.inventory - INFO - Searching ap-northeast-1 for instance.
+      2017-07-20T21:11:08 - aws_ir.libs.inventory - INFO - Searching sa-east-1 for instance.
+      2017-07-20T21:11:09 - aws_ir.libs.inventory - INFO - Searching ca-central-1 for instance.
+      2017-07-20T21:11:09 - aws_ir.libs.inventory - INFO - Searching ap-southeast-1 for instance.
+      2017-07-20T21:11:10 - aws_ir.libs.inventory - INFO - Searching ap-southeast-2 for instance.
+      2017-07-20T21:11:11 - aws_ir.libs.inventory - INFO - Searching eu-central-1 for instance.
+      2017-07-20T21:11:12 - aws_ir.libs.inventory - INFO - Searching us-east-1 for instance.
+      2017-07-20T21:11:13 - aws_ir.libs.inventory - INFO - Searching us-east-2 for instance.
+      2017-07-20T21:11:13 - aws_ir.libs.inventory - INFO - Searching us-west-1 for instance.
+      2017-07-20T21:11:13 - aws_ir.libs.inventory - INFO - Searching us-west-2 for instance.
+      2017-07-20T21:11:14 - aws_ir.libs.case - INFO - Inventory complete.  Proceeding to resource identification.
+      2017-07-20T21:11:14 - aws_ir.plans.host - INFO - Proceeding with incident plan steps included are ['gather_host', 'isolate_host', 'tag_host', 'snapshotdisks_host', 'examineracl_host', 'get_memory', 'stop_host']
+      2017-07-20T21:11:14 - aws_ir.plans.host - INFO - Executing step gather_host.
+      2017-07-20T21:11:15 - aws_ir.plans.host - INFO - Executing step isolate_host.
+      2017-07-20T21:11:16 - aws_ir.plans.host - INFO - Executing step tag_host.
+      2017-07-20T21:11:17 - aws_ir.plans.host - INFO - Executing step snapshotdisks_host.
+      True
+      2017-07-20T21:11:17 - aws_ir.plans.host - INFO - Executing step examineracl_host.
+      2017-07-20T21:11:19 - aws_ir.plans.host - INFO - Executing step get_memory.
+      2017-07-20T21:11:19 - aws_ir.plans.host - INFO - attempting memory run
+      2017-07-20T21:11:19 - aws_ir.plans.host - INFO - Attempting run margarita shotgun for ec2-user on 52.40.162.126 with /Users/akrug/Downloads/testing-041.pem
+      2017-07-20T21:11:21 - margaritashotgun.repository - INFO - downloading https://threatresponse-lime-modules.s3.amazonaws.com/modules/lime-4.9.32-15.41.amzn1.x86_64.ko as lime-2017-07-21T04:11:21-4.9.32-15.41.amzn1.x86_64.ko
+      2017-07-20T21:11:25 - margaritashotgun.memory - INFO - 52.40.162.126: dumping memory to s3://cloud-response-a0f2d7e68ef44c36a79ccfe4dcef205a/52.40.162.126-2017-07-21T04:11:19-mem.lime
+      2017-07-20T21:15:43 - margaritashotgun.memory - INFO - 52.40.162.126: capture 10% complete
+      2017-07-20T21:19:37 - margaritashotgun.memory - INFO - 52.40.162.126: capture 20% complete
+      2017-07-20T21:23:41 - margaritashotgun.memory - INFO - 52.40.162.126: capture 30% complete
+      2017-07-20T21:28:17 - margaritashotgun.memory - INFO - 52.40.162.126: capture 40% complete
+      2017-07-20T21:32:42 - margaritashotgun.memory - INFO - 52.40.162.126: capture 50% complete
+      2017-07-20T21:37:18 - margaritashotgun.memory - INFO - 52.40.162.126: capture 60% complete
+      2017-07-20T21:39:18 - margaritashotgun.memory - INFO - 52.40.162.126: capture 70% complete
+      2017-07-20T22:00:13 - margaritashotgun.memory - INFO - 52.40.162.126: capture 80% complete
+      2017-07-20T22:04:19 - margaritashotgun.memory - INFO - 52.40.162.126: capture 90% complete
+      2017-07-20T22:17:32 - margaritashotgun.memory - INFO - 52.40.162.126: capture 100% complete
+      2017-07-20T21:41:52 - aws_ir.plans.host - INFO - memory capture completed for: ['52.40.162.126'], failed for: []
+      2017-07-20T21:41:52 - aws_ir.plans.host - INFO - Executing step stop_host.
+
+   Processing complete for cr-17-072104-7d5f
+   Artifacts stored in s3://cloud-response-a0f2d7e68ef44c36a79ccfe4dcef205a
+
+Instance Compromise -- Isolation Achieved
+*******************
+
+See below that I've connected to the compromised workstation from my examiner IP address.  SSH is all
+that is permitted due to the NACL and Security Group additions.
+
+.. code-block:: bash
+
+   [root@ip-172-31-9-119 ec2-user]# yum install iotop
+   Loaded plugins: priorities, update-motd, upgrade-helper
+   Resolving Dependencies
+   --> Running transaction check
+   ---> Package iotop.noarch 0:0.3.2-7.6.amzn1 will be installed
+   --> Finished Dependency Resolution
+
+   Dependencies Resolved
+
+   iotop-0.3.2-7.6.amzn1.noarch.r FAILED
+   http://packages.us-west-1.amazonaws.com/2017.03/main/201703c0ffee/x86_64/Packages/iotop-0.3.2-7.6.amzn1.noarch.rpm?instance_id=i-0d4216a9fda54fcb6&region=us-west-2: [Errno 12] Timeout on http://packages.us-west-1.amazonaws.com/2017.03/main/201703c0ffee/x86_64/Packages/iotop-0.3.2-7.6.amzn1.noarch.rpm?instance_id=i-0d4216a9fda54fcb6&region=us-west-2: (28, 'Connection timed out after 10000 milliseconds')
+   Trying other mirror.
+   ^C
+
+   Exiting on user cancel
+   [root@ip-172-31-9-119 ec2-user]# ping 4.2.2.2
+   PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.
+   ^C
+   --- 4.2.2.2 ping statistics ---
+   4 packets transmitted, 0 received, 100% packet loss, time 3076ms
+
+   [root@ip-172-31-9-119 ec2-user]#
+
 
 
 User Guide

diff --git a/aws_ir/__init__.py b/aws_ir/__init__.py
@@ -1,15 +1,14 @@
 __version__ = '0.2.2'
 
-import os
+from datetime import datetime
+
 import logging
+import os
 import time
-from datetime import datetime
 
 
 def set_stream_logger(name="aws_ir", level=logging.INFO,
                       format_string=None):
-    """
-    """
 
     if format_string is None:
         format_string = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
@@ -26,12 +25,9 @@ def set_stream_logger(name="aws_ir", level=logging.INFO,
 
 def set_file_logger(case_number, name="aws_ir", level=logging.INFO,
                     base_dir="/tmp", desc="AWS_IR Action"):
-    """
-    """
-
     log_file = "{base_dir}/{case_number}-aws_ir.log".format(
-                   base_dir=base_dir, case_number=case_number
-               )
+        base_dir=base_dir, case_number=case_number
+    )
 
     logger = logging.getLogger(name)
     logger.setLevel(level)
@@ -44,12 +40,11 @@ def set_file_logger(case_number, name="aws_ir", level=logging.INFO,
     fileHandler.setFormatter(fileFormatter)
     logger.addHandler(fileHandler)
 
+
 def wrap_log_file(case_number, base_dir="/tmp"):
-    """
-    """
     log_file = "{base_dir}/{case_number}-aws_ir.log".format(
-                   base_dir=base_dir, case_number=case_number
-               )
+        base_dir=base_dir, case_number=case_number
+    )
     # if log_file exists and is not empty  append a closing "]" to the file
     if os.path.isfile(log_file) and (os.path.getsize(log_file) > 0):
         with open(log_file, 'a') as f:
@@ -63,6 +58,7 @@ def wrap_log_file(case_number, base_dir="/tmp"):
             f.flush()
             f.close()
 
+
 class TimesketchLogger(logging.getLoggerClass()):
 
     def __init__(self, *args, **kwargs):
@@ -71,6 +67,7 @@ def __init__(self, *args, **kwargs):
     def _log(self, level, msg, args, exc_info=None, extra=None):
         super(TimesketchLogger, self)._log(level, msg, args, exc_info=exc_info,
                                            extra=self.__get_times())
+
     def __get_times(self):
         tm = int(time.time())
         dt = datetime.utcfromtimestamp(tm).isoformat()
@@ -82,5 +79,7 @@ class NullHandler(logging.Handler):
     def emit(self, record):
         pass
 
+
 logging.setLoggerClass(TimesketchLogger)
+
 logging.getLogger('aws_ir').addHandler(NullHandler())