New SQLite Output and Powerpipe Mod! (#101)

* ignore-db-files * readme * readme * lib/config/configuration.py * support-sqlite-and-hardcode-regions * output-sqlite * delete-santandard-arn * add-powerpipe-mod * fix-outputs * powerpipe-imgs * fix-empty-instance-profile * avoid-unique-errors * README * imgs * imgs * imgs * imgs * imgs * imgs * readme
gabrielsoltz · Apr 6, 2024 · 853fa09 · 853fa09
1 parent e04155c
commit 853fa09
Show file tree

Hide file tree

Showing 16 changed files with 1,376 additions and 102 deletions.
diff --git a/.gitignore b/.gitignore
@@ -136,6 +136,8 @@ outputs/*.html
 outputs/*.csv
 outputs/*.json
 outputs/*.xlsx
+outputs/*.db
+
 
 # Local .terraform directories
 **/.terraform/*

diff --git a/README.md b/README.md
@@ -36,6 +36,13 @@
 
 **MetaHub** is an open-source security tool for **impact-contextual vulnerability management**. It can automate the process of **contextualizing** security findings based on your environment and your needs, YOUR **context**, identifying **ownership**, and calculate an **impact scoring** based on it that you can use for defining prioritization (where should you start?) and automations like remediations, alerts or tickets. The tool is for AWS environments and you can use it with [AWS Security Hub](https://aws.amazon.com/security-hub) or any [ASFF](https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-findings-format.html) compatible scanners (like [Prowler](https://github.com/prowler-cloud/prowler)).
 
+> :information_source: Explore and extend the new [MetaHub Dashboards using Powerpipe](#use-it-with-powerpipe)!
+
+<p float="left">
+  <img src="docs/imgs/powerpipe_findings.png" width="500" />
+  <img src="docs/imgs/powerpipe_resources.png" width="500" />
+</p>
+
 **MetaHub** describes your [**context**](#context) by connecting to your affected resources in your affected accounts. It can describe information about your AWS account and organization, the affected resources tags, related CloudTrail events, your affected resource configurations **and all their associations**: if you are contextualizing a security finding affecting an EC2 Instance, **MetaHub** will not only connect to that instance itself but also its IAM Roles; from there, it will connect to the IAM Policies associated with those roles. It will connect to the Security Groups and analyze all their rules, the VPC and the Subnets where the instance is running, the Volumes, the Auto Scaling Groups, and more. You can apply [**filters**](#filters) to automate detecting other resources with the same properties and do in-depth investigations.
 
 After fetching all the information from your **context**, **MetaHub** will evaluate the [**impact**](#impact) conditions for all your resources: [**exposure**](#exposure), [**access**](#access), [**encryption**](#encryption), [**status**](#status), [**environment**](#environment), [**application**](#application), and [**owner**](#owner) and based on those calculations and in addition to the information about the security [**findings**](#findings) affecting the resource all **together**, **MetaHub** will generate a **score** for each finding and affected resource.
@@ -540,7 +547,7 @@ The impact module in MetaHub focuses on understanding the 7 key properties of th
 
 ## Exposure
 
-**Exposure** evaluates the how the the affected resource is exposed to other networks. For example, if the affected resource is public, if it is part of a VPC, if it has a public IP or if it is protected by a firewall or a security group.
+**Exposure** evaluates how the the affected resource is exposed to other networks. For example, if the affected resource is effectively public or just public, if it is part of a VPC, if it has a public IP or if it is protected by a firewall or a security group.
 
 | **Possible Statuses**   | **Value** | **Description**                                                                                                |
 | ----------------------- | :-------: | -------------------------------------------------------------------------------------------------------------- |
@@ -1047,6 +1054,7 @@ If you want to generate `json-short`, `json-full` and `html` outputs, you can us
 - [HTML](#html)
 - [CSV](#csv)
 - [XLSX](#xlsx)
+- [SQLite](#sqlite)
 
 ## JSON
 
@@ -1322,6 +1330,87 @@ For example, you can generate an HTML output with Tags and add "Owner" and "Envi
 ./metahub --output-modes html --output-tag-columns Owner Environment
 ```
 
+## SQLite
+
+**MetaHub** can save the findings and context in a SQLite database. This database can be used for further analysis, reporting, and integration with other tools, like [Powerpipe](https://powerpipe.io/). The database will be saved in the `outputs/` folder with the execution date and extension `.db`.
+
+### Use it with Powerpipe
+
+You can use the SQLite database generated by MetaHub with [Powerpipe](https://powerpipe.io/). You can find MetaHub PowerPipe mod under the folder `powerpipe/`.
+
+<p float="left">
+  <img src="docs/imgs/powerpipe_findings.png" width="500" />
+  <img src="docs/imgs/powerpipe_resources.png" width="500" />
+</p>
+
+To use it, you need to have Powerpipe [installed in your system](https://powerpipe.io/downloads).
+
+Once installed, you can run the following command:
+
+```sh
+powerpipe server --database sqlite:../outputs/metahub-20240331-182942.db
+```
+
+### Build your own dashboards
+
+Building dashboards with Powerpipe is supper easy and powerfull. You can check the [Powerpipe documentation](https://powerpipe.io/docs) for more information.
+
+Below you will find the tables available in the SQLite database.
+
+#### Resources
+
+| Column Name                     | Data Type | Description                                                                                    |
+| ------------------------------- | --------- | ---------------------------------------------------------------------------------------------- |
+| resource_arn                    | VARCHAR   | The Amazon Resource Name (ARN) of the resource, serving as the primary key.                    |
+| resource_type                   | VARCHAR   | The type of the resource.                                                                      |
+| resource_region                 | VARCHAR   | The region the resource is located in.                                                         |
+| resource_account_id             | VARCHAR   | The account ID associated with the resource. Foreign key that references accounts(account_id). |
+| resource_account_alias          | VARCHAR   | The alias of the account associated with the resource.                                         |
+| resource_tags                   | TEXT      | Tags associated with the resource.                                                             |
+| resource_exposure               | VARCHAR   | The exposure level of the resource.                                                            |
+| resource_access                 | VARCHAR   | The access level of the resource.                                                              |
+| resource_encryption             | VARCHAR   | The encryption status of the resource.                                                         |
+| resource_status                 | VARCHAR   | The current status of the resource.                                                            |
+| resource_application            | VARCHAR   | The application associated with the resource.                                                  |
+| resource_environment            | VARCHAR   | The environment (e.g., production, staging) the resource is in.                                |
+| resource_owner                  | VARCHAR   | The owner of the resource.                                                                     |
+| resource_score                  | INTEGER   | A score associated with the resource.                                                          |
+| resource_findings_score         | INTEGER   | A score based on findings associated with the resource.                                        |
+| resource_findings_critical      | INTEGER   | The number of critical findings.                                                               |
+| resource_findings_high          | INTEGER   | The number of high severity findings.                                                          |
+| resource_findings_medium        | INTEGER   | The number of medium severity findings.                                                        |
+| resource_findings_low           | INTEGER   | The number of low severity findings.                                                           |
+| resource_findings_informational | INTEGER   | The number of informational findings.                                                          |
+
+#### Findings
+
+| Column Name              | Data Type | Description                                                        |
+| ------------------------ | --------- | ------------------------------------------------------------------ |
+| finding_id               | VARCHAR   | The unique identifier for the finding, serving as the primary key. |
+| finding_title            | VARCHAR   | The title of the finding.                                          |
+| finding_severity         | VARCHAR   | The severity of the finding.                                       |
+| finding_workflowstatus   | VARCHAR   | The workflow status of the finding.                                |
+| finding_recordstate      | VARCHAR   | The record state of the finding.                                   |
+| finding_compliancestatus | VARCHAR   | The compliance status of the finding.                              |
+| finding_productarn       | VARCHAR   | The ARN of the product generating the finding.                     |
+| finding_resource_arn     | VARCHAR   | The ARN of the associated resource.                                |
+
+#### Accounts
+
+| Column Name                     | Data Type | Description                                                                   |
+| ------------------------------- | --------- | ----------------------------------------------------------------------------- |
+| account_id                      | INTEGER   | The unique identifier for the account, serving as the primary key.            |
+| account_alias                   | VARCHAR   | The alias of the account.                                                     |
+| account_organizations_id        | VARCHAR   | The ID of the organization the account belongs to.                            |
+| account_organizations_arn       | VARCHAR   | The ARN of the organization the account belongs to.                           |
+| account_master_account_id       | VARCHAR   | The ID of the master account, if this account is part of an AWS Organization. |
+| account_master_account_email    | VARCHAR   | The email address associated with the master account.                         |
+| account_alternate_contact_type  | VARCHAR   | The type of alternate contact (e.g., billing, security).                      |
+| account_alternate_contact_name  | VARCHAR   | The name of the alternate contact.                                            |
+| account_alternate_contact_email | VARCHAR   | The email address of the alternate contact.                                   |
+| account_alternate_contact_phone | VARCHAR   | The phone number of the alternate contact.                                    |
+| account_alternate_contact_title | VARCHAR   | The title of the alternate contact.                                           |
+
 # Filters
 
 You can filter the security findings and resources that you get from your source in different ways and combine all of them to get exactly what you are looking for, then re-use those filters to create automations, alerts, reports, and more.

diff --git a/docs/imgs/powerpipe_findings.png b/docs/imgs/powerpipe_findings.png
diff --git a/docs/imgs/powerpipe_resources.png b/docs/imgs/powerpipe_resources.png
diff --git a/lib/config/configuration.py b/lib/config/configuration.py
@@ -178,7 +178,7 @@
 # You can define the columns that will be displayed in the output HTML, CSV AND XLSX.
 # You can also use `--output-config-columns` and `--output-tags-columns` to override these values.
 # If you want all fields as columns, comment the following lines.
-config_columns = []
+config_columns = ["public_ip"]
 tag_columns = ["Name", "Owner"]
 account_columns = ["Alias"]
 impact_columns = [

diff --git a/lib/context/resources/AwsAutoScalingLaunchConfiguration.py b/lib/context/resources/AwsAutoScalingLaunchConfiguration.py
@@ -32,6 +32,7 @@ def __init__(
         # Associated MetaChecks
         self.security_groups = self._describe_launch_configuration_security_groups()
         self.iam_roles = self._describe_launch_configuration_iam_roles()
+        self.iam_roles = {}
         self.autoscaling_groups = (
             self._describe_launch_configuration_autoscaling_group()
         )
@@ -85,7 +86,8 @@ def _describe_launch_configuration_iam_roles(self):
             arn = IamHelper(
                 self.logger, self.finding, self.sess
             ).get_role_from_instance_profile(instance_profile)
-            iam_roles[arn] = {}
+            if arn:
+                iam_roles[arn] = {}
 
         return iam_roles
 

diff --git a/lib/context/resources/AwsEc2LaunchTemplate.py b/lib/context/resources/AwsEc2LaunchTemplate.py
@@ -94,7 +94,8 @@ def _describe_launch_template_versions_iam_roles(self):
                 arn = IamHelper(
                     self.logger, self.finding, self.sess
                 ).get_role_from_instance_profile(instance_profile)
-                iam_roles[arn] = {}
+                if arn:
+                    iam_roles[arn] = {}
 
         return iam_roles
 

diff --git a/lib/context/resources/ContextHelpers.py b/lib/context/resources/ContextHelpers.py
@@ -28,4 +28,7 @@ def get_role_from_instance_profile(self, instance_profile):
             )
             return False
 
-        return response["InstanceProfile"]["Roles"][0]["Arn"]
+        if response["InstanceProfile"]["Roles"]:
+            return response["InstanceProfile"]["Roles"][0]["Arn"]
+
+        return False
diff --git a/lib/helpers.py b/lib/helpers.py
@@ -6,12 +6,7 @@
 
 import yaml
 
-from lib.AwsHelpers import (
-    get_account_alias,
-    get_account_id,
-    get_available_regions,
-    get_region,
-)
+from lib.AwsHelpers import get_account_alias, get_account_id, get_region
 from lib.config.configuration import sh_default_filters
 from lib.securityhub import set_sh_filters
 
@@ -84,7 +79,37 @@ def get_parser():
     )
     group_security_hub.add_argument(
         "--sh-region",
-        choices=get_available_regions(get_logger("ERROR"), "securityhub"),
+        choices=[
+            "af-south-1",
+            "ap-east-1",
+            "ap-northeast-1",
+            "ap-northeast-2",
+            "ap-northeast-3",
+            "ap-south-1",
+            "ap-south-2",
+            "ap-southeast-1",
+            "ap-southeast-2",
+            "ap-southeast-3",
+            "ap-southeast-4",
+            "ca-central-1",
+            "ca-west-1",
+            "eu-central-1",
+            "eu-central-2",
+            "eu-north-1",
+            "eu-south-1",
+            "eu-south-2",
+            "eu-west-1",
+            "eu-west-2",
+            "eu-west-3",
+            "il-central-1",
+            "me-central-1",
+            "me-south-1",
+            "sa-east-1",
+            "us-east-1",
+            "us-east-2",
+            "us-west-1",
+            "us-west-2",
+        ],
         default=[],
         help="Specify the AWS Region where Security Hub is running",
         required=False,
@@ -220,6 +245,7 @@ def get_parser():
             "csv",
             "xlsx",
             "lambda",
+            "sqlite",
         ],
         default=[
             "json-short",
@@ -229,6 +255,7 @@ def get_parser():
             "html",
             "csv",
             "xlsx",
+            "sqlite",
         ],
         nargs="*",
         help="Specify the Outputs you want to generate. By deafault all of them are enabled. If you only want HTML and XLSX: --output-modes html xlsx",