Skip to content

Data Collection

Jump to bottom Edit New page
Kieran Smith edited this page Jul 13, 2022 · 4 revisions

Before you can begin collecting data from an AWS account, you'll need to configure an account to use.

Once you're set up, you can run the ingestor, using default options, with: awspx ingest

[root@localhost ~]# awspx ingest

 ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
 │                                 Create profile                                                 │
 ├────────────────────────────────────────────────────────────────────────────────────────────────┤
 │    AWS Access Key ID [None]:                     ***************E9UX5                          │
 │    AWS Secret Access Key [None]:                 ***********************************Vukdh      │
 │    Default region name [None]:                                                                 │
 │    Default output format [None]:                                                               │
 └────────────────────────────────────────────────────────────────────────────────────────────────┘
                               Saved profile 'default'

      Identity: arn:aws:iam::000000000000:user/awspx                ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
      Services: IAM, EC2, S3, Lambda
      Database: default.db
      Account:  000000000000
      Region:   eu-west-1

...

Next, explore this environment.

Required AWS permissions

You'll need to setup an IAM identity for each AWS account you plan on ingesting:

  1. Create an IAM user

  2. Generate an access key

  3. Attach the following built-in managed policies:

    • SecurityAudit (required)
    • ReadOnlyAccess (only if you want to ingest S3 Objects)

To use a role instead, see the --env ingestion option.

Command-line Options

The awspx command-line utility is a bash script (/usr/local/bin/awspx) that wraps cli.py: all commands are forwarded and executed within the context of the awspx docker container hosting Neo4j.

You can use the CLI to manage what is available from the web interface using ingestion, database, and supporting functions.

Argument Description
ingest Ingest data from an AWS account.
db Manage databases used for visualization, ingestion, and attack computation.
profile Manage AWS credential profiles used for ingestion.
attacks Compute attacks using the active database.
update Update awspx to the latest version.

For a full list of command-line options run: awspx --help

Ingestion

The ingestor collects and processes AWS account data, expressing and storing it as a graph. This data includes resources and their relationships. Action information — expressed as Policies, or variants thereof — is automatically resolved; as are the attacks that incorporate them. For additional information on how it all works, and the thinking behind it, check out this blog post.

Argument Description
--assume-role ARN of a role to assume for ingestion (useful for cross-account ingestion).
--assume-role-duration Maximum session duration in seconds (for --assume-role).
--database Database to store results (defaults to <profile>.db).
--env Use AWS credential environment variables.
--include-conditional-attacks Include conditional actions when computing attacks (default False).
--max-attack-depth Maximum search depth for attacks (default None).
--max-attack-iterations Maximum number of iterations to run each attack (default False).
--only-arns Resources to include by ARN, all other resources will be excluded.
--only-attacks Attacks to include by name, all other attacks will be excluded.
--only-types Resource to include by type, all other resource types will be excluded.
--profile Profile to use for ingestion (corresponds to a [section] in ~/.aws/credentials)..
--region Region to ingest (defaults to profile region, or eu-west-1 if not set).
--services One or more services to ingest (eg: IAM EC2 S3 Lambda).
--skip-arns Resources to exclude by ARN.
--skip-attacks Attacks to exclude by name.
--skip-types Resources to exclude by type.
--skip-actions-all Skip policy resolution (actions will not be processed).
--skip-attacks-all Skip attack path computation (it can be run later with awspx attacks).
--quick Skips supplementary ingestion functions (i.e. speed at the cost of infromation).
--verbose Enable verbose output.

Run awspx ingest --help for a full list of ingestion options.

Default options

Running the ingestor without any arguments will default to the following behavior:

  • The profile will be set to default (--profile default).
  • The database will be set to default.db (--db default.db)
  • All services will be ingested
  • Actions will automatically be resolved from Policy and ACL information (--skip-actions-all False)
  • Attack paths are computed automatically (--skip-actions-all False --skip-attacks-all False).

Ingestion examples

  1. Assuming the role Audit, using the profile prd

    awspx ingest --profile prd --assume-role arn:aws:iam::000000000000:role/Audit --assume-role-duration 3600

  2. Using environment variables

    export AWS_ACCESS_KEY_ID="WY4XH9ZUFXVNIIG6C7JN"
export AWS_SECRET_ACCESS_KEY="EDzukCRFBVDgq0//5yr01usTpHhniJHJ"
export AWS_SESSION_TOKEN="keRJJBEWc3usCFS4I7wxq9MT6AYd7A53zN1bhvZ2ojHgfNZbmf5/////NWkyKutOZGw=="
awspx ingest --env

  3. Skipping ALL S3 objects

     awspx ingest --skip-types AWS::S3::Object

  4. Skipping the S3 Buckets logs and pii (and all of their Objects)

      awspx ingest --skip-arns arn:aws:s3:::logs arn:aws:s3:::pii

  5. Ingesting only IAM and EC2 resources, from the region eu-west-2

    awspx ingest --services IAM EC2 --region eu-west-2

  6. Skipping Policy/ACL resolution (attacks will be skipped implicitly)

    awspx ingest --skip-actions-all

Supported services

See Contributing for more information on how to extend this list.

Regions

Some services (e.g. IAM) are region independent, while others (e.g. EC2) are not. You can specify the target region using the --region option, or omit it to fallback to the value associated with the profile. If neither value is set, awspx will default to eu-west-1.

For more information on supported regions, see the relevant AWS docs.

Profiles

Credential profiles essentially wrap aws configure, allowing you to manage and store credentials for those accounts you frequently interact with.

Argument Description
--create Create a new profile using aws configure.
--delete Delete a saved profile.
--list List saved profiles.

Run awspx profile --help for a full list of profile options:

Profile examples

  1. Creating a profile called prd (once completed, it will be available for ingestion, e.g. awspx ingest --profile prd)

    awspx profile --create prd

  2. Deleting a profile called dev

    awspx profile --delete dev

  3. Listing all profiles

    awspx profile --list

Databases

Each database corresponds to a single .db file used by Neo4j and, by extension, awpsx's web interface. These files are created by the ingestor and named according to the profile that was used at runtime.

Run awspx db --help for a full list of available options.

Argument Description
--use Switch to the specified database.
--list List available databases.
--load-zip Create/overwrite database with ZIP file content.
--verbose Enable verbose output.

Database Examples

  1. Listing all available databases

    awspx db --list

  2. Switching the active database to dev.db (you will need to refresh your browser)

    awspx db --use dev.db

  3. Loading the sample database sample.zip into sample.db

    awspx db --load-zip sample.zip

ZIP files

Before ingested data is written to a database, it is saved as a .zip file. This means historical data can be readily loaded without rerunning the ingestor.

To load a ZIP, say sample.zip included with awspx, you can execute awspx db --load-zip sample.zip, which will:

  • Search /opt/awspx/data — or ~/bin/awspx/data on macOS — for sample.zip
  • Set the database to sample.db (based on the last field delimited by _ preceding .zip)
  • Load sample.zip into sample.db, overwriting any preexisting data.

ZIPs do no not include attack information. awspx attacks` must be run separately.

Attacks

Attacks are the result of post-processing graph data to contextualize those actions that together constitute compromise. In graph terms, each new Attack that is discovered will translate to one or more new edges. Since this may give rise to additional attacks, this process is repeated until it converges, or the maximum number of iterations is reached. Attacks are automatically computed as part of the ingestion process unless explicitly excluded with --skip-attacks-all, or implicitly with --skip-actions-all.

Argument Description
--skip-attacks Attacks to exclude by name.
--only-attacks Attacks to include by name, all other attacks will be excluded.
--max-attack-iterations Maximum number of iterations to run each attack (default 5).
--max-attack-depth Maximum search depth for attacks (default None).
--include-conditional-attacks Include conditional actions when computing attacks (default False).
--verbose Enable verbose output.

Run awxpx attacks --help for a full list of available options.

Attack examples

  1. Running through attacks at most once

    awspx attacks --max-attack-iterations 1

  2. Setting the maximum search depth to 0 hops (i.e. directly available actions only)

    awspx attacks --max-attack-depth 0

  3. Utilizing conditional actions as well (i.e. policies including Conditions or Resource fields incorporating variables)

    awspx attacks --include-conditional-attacks

  4. Skipping attacks involving iam:CreatePolicyVersion

    awspx attacks --skip-attack CreatePolicyVersion

  5. Searching for attacks that would directly enable adding a user to a group

    awspx attacks --only-attacks AddUserToGroup
Add a custom footer

1. Setup

2. Data Collection

3. Data Exploration

4. Contributing

5. FAQs

Clone this wiki locally