AWS Resource Discovery Scripts

A collection of shell scripts to discover and audit AWS resources across your accounts. These scripts help identify resources created outside Infrastructure as Code (IaC) tools like Terraform, making it easier to bring your infrastructure under version control.

Features

Individual Resource Discovery: Run targeted scans for specific AWS resource types
Master Aggregation Script: Scan all resource types at once with consolidated reporting
Multiple Output Formats: JSON (raw data), Table (quick view), or Human-readable (detailed)
Draw.io Infrastructure Diagrams: Auto-generate visual architecture diagrams in draw.io format
Timestamped Reports: All outputs are timestamped for audit trails
Minimal Dependencies: Uses only AWS CLI, Python3, and standard Unix tools (jq, bash)

Resource Types Supported

Resource Type	Script	Description
IAM Roles	`discover-iam-roles.sh`	Lists non-AWS managed roles with attached policies
OIDC Providers	`discover-oidc-providers.sh`	Discovers IAM OIDC identity providers (GitLab, GitHub, etc.)
IAM Policies	`discover-iam-policies.sh`	Lists customer-managed IAM policies
EC2 Instances	`discover-ec2-instances.sh`	Discovers EC2 instances with state and metadata
S3 Buckets	`discover-s3-buckets.sh`	Lists S3 buckets with encryption and versioning status
VPCs	`discover-vpcs.sh`	Discovers VPCs with CIDR blocks and subnet counts

Prerequisites

Required Tools

AWS CLI v2 - Installation Guide
jq - JSON processor (Download)
bash - Shell (comes with Linux/macOS)

Install on Ubuntu/Debian:

sudo apt update
sudo apt install awscli jq

Install on macOS:

brew install awscli jq

AWS Credentials

Configure AWS CLI with your credentials:

# Using AWS SSO (recommended)
aws configure sso

# Or using access keys
aws configure

Verify your credentials:

aws sts get-caller-identity

IAM Permissions Required

The scripts require read-only IAM permissions for the resources you want to discover:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:ListRoles",
        "iam:GetRole",
        "iam:ListAttachedRolePolicies",
        "iam:ListOpenIDConnectProviders",
        "iam:GetOpenIDConnectProvider",
        "iam:ListPolicies",
        "iam:GetPolicy",
        "ec2:DescribeInstances",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeRouteTables",
        "s3:ListAllMyBuckets",
        "s3:GetBucketLocation",
        "s3:GetBucketVersioning",
        "s3:GetBucketEncryption",
        "s3:GetPublicAccessBlock"
      ],
      "Resource": "*"
    }
  ]
}

Installation

Clone the repository:

git clone https://github.com/hmbldv/aws-scripts.git
cd aws-scripts

Navigate to the scripts directory:

cd scripts

All scripts are executable and ready to run.

Usage

Individual Scripts

Each script can be run independently to discover a specific resource type:

# Discover IAM roles
./discover-iam-roles.sh [--format json|table|human]

# Discover OIDC providers
./discover-oidc-providers.sh [--format json|table|human]

# Discover IAM policies
./discover-iam-policies.sh [--format json|table|human]

# Discover EC2 instances
./discover-ec2-instances.sh [--format json|table|human]

# Discover S3 buckets
./discover-s3-buckets.sh [--format json|table|human]

# Discover VPCs
./discover-vpcs.sh [--format json|table|human]

Master Aggregation Script

Run all discovery scripts at once:

./discover-all.sh [--format json|table|human] [--account-name NAME]

Examples:

# Human-readable format (default)
./discover-all.sh human production-account

# JSON format for programmatic parsing
./discover-all.sh json

# Table format for quick console viewing
./discover-all.sh table

Output Formats

JSON Format

Raw AWS CLI JSON output suitable for programmatic processing:

./discover-iam-roles.sh json

Output: output/iam-roles_20251124_143022.json

Use case: Parse with jq, import into databases, feed into other tools

Table Format

AWS CLI table output for quick visual inspection:

./discover-ec2-instances.sh table

Output: Console table + output/ec2-instances_20251124_143022.txt

Use case: Quick glance at resource names and IDs

Human-Readable Format

Detailed, formatted reports with all resource metadata:

./discover-s3-buckets.sh human

Output: output/s3-buckets_20251124_143022_human.txt

Example:

S3 Buckets Discovery Report
===========================

Bucket: terraform-state-266735821834-us-west-1
  Region: us-west-1
  Created: 2025-11-21T17:34:56.000Z
  Versioning: Enabled
  Encryption: AES256
  Public Access Blocked: true

Use case: Audit reports, documentation, sharing with teammates

Report Structure

Master Aggregation Output

When running discover-all.sh, a timestamped report directory is created:

output/
└── report_20251124_143022/
    ├── SUMMARY.txt                          # Resource counts and metadata
    ├── CONSOLIDATED_REPORT.txt              # All human-readable reports combined
    ├── aws-infrastructure.drawio            # Visual infrastructure diagram (NEW!)
    ├── discovery.log                        # Script execution log
    ├── iam-roles_20251124_143022.json       # Raw IAM roles data
    ├── iam-roles_20251124_143022_human.txt  # Human-readable IAM roles
    ├── oidc-providers_20251124_143022.json
    ├── oidc-providers_20251124_143022_human.txt
    ├── ec2-instances_20251124_143022.json
    ├── ec2-instances_20251124_143022_human.txt
    ├── s3-buckets_20251124_143022.json
    ├── s3-buckets_20251124_143022_human.txt
    ├── vpcs_20251124_143022.json
    ├── vpcs_20251124_143022_human.txt
    ├── iam-policies_20251124_143022.json
    └── iam-policies_20251124_143022_human.txt

SUMMARY.txt Example

AWS Resource Discovery Summary Report
======================================

Account Information
-------------------
Account ID: 266735821834
Account Name: production-account
Region: us-west-1
Caller Identity: arn:aws:sts::266735821834:assumed-role/...
Scan Date: 2025-11-24 14:30:22

Resource Counts
---------------
IAM Roles (non-AWS): 12
OIDC Providers: 1
IAM Policies (Customer-Managed): 5
EC2 Instances: 4
S3 Buckets: 3
VPCs: 1

Draw.io Infrastructure Diagram

The master aggregation script automatically generates a visual infrastructure diagram in draw.io format. This diagram includes:

Diagram Features:

VPC Container: Shows your VPC with CIDR block as a swimlane
EC2 Instances: Color-coded boxes (green=running, red=stopped) with instance details
S3 Buckets: Cylinder shapes positioned outside the VPC
IAM Roles: Hexagon shapes showing roles and their purposes
Auto-layout: Resources automatically positioned for clarity

How to use:

Open the generated aws-infrastructure.drawio file
Visit https://app.diagrams.net (or use desktop app)
Click "Open Existing Diagram"
Select the .drawio file from your report directory
Edit, annotate, and export as needed (PNG, PDF, SVG)

Standalone Usage:

You can also run the diagram generator independently:

# Generate diagram for current report directory
python3 scripts/generate-drawio.py output/report_20251124_143022/

# Or specify custom output directory
python3 scripts/generate-drawio.py /path/to/output/

The diagram is perfect for:

Architecture documentation
Security reviews and presentations
Onboarding new team members
Visualizing multi-account AWS Organizations

Use Cases

1. Terraform Import Discovery

Problem: You have AWS resources created manually or via CloudFormation that need to be managed by Terraform.

Solution:

# Run discovery
./discover-all.sh human production

# Review CONSOLIDATED_REPORT.txt to identify unmanaged resources
cat output/report_*/CONSOLIDATED_REPORT.txt

# Use terraform import for each resource
terraform import aws_iam_role.devops_operator devops-operator

2. Security Audit

Problem: Quarterly security audit requires documentation of all IAM roles and their permissions.

Solution:

# Generate audit report
./discover-iam-roles.sh human
./discover-iam-policies.sh human

# Share human-readable reports with security team

3. Multi-Account Scanning

Problem: Your organization has multiple AWS accounts and you need to inventory resources across all of them.

Solution:

# Loop through AWS profiles
for PROFILE in prod staging dev; do
  export AWS_PROFILE=$PROFILE
  ./discover-all.sh human $PROFILE
done

# Compare reports across accounts
diff output/report_prod_*/SUMMARY.txt output/report_staging_*/SUMMARY.txt

4. Cost Optimization

Problem: Identify unused or forgotten resources that are incurring costs.

Solution:

# Discover EC2 instances
./discover-ec2-instances.sh human

# Review for stopped instances
grep -A 7 "State: stopped" output/*ec2-instances*_human.txt

# Discover S3 buckets and check for empty/unused ones
./discover-s3-buckets.sh human

Advanced Usage

Filtering Results with jq

Use jq to filter JSON output:

# Find only running EC2 instances
./discover-ec2-instances.sh json
cat output/ec2-instances_*.json | jq '.Reservations[].Instances[] | select(.State.Name=="running")'

# Get all IAM roles with "admin" in the name
./discover-iam-roles.sh json
cat output/iam-roles_*.json | jq '.Roles[] | select(.RoleName | contains("admin"))'

# List S3 buckets without encryption
./discover-s3-buckets.sh json
# (then parse with jq to check encryption settings)

Cross-Region Discovery

The scripts use your configured AWS region by default. To scan multiple regions:

for REGION in us-west-1 us-east-1 eu-west-1; do
  export AWS_DEFAULT_REGION=$REGION
  ./discover-all.sh human "prod-$REGION"
done

Automated Scheduled Scans

Set up a cron job to run daily scans:

# Edit crontab
crontab -e

# Add daily scan at 2 AM
0 2 * * * cd /home/user/aws-scripts/scripts && ./discover-all.sh human daily-scan >> /var/log/aws-discovery.log 2>&1

Troubleshooting

Error: "Token has expired and refresh failed"

Problem: AWS SSO session expired

Solution:

aws sso login

Error: "An error occurred (AccessDenied)"

Problem: Insufficient IAM permissions

Solution: Ensure your IAM role/user has the required read permissions (see Prerequisites)

Error: "jq: command not found"

Problem: jq is not installed

Solution:

# Ubuntu/Debian
sudo apt install jq

# macOS
brew install jq

Empty Output

Problem: No resources found

Possible causes:

Wrong AWS region configured
Resources don't exist in this account
Filters excluding all results

Solution:

# Verify account
aws sts get-caller-identity

# Check region
aws configure get region

Project Structure

aws-scripts/
├── README.md                        # This file
├── scripts/                         # Discovery scripts
│   ├── discover-all.sh              # Master aggregation script
│   ├── discover-iam-roles.sh        # IAM roles discovery
│   ├── discover-oidc-providers.sh   # OIDC providers discovery
│   ├── discover-iam-policies.sh     # IAM policies discovery
│   ├── discover-ec2-instances.sh    # EC2 instances discovery
│   ├── discover-s3-buckets.sh       # S3 buckets discovery
│   ├── discover-vpcs.sh             # VPCs discovery
│   └── generate-drawio.py           # Draw.io diagram generator
├── output/                          # Generated reports (gitignored)
└── docs/                            # Additional documentation

Contributing

Contributions are welcome! To add a new resource discovery script:

Follow the existing script template structure
Support all three output formats: json, table, human
Add error handling with set -e
Include timestamps in output files
Update this README with the new resource type
Test with multiple AWS accounts and regions

Roadmap

Completed

Draw.io diagram export - Auto-generate visual architecture diagrams

Planned Enhancements

License

MIT License - Free to use and modify

Credits

Author: hmbldv Purpose: DevSecOps resource discovery and IaC migration Repository: https://github.com/hmbldv/aws-scripts

Related Projects

Terraformer - Generate Terraform files from existing infrastructure
Former2 - Web-based tool to generate IaC from AWS resources
aws-nuke - Delete all resources in an AWS account
CloudMapper - AWS environment visualization

Built with: ❤️ and bash Last Updated: 2025-11-24

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
scripts		scripts
.gitignore		.gitignore
README.md		README.md

hmbldv/aws-scripts

Folders and files

Latest commit

History

Repository files navigation

AWS Resource Discovery Scripts

Features

Resource Types Supported

Prerequisites

Required Tools

AWS Credentials

IAM Permissions Required

Installation

Usage

Individual Scripts

Master Aggregation Script

Output Formats

JSON Format

Table Format

Human-Readable Format

Report Structure

Master Aggregation Output

SUMMARY.txt Example

Draw.io Infrastructure Diagram

Use Cases

1. Terraform Import Discovery

2. Security Audit

3. Multi-Account Scanning

4. Cost Optimization

Advanced Usage

Filtering Results with jq

Cross-Region Discovery

Automated Scheduled Scans

Troubleshooting

Error: "Token has expired and refresh failed"

Error: "An error occurred (AccessDenied)"

Error: "jq: command not found"

Empty Output

Project Structure

Contributing

Roadmap

Completed

Planned Enhancements

License

Credits

Related Projects

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages