Skip to content
Repo to experiment with serverless framework and automation
Branch: master
Clone or download
caggle Merge pull request #30 from mozilla/tenable_scans
Implement Tenable.io scanning logic
Latest commit 59e7cae Apr 17, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
examples Deploy and ensure everything is working Apr 17, 2019
lib Putting together handler and some tests Apr 11, 2019
scanners Deploy and ensure everything is working Apr 17, 2019
tests Try to fix tests Apr 16, 2019
vendor/nmap-standalone Reduce uploaded lambda function size Apr 9, 2019
.gitignore Update .gitignore Mar 12, 2019
.travis.yml Deploy and ensure everything is working Apr 17, 2019
CODE_OF_CONDUCT.md Add Mozilla Code of Conduct file Mar 30, 2019
LICENSE
Makefile Update README Apr 17, 2019
README.md Update README Apr 17, 2019
handler.py
kms_policy.json
package.json
requirements.txt
serverless.yml Update README Apr 17, 2019
setup.cfg Address review comments Apr 10, 2019

README.md

vautomator-serverless

Repository to experiment with serverless framework and automation.

This project uses serverless framework and attempts to create a serverless environment that could be used to automate vulnerability assessment tasks from multiple ingestion points, such as on-demand submission of a host via a REST API, regular scanning of a known list of hosts, and opportunistically scanning of hosts appearing in Certificate Transparency logs.

This is under development with more features being added as different branches. The tool currently supports:

  • Addition of a target to the scan queue for port scan by an API endpoint (/ondemand/portscan).
  • Addition of a target to the scan queue for HTTP Observatory scan by an API endpoint (/ondemand/httpobservatory)
  • Addition of a target to the scan queue for TLS Observatory scan by an API endpoint (/ondemand/tlsobservatory)
  • Addition of a target to the scan queue for SSH Observatory scan by an API endpoint (/ondemand/sshobservatory).
  • [OPTIONAL] Addition of a target to the scan queue for a Tenable.io scan by an API endpoint (/ondemand/tenablescan).
  • Performing requested scan type (port, HTTP Observatory, TLS Observatory or SSH Observatory) on hosts in the queue
  • Scheduled port scans from a hard-coded list of hosts (disabled by default)
  • Scheduled HTTP Observatory scans from a hard-coded list of hosts (for PoC purposes, runs once a day)
  • Scheduled TLS Observatory scans from a hard-coded list of hosts (for PoC purposes, runs once a day)
  • Scheduled SSH Observatory scans from a hard-coded list of hosts (for PoC purposes, runs once a day)
  • Manually add a host to the scan queue (for PoC purposes).

All API endpoints are currently protected by an API key. This will be replaced with SSO integration.

Results from all scans are placed in an S3 bucket specified in serverless.yml.

Port scans are performed using a statically compiled nmap binary, packaged within the serverless application.

Note: UDP port scans are not supported as Lamdba functions can not be as root/privileged users.

Get ready to deploy

  1. Install Python3 and aws-cli.

  2. Install serverless framework: npm install -g serverless

  3. Download the repo: git clone https://github.com/mozilla/vautomator-serverless.git && cd vautomator-serverless

  4. Customise your serverless.yml file, in particular the custom and provider sections where you can specify your own S3 bucket name/SQS name/KMS key (if using Tenable.io integration, see step 6) etc. or specify multiple environments, tag your resources etc.

  5. Setup your AWS profile and credentials. An account or role with at least the permissions listed in serverless.yml is required in order to deploy and run this. Currently we are using a role in the Mozilla Infosec Dev AWS account using role assumption.

  6. Once your AWS profile is set up, modify the Makefile to specify your AWS region and AWS profile. Serverless framework supports role assumption, and so does the Makefile, as long as your AWS config and credentials files are setup as per here.

  7. [OPTIONAL] If you want Tenable.io integration via the /ondemand/tenablescan endpoint (otherwise skip to step 8):

    • Create a Tenable.io user account with standard user permissions, and create an API key for this account.

    • Modify the top of the Makefile as follows:

      AWS_PROFILE := <YOUR-AWS-PROFILE/ROLE>
      # Y for Tenable.io support, N or blank if not
      TENABLE_IO := Y / N 
      
      # If you would like to create a dedicated KMS for vautomator
      # Blank if you would like to use default AWS SSM key for
      # encrypted storage
      KMS_POLICY_FILE := <YOUR-KMS-POLICY-JSON-FILE>
      
      # Blank if a policy file is specified, 
      # or if you would like to use default AWS SSM key
      KMS_KEYID := <YOUR-KMS-KEY-ID> 
      
    • Once this is done, run make setup TIOA=<Tenable-Access-Key> TIOS=<Tenable-Secret-Key>. TIOA and TIOS are API keys generated in the first point above. Based on the above values in Makefile, this will create a new or use the default AWS KMS key, and store the Tenable API keys in SSM in encrypted form using the KMS key. NOTE: The most straightforward option is to specify the AWS profile and Y for TENABLE_IO, and leave other variables blank.

  8. [OPTIONAL] run: make validate to check if your serverless.yml is correctly formatted without deploying a stack.

  9. Run make deploy to deploy to AWS!

  10. If you have no CloudFormation errors and if you see Service Information listing your lambda functions/endpoints, you are good to go.

Examples

  • Once deployed properly, you can kick of an ondemand scan on a host, using: API_GW_URL=<YOUR-REST-ENDPOINT> python3 examples/ondemand_tasker.py
Provide the FQDN (Fully Qualified Domain Name) you want to scan: infosec.mozilla.org
INFO:root:Sending POST to <YOUR-REST-ENDPOINT>
INFO:root:Triggered a Port Scan of: infosec.mozilla.org
INFO:root:Sending POST to <YOUR-REST-ENDPOINT>
INFO:root:Triggered an HTTP Observatory Scan of: infosec.mozilla.org
INFO:root:Sending POST to <YOUR-REST-ENDPOINT>
INFO:root:Triggered a TLS Observatory Scan of: infosec.mozilla.org
INFO:root:Sending POST to <YOUR-REST-ENDPOINT>
INFO:root:Triggered an SSH Observatory Scan of: infosec.mozilla.org
INFO:root:Sending POST to <YOUR-REST-ENDPOINT>
INFO:root:Triggered an Tenable Scan of: infosec.mozilla.org

To confirm all scans were performed and results were stored in S3 bucket:

$ aws --profile <YOUR-PROFILE> s3 ls s3://<YOUR-S3-BUCKET> | sort -r
2019-04-10 00:03:17      10487 infosec.mozilla.org_httpobservatory.json
2019-04-10 00:03:22      77028 infosec.mozilla.org_tlsobservatory.json
2019-04-10 00:03:37       5709 infosec.mozilla.org_tcpscan.json
...
  • Scheduled Observatory scans will run once a day: serverless logs -f cronHttpObservatoryScan
START RequestId: 6a3bc71b-e369-498c-849c-f522e79ce734 Version: $LATEST
2019-03-11 17:16:59.974 (+11:00)	6a3bc71b-e369-498c-849c-f522e79ce734	[INFO]	Tasking Observatory Scan of: "infosec.mozilla.org"
2019-03-11 17:17:00.439 (+11:00)	6a3bc71b-e369-498c-849c-f522e79ce734	[INFO]	Completed Observatory Scan of "infosec.mozilla.org"
  • Verify the queued scans actually run: sls logs -f RunScanQueue
START RequestId: 2298e8f2-17ac-5e78-8608-bdf388a42dba Version: $LATEST
END RequestId: 2298e8f2-17ac-5e78-8608-bdf388a42dba
REPORT RequestId: 2298e8f2-17ac-5e78-8608-bdf388a42dba	Duration: 590.75 ms	Billed Duration: 600 ms 	Memory Size: 1024 MB	Max Memory Used: 86 MB
You can’t perform that action at this time.