Skip to content
Be the first to know on a need-to-know basis.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Captures
static/assets
LICENSE Update Dec 30, 2018
README.md
aa_adhoc.py Add flag for max redirects. Format domains with protocol if missing. Jan 25, 2019
aa_certstream.py Add flag to specify a different certstream server Feb 8, 2019
aa_urlscan.py Query a string in the filename instead of extension. Add flag for max… Jan 25, 2019
aa_whoisds.py Sort renewly registered domains before adding to the queue Jan 19, 2019
commons.py Update summary function to show ctl_server. Cleanup and reorder funct… Feb 8, 2019
config.yaml Update Feb 8, 2019
confusables.py
dns_twist.sh Initial commit Jan 18, 2019
py_pkg_update.sh
requirements.txt

README.md

[Analyst Arsenal (A²)™]

Be the first to know on a need-to-know basis.

Description

With aa_adhoc, run through a list of URLs and check sites for malicious files based on predefined file extensions.

With aa_certstream, find out when a phishing kit has been staged on a domain. With this information, you can be amongst the first to:

  • Know
  • Block
  • Report
  • Analyze

aa_certstream

With aa_urlscan, easily search urlscan.io and check sites for malicious files based on predefined file extensions.

With aa_whoisds, download a list of newly registered domains from WHOIS Domain Search, score the domains, and search for signs of malicious activity.

Prerequisites

  • Ubuntu 18.04+ (should work on other Linux distros)
  • Python 2.7.14
  • DEB Packages:
    • gcc
    • Git (optional)
    • Torsocks (optional: used with flag --tor)

Setup

  1. Open a terminal and run the following command:
    git clone https://github.com/ecstatic-nobel/analyst_arsenal.git  
    cd analyst_arsenal  
    bash py_pkg_update.sh  

Usage

aa_adhoc
The following command will:

  • Make requests to the domains retrieved from a file
  • Download files from the site when an open directory is found hosting a file with the desired file extension

1 positional arguments needed:

  • Input File : Path to the file containing URLs

Optional arguments:

  • --directory : Download data to CAP_DIR (default: ./Captures)
  • --level : Recursion depth (default=1, infinite=0)
  • --max-redirect : Maximum redirects (default=0)
  • --quiet : Don't show wget output
  • --threads : Numbers of threads to spawn
  • --timeout : Set the connection timeout to TIMEOUT
  • --tor : Download files via the Tor network
  • --very-verbose : Show error messages
python aa_adhoc.py <INPUT_FILE> [--directory] [--level] [--max-redirect] [--quiet] [--threads] [--timeout] [--tor] [--very-verbose]  

aa_certstream
The following command will:

  • Stream CT logs via Certstream
  • Score and add suspicious domains to a queue while other domains continue to be scored
  • Simultaneously make requests to the domains in the queue to search for predefined file extensions
  • Recursively download the site when an open directory is found hosting a file with a particular extension

Optional arguments:

  • --ctl-server : Certstream server URL to connect to
  • --dns-twist : Check the twisted keywords found in dns_twisted.yaml
  • --directory : Download data to CAP_DIR (default: ./Captures)
  • --level : Recursion depth (default=1, infinite=0)
  • --log-nc : File to store domains that have not been checked
  • --quiet : Don't show wget output
  • --score : Minimum score to trigger a session (Default: 75)
  • --threads : Numbers of threads to spawn
  • --timeout : Set the connection timeout to TIMEOUT
  • --tor : Download files via the Tor network
  • --verbose : Show domains being scored
  • --very-verbose : Show error messages
python aa_certstream.py [--ctl-server] [--dns-twist] [--directory] [--level] [--log-nc] [--quiet] [--score] [--threads] [--timeout] [--tor] [--verbose] [--very-verbose]  

aa_urlscan
The following command will:

  • Make requests to the domains retrieved from urlscan.io
  • Recursively download the site when an open directory hosting a file with the desired file extension

3 positional arguments needed:

  • Query Type : automatic, manual, certstream, openphish, phishtank, twitter, urlhaus
  • Delta : Number of days back to search (GMT)
  • Query String : String to search (and does not include spaces)

Optional arguments:

  • --directory : Download data to CAP_DIR (default: ./Captures)
  • --level : Recursion depth (default=1, infinite=0)
  • --max-redirect : Maximum redirects (default=0)
  • --quiet : Don't show wget output
  • --threads : Numbers of threads to spawn
  • --timeout : Set the connection timeout to TIMEOUT
  • --tor : Download files via the Tor network
  • --very-verbose : Show error messages
python aa_urlscan.py <QUERY_TYPE> <DELTA> <QUERY_STRING> [--directory] [--level] [--max-redirect] [--quiet] [--threads] [--timeout] [--tor] [--very-verbose]  

Note: If the path is a file, it will be automatically downloaded.

aa_whoisds

  • Download a list of newly registered domains from WHOIS Domain Search (whoisds.com)
  • Score and add suspicious domains to a queue while other domains continue to be scored
  • Simultaneously make requests to the domains in the queue to search for predefined file extensions
  • Recursively download the site when an open directory is found hosting a file with a particular extension

1 positional argument needed:

  • Delta : Number of days back to search (GMT)

Optional arguments:

  • --dns-twist : Check the twisted keywords found in dns_twisted.yaml
  • --directory : Download data to CAP_DIR (default: ./Captures)
  • --level : Recursion depth (default=1, infinite=0)
  • --log-nc : File to store domains that have not been checked
  • --quiet : Don't show wget output
  • --score : Minimum score to trigger a session (Default: 75)
  • --threads : Numbers of threads to spawn
  • --timeout : Set the connection timeout to TIMEOUT
  • --tor : Download files via the Tor network
  • --verbose : Show domains being scored
  • --very-verbose : Show error messages
python aa_whoisds.py <DELTA> [--dns-twist] [--directory] [--level] [--log-nc] [--quiet] [--score] [--threads] [--timeout] [--tor] [--verbose] [--very-verbose]  

Things to know

  • Be responsible!!!
  • Output messages:
    • Complete: download complete or the site canceled it prematurely
    • Critical: a domain was found with a score above 120
    • Directory: the output directory is unavailable
    • Download: checks passed and a download was started
    • Empty: the output directory was empty and removed
    • Failed: a connection to the site couldn't be made
    • Session: checking the site for data included in external.yaml
    • Suspicious: a domain was found with a score above 90
    • Triggered: a domain was found with the minimum score specified
  • Check the queue_file.txt file to get a better understanding of how large the queue is. If it's too large, either increase the threads, raise the score, or decrease the level.
  • If the keywords in config.yaml have been modified and --dns-twist is going to be used, regenerate dns_twisted.yaml by running the following command:
    bash dnstwist.sh PATH_TO_DNSTWIST_SCRIPT
  • Using the --dns-twist flag will default to a minimum of 20 threads
  • Downloads via Tor happen over 127.0.0.1:9050
  • These scripts will not check Torsocks settings

Please fork, create merge requests, and help make this better.

You can’t perform that action at this time.