This is a simple crawler that crawls a domain and returns a list of all the links on the domain. Makes use of the powerful requests library to make HTTP requests and implements multithreading to speed up the process. Also includes an optional password cracker script that looks for any POST endpoint on the domain that require authentication and performs a dictionary attack using Hydra.
python test_script.py <domain>
$ python test_script --help
usage: test_script.py [-h] [-o OUTPUT_DIR] [-t THREADS] [-u USERNAME] [-p PASSWORD_FILE] domain
Simple script to crawl a domain for subdomains and directories
positional arguments:
domain Specify a domain
options:
-h, --help Show this help message and exit
-o OUTPUT_DIR, --output_dir OUTPUT_DIR Specify the output directory
-t THREADS, --threads THREADS Specify the number of threads to use, (default: 10)
-u USERNAME, --username USERNAME Specify the username
-p PASSWORD_FILE, --password_file PASSWORD_FILE Specify the password file
The program will crawl the given domain, checking for all subdomains and directories present in the input_files directory. It will then look through the html to find a list of all files the domain links to . At the end of the program, it will write all valid subdomains, directories, and files to the output_files directory.
git clone https://github.com/ElioaChukri/domain_crawler.git
cd domain_crawler
pip install -r requirements.txt
This script also includes an optional password cracker. It will attempt to crack the password of a given POST endpoint using a given wordlist. It will then output the password to the terminal. The password cracker makes use of the hydra CLI tool to accomplish this.
The password cracker is optional and can be used by adding any of the two -u or -p flags to the command line arguments. It can also be run independently by running the password_cracker.py script. When run independently, the password cracker will check the urls present in the output_files/valid_dirs.bat and output_files/valid_subdomains.bat files for any password protected POST endpoints. It will then attempt to crack the password of the given endpoint using the given wordlist.
python password_cracker.py -u <username> -p <password_file>
Depends on: hydra