Web Crawler

Web Crawler is a Python script designed to fetch URLs and check given domain names.

Description

The Web Crawler is a Python-based application designed to crawl web pages and retrieve URLs similar to web.archive.org. It provides functionalities to fetch URLs from specified hosts or a list of domains and to check the status of given domain names.

Key Features:

• URL Fetching: Fetch URLs from a specific host or a list of domains.
• Domain Name Checking: Check the status of given domain names.
• Multithreading Support: Utilize multiple threads to improve performance.
• Subdomain Crawling: Option to enable or disable subdomain crawling.
• Output File Saving: Save the results to an output file for further analysis.

Requirements

• Python 3.x

Installation

1. Clone the repository:

git clone https://github.com/MrHarshvardhan/Web_Crawler

Arguments

• function: Specify the function to execute (pull or check).
• --host HOST: Domain/Host Name.
• --threads THREADS: The number of threads (default: 5).
• --with-subs WITH_SUBS: Specify yes or no (default: yes).
• --loadfile LOADFILE: File location.
• -o OUTPUTFILE, --outputfile OUTPUTFILE: Output file to save results.

Examples

Pull Function

To fetch URLs from a specific host:

python web_crawler.py pull --host example.com

python web_crawler.py -d pull --host example.com

To fetch URLs from a list of domains:

python web_crawler.py pull --loadfile domains.txt

To check the status of given domain names:

python web_crawler.py check --loadfile domains.txt

Additional Options

To specify the number of threads:

python web_crawler.py pull --host example.com --threads 10

To save the results to an output file:

python  web_crawler.py pull --host example.com --outputfile results.txt