Simple Web Crawler Tool

A user-friendly desktop application built with Python and Tkinter that crawls a single web page, extracts all hyperlinks, and saves a structured breakdown of those links to a local text file.

Overview

This GUI-based tool lets you enter a URL, fetches the page HTML, parses all anchor tags (), and records for each link:

• Full URL
• Base domain (netloc)
• Path
• Query parameters (key/value lists)

It’s useful for quick audits, link analysis, or learning how web pages structure URLs.

Functions (Code-Level)

All logic lives in web_crawler_tool.py inside the WebCrawlerTool class:

• __init__(self, root)

  • Builds the Tkinter UI, sets title, window size, icon (img.png), creates input fields and buttons, and wires event handlers.

• gradient_background(self)

  • Draws a simple gradient-like background using a Canvas and sets a resize handler.

• on_resize(self, event)

  • Redraws the gradient when the window size changes.

• on_button_hover(self, event) / on_button_leave(self, event)

  • Changes the Start button color on hover for better UX.

• generate_output_file_name(self, url)

  • Sanitizes the URL’s domain into a filesystem-safe string and returns <sanitized_domain>_crawl_output.txt.

• start_crawling(self)

  • Core workflow:
    1. Reads the URL from the input box.
    2. Validates it’s non-empty.
    3. Fetches the page via requests.get(url).
    4. Parses HTML with BeautifulSoup(..., 'html.parser').
    5. Iterates all <a href=...> links and, for each:
       • Parses with urllib.parse.urlparse and parse_qs.
       • Writes Full URL, Base URL (netloc), Path, and Parameters to output file.
    6. Shows success or error via message boxes and status label.

The application entrypoint is at the bottom of the file:

• Creates the Tk root, instantiates WebCrawlerTool, and starts the Tk event loop (root.mainloop()).

Where Output Goes

• Output file is saved in the same directory where you run the script.
• Filename pattern: <domain>_crawl_output.txt (example: www_example_com_crawl_output.txt).

Example Output Snippet

Crawled URL: https://www.example.com
==================================================

Full URL: /about-us
Base URL: 
Path: /about-us
Parameters:
--------------------------------------------------

Full URL: https://www.iana.org/domains/example
Base URL: www.iana.org
Path: /domains/example
Parameters:
--------------------------------------------------

Libraries Used

• tkinter (standard library) — GUI
• requests — HTTP requests
• beautifulsoup4 — HTML parsing
• urllib.parse (standard library) — URL parsing
• re (standard library) — filename sanitization

Install third-party dependencies:

pip install requests beautifulsoup4

Setup

1. Ensure Python 3.x is installed.
2. Clone or download this repository.
3. Place img.png in the same directory as web_crawler_tool.py (used for the window icon). If you don’t want an icon, remove the PhotoImage/iconphoto lines in the code.
4. Install dependencies:
   • Windows (PowerShell):

     py -m pip install --upgrade pip
     py -m pip install requests beautifulsoup4
     

   • macOS/Linux:

     python3 -m pip install --upgrade pip
     python3 -m pip install requests beautifulsoup4
     

Optional: use a virtual environment

python -m venv .venv
# Windows PowerShell
.\.venv\Scripts\Activate.ps1
pip install requests beautifulsoup4

How to Run

• Windows (PowerShell):

  py web_crawler_tool.py
  

• macOS/Linux:

  python3 web_crawler_tool.py
  

How to Use

1. Enter a full URL (including http:// or https://), e.g. https://www.example.com.
2. Click “Start Crawling”.
3. When finished, check the generated <domain>_crawl_output.txt in the current directory.

Notes & Limitations

• Crawls only the provided page (no recursive crawling to other pages).
• If links are relative (e.g., /about), they will appear as-is in the “Full URL” field.
• Some sites may block automated requests; respect site policies and robots.txt.
• Requires internet connectivity.

Troubleshooting

• requests.exceptions.* errors: Check your internet connection and URL correctness.
• Empty or few results: The target page may render links via JavaScript (not handled by this simple HTML parser).
• Icon error: Ensure img.png exists or remove the icon lines in the code.

Roadmap Ideas

• Depth-limited recursive crawling.
• Resolve relative links to absolute using the base page URL.
• Export to CSV/JSON.
• In-app results preview.
• Basic robots.txt handling and rate limiting.

Author

Created by Kalpesh Parashar.

License

MIT License.