Webpage Grabber

A modular Python application for downloading web pages and their assets (CSS, JavaScript, images) with concurrent downloads and progress tracking.

Features

• Downloads complete web pages with all associated assets
• Concurrent asset downloading with progress bars
• Configurable asset types and download settings
• Automatic file organization by asset type
• ZIP file creation of downloaded content
• Robust error handling and retry mechanism
• Progress tracking with tqdm
• Asynchronous I/O operations

Project Structure

production/
├── src/
│   ├── config/            # Configuration classes
│   ├── utils/            # Utility functions and classes
│   ├── downloader/       # Core downloading functionality
│   └── cli/             # Command-line interface
└── tests/               # Unit tests (future addition)

Installation

1. Clone the repository
2. Install requirements:

   pip install -r requirements.txt

Usage

Windows Users

Double-click run_webpage_grabber.bat to run the application.

Command Line Users

Run from terminal/command prompt:

python start.py

The application will:

1. Prompt for a webpage URL
2. Download the webpage and its assets
3. Ask for a location to save the ZIP file
4. Create a ZIP archive with all downloaded content

Configuration

Modify DownloadConfig in src/config/download_config.py to customize:

• Maximum retries for failed downloads
• Timeout settings
• Concurrent download limit
• SSL verification
• User agent string
• Allowed asset types
• Maximum file size

Dependencies

• aiohttp: Asynchronous HTTP client/server
• tqdm: Progress bar
• urllib3: HTTP client
• pathlib: Object-oriented filesystem paths

Contributing

Feel free to submit issues, fork the repository, and create pull requests for any improvements.