Initial version

Selenium Scraper Starter v1.0

This initial version lays the groundwork for building dynamic and scalable web scraping tools using Selenium and Flask. It streamlines the process with automated browser interaction, a simple REST API for scraper management, and robust logging and environment configuration.

Key Features:

• Selenium Automation: Efficiently interact with web pages that require JavaScript rendering.
• Flask Backend: Handle scraper operations with a REST API, including authentication and logging.
• Bearer Authentication: Secure API access using bearer tokens.
• Environment Management: Easily switch between environments using .env configuration.
• Docker Support: Seamless containerization for consistent development and deployment.
• Logging System: Track scraper operations and errors for better debugging.

Local Setup:

Ensure the following before getting started:

• Python 3.x
• .env file for environment variables (use .env.example as a template)
• HTTP client (e.g., Postman) to interact with the API
• Chrome browser for Selenium

Run the Project:

1. Set up a virtual environment.
2. Install dependencies with pip install -r requirements.txt.
3. Start the server with python main.py.
4. Access the server at http://localhost3000.

Workflow:

1. Set up your .env file.
2. Configure the base URL in utils/config.py.
3. Add your own API endpoints to main.py.
4. Build scraper logic as reusable actions within the actions/ folder.

Project Structure:

├── main.py # Flask entry point
├── actions # Scraper logic
├── controller # API request handlers
├── temp_downloads # Temp files during scraping
└── utils # Helper functions

Known Errors and Solutions:

1. Permission Errors with Chromedriver: Ensure proper permissions for the chromedriver or that it's correctly installed.
2. Driver Issues: Verify that the script uses the right path for the chromedriver.

Bibliography:

• Selenium Documentation
• Flask Documentation