Amazon Product Scraper
This is a Python script that uses Selenium and BeautifulSoup to scrape product data from Amazon.in. It is designed to search for a specific term (e.g., "laptop"), navigate through the search result pages, and extract key information about each product.
Disclaimer: Web scraping Amazon is against their Terms of Service. This project is for educational purposes only. Use it at your own risk. Amazon has strong anti-scraping measures (like CAPTCHAs), which this script attempts to work around with manual user assistance.
Features
Scrapes product data for a given search term.
Navigates through multiple search result pages.
Manual CAPTCHA Solving: Includes a pause to allow the user to manually solve the CAPTCHA and let the script continue.
Extracts the following data points for each product:
Product Title
Price (in ₹)
Rating (e.g., "4.5 out of 5 stars")
Image URL
Ad / Organic (Identifies if the listing is a "Sponsored" ad)
Saves all collected data to a timestamped CSV file in an output/ directory.
File Structure
amazon_scraper/ ├── .venv/ # Virtual environment directory ├── output/ # Folder for all scraped CSV files ├── chromedriver.exe # (Required) Selenium WebDriver for Chrome ├── scraper.py # The main Python scraper script └── requirements.txt # Python dependencies
Setup & Installation
- Prerequisites
Python 3.8 or newer.
Google Chrome browser (the script is built for Chrome).
- Clone the Repository
git clone https://github.com/your-username/amazon-scraper.git cd amazon-scraper
- Set Up a Virtual Environment
It is highly recommended to use a virtual environment.
python -m venv .venv ..venv\Scripts\activate
python3 -m venv .venv source .venv/bin/activate
- Install Dependencies
Install all required Python libraries from requirements.txt.
pip install -r requirements.txt
(If you don't have a requirements.txt file, run pip install selenium pandas beautifulsoup4)
- Download ChromeDriver (Crucial Step)
This script will not work without the correct chromedriver.
Check your Chrome Version: Open Chrome, go to ... > Help > About Google Chrome and note your version (e.g., 142.0.7288.0).
Download Driver: Go to the ChromeDriver download page.
Match the Version: Download the chromedriver-win64.zip (or your OS) that exactly matches your Chrome browser version.
Place the File: Unzip the file and place chromedriver.exe in the root of the project folder, next to scraper.py.
How to Run
Make sure your virtual environment is activated.
Run the script from your terminal:
python scraper.py
ACTION REQUIRED: The script will open an automated Chrome window and then pause. Your terminal will show:
================================================== ACTION REQUIRED: The browser has been opened. Please solve any CAPTCHA you see in the automated browser. Once the laptop search results are visible... ...press the 'Enter' key here in the terminal to continue.
Switch to the automated Chrome window, solve the "I'm not a robot" CAPTCHA, and wait for the "laptop" search results to load.
Once you see the products, switch back to your terminal and press Enter.
The script will now take over, scrape the data, and proceed to the next pages.
Output
The script will create an output/ folder and save the results in a file named with the search term and a timestamp.
Example: output/laptop_data_2025-11-16_15-30-05.csv
Libraries Used
Selenium: For automating the Chrome browser to handle dynamic content and CAPTCHAs.
BeautifulSoup4: For parsing the page's HTML and extracting data.
Pandas: For organizing the data and saving it to a CSV file.