- Start the Python venv:
pip3 install -r requirements.txt
source .venv/bin/activate
- Run a spider:
scrapy crawl XXXXX
Production use of these scraping tools can be found on my personal website.
I generally update the CSVs once every couple of weeks.
A web scraper designed to shop at uniqlo.
After activating the virtual environment, navigate to /uni_scraper/, and run:
bash scrape.sh
to scrape all the data from UNIQLO's website. The result is stored in uniqlo_all_items.csv.
The data includes:
- Category
- Composition
- Product Description (cleaned)
- Gender
- Name
- Price (in CAD)
- Product ID
- Rating (average)
- Size chart (specific to the item)
- Washing info
Filter these results through:
python3 filter_data.py [options]
type python3 filter_data.py --help for more information. The results are shown in an appropriately named .csv in the same directory.
To efficiently find the item of interest, search the product ID in Google.
14/06/2024 889 items scraped 327 duplicates across men/women
Drop the result CSV from a scrape into /data, or use the one that's already there.
Run the web server:
python3 -m http.server
Navigate to http://localhost:8000/