gi_scraper

Scrapes product catalogs from a list of Shopify storefronts into three normalized CSVs.

Every Shopify store exposes its catalog at /products.json. This walks that feed for each brand in the list and lands flat, joinable tables:

Table	Grain	Key
{site}_products.csv	one row per product	product_id
{site}_variants.csv	one row per size/colorway	variant_id, FK product_id
{site}_images.csv	one row per image	image_id, FK product_id

Every row carries site_name, so the per-brand files concatenate into one multi-tenant table without collisions.

Design notes

• Typed rows, not dicts. Each record is built as a dataclass (ShopifyProduct, ShopifyVariant, ShopifyImage), so a field the API stops returning fails loudly at construction instead of silently writing a blank column. Column order lives in shopify_columns.py and is handed to csv.DictWriter, which keeps the header stable.
• Pagination is length-driven. Shopify caps /products.json at 250 records per page, so a short page means the last one. Page number is a query parameter built fresh each request rather than appended to the previous URL — otherwise page three asks for &page=2&page=3 and the store decides which one it honors.
• TLS fingerprinting. Plain requests gets blocked by some of these storefronts, so requests go through curl_cffi impersonating Chrome. This is the same public endpoint a browser hits; the impersonation is about matching a normal client handshake, not about bypassing auth.
• Politeness. Four seconds between page requests, and only one pass per brand.

Usage

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python shopify_scraper.py      # writes output/{site}_{table}.csv

output/ is gitignored — the scraped catalogs are other people's commercial data, so this repo ships the scraper, not the harvest.