Beer Scraper

Project to scrape German beer vendors finding the best deals and cheapest options!

Development

• Poetry
• Scrapy
• Unix system required (WSL/Linux/MacOS)
  • Usage of playwright for dynamic websites is only supported in Unix systems

Vendor websites:

• Beertasting - Done
• Beerwulf - To Do
• Beyond Beer - Done
• Bierlinie - Done
• Biermarket - Done
• Bierothek - Done
• BierPost - In progress
• BierSelect - Done
• Craftbeer Shop - Done
• Flaschenpost - Semi Done
• Hier gibt's Bier - To Do
• Holy Craft - Done
• Meibier - Done
• Mein Biershop - To Do
• Mr Hop - To Do
• Ratsherrn - Done
• REWE - On Hold

Run scrapers:

cd $HOME/beer-scraper/
make scrape-all

or:

> cd $HOME/beer-scraper/
> poetry shell
> scrapy crawl "beyond_beer"
> scrapy crawl "bierothek" 
> scrapy crawl "craftbeer_shop"   

Testing around with playwright

# OPEN WSL SHELL!
cd $HOME/../mnt/c/Users/tomja/Desktop/Programming/beer-scraper

Useful Links:

• Burplist GitHub
• Scrapy BigQuery Pipeline
• Selfmade Pipeline
• Scraper Heroku Deployment