Data for use with BrewDay
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
brewdata
scraper
.gitignore
CONTRIBUTING.md
LICENSE.txt
MANIFEST.in
Makefile
README.md
README.rst
make_pypi_readme.sh
requirements.txt
scrapy.cfg
setup.cfg
setup.py

README.md

BrewData

Data for use with BrewDay.

Usage

To use BrewData in another project you can do the following:

>>> import brewdata
>>> brewdata.where()
'/usr/local/lib/python2.7/site-packages/brewdata/'

It may be more useful to find the specific location of data:

>>> from brewdata import cereals as cereals_data
>>> cereals_data()
'/usr/local/lib/python2.7/site-packages/brewdata/cereals'

Format

The beer data is split into four sections: Cereals, Hops, Water, and Yeast. The data is provided in *.json files, one file per data item.

Scraper

Using the python project Scrapy the data is collected from publicly available websites where possible. Try to ensure that the source is always listed in any files scraped from websites. Permission should always be asked for to ensure there is no copyright problems.

Running the scraper

The scrapers for each data type are easy to run:

make scrape_cereals
...
make scrape_hops
...
make scrape_yeast
...

Shell commands

$ scrapy shell "http://www.hopslist.com/"

Sources

The data comes from several scraped websites. Additional data can be found in other listed sources. Scraped sources have "(scraped)" next to them.

Cereals

Hops

Yeast