Skip to content
Fantastic Data and Where To Find Them: An introduction to APIs, RSS, and Scraping
HTML CSS JavaScript Jupyter Notebook Python
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
api updates May 12, 2017
img
reveal.js fix reveal typo May 10, 2017
rss updates May 12, 2017
scraping
.gitignore syntax highlighting in slides May 13, 2017
LICENSE slides and rss update May 10, 2017
README.md added environment infor to readme May 17, 2017
index.html slides udpated May 17, 2017
requirements.txt scotch update May 17, 2017
slides.ipynb slides udpated May 17, 2017

README.md

Fantastic Data and Where To Find Them: An introduction to APIs, RSS, and Scraping

This repository contains the notebooks, slides, and sample code for the tutorial given at PyCon on Wendesday, May 17, 2017.

Whether you’re building a custom web application, getting started in machine learning, or just want to try something new, everyone needs data. And while the web offers a seemingly boundless source for custom data sets, the collection of that data can present a whole host of obstacles. From ever-changing APIs to rate-limiting woes, from nightmarishly nested XML to convoluted DOM trees, working with APIs and web scraping are challenging but critically useful skills for application developers and data scientists alike. In this tutorial, we’ll introduce RESTful APIs, RSS feeds, and web scraping in order to see how different ingestion techniques impact application development. We’ll explore how and when to use Python libraries such as feedparser, requests, beautifulsoup, and urllib. And finally we will present common data collection problems and how to overcome them.

We’ll take a hands-on, directed exercise approach combined with short presentations to engage a range of different APIs (with and without authentication), explore examples of how and why you might web scrape, and learn the ethical and legal considerations for both. To prepare attendees to create their own data ingestion scripts, the tutorial will walk through a set of examples for robust and responsible data collection and ingestion. This tutorial will conclude with a case study of Baleen, an automated RSS ingestion service designed to construct a production-grade text corpus for NLP research and machine learning applications. Exercises will be presented both as Jupyter Notebooks and Python scripts.

Contents

There is a directory for each topic that contains the tutorial notebook and sample code. The slides were created with Jupyter notebook and reveal.js. They are hosted on GitHub Pages and can be viewed here.

Getting Started

Step one is to clone or download the repository, unzip it and then cd into the project directory.

Note, this code requires Python 3 or Anaconda to run, as well as several third party libraries. Make sure that you have Jupyter notebook installed, and if you're using Python that you have pip installed. One you do, you can install the required third party dependencies as follows:

$ pip install -r requirements.txt

or

$ conda install --yes --file requirements.txt

Environment Considerations

All code has been tested in Python 3.5. Some of the notebooks will run in 2.7, but not all. See the setup note below for a note on running the code in Python 3.6.

If you running Windows and do not have extensive expereince with the libraries in the requirements.txt file, I recommend installing Anaconda and creating a Python 3.5 environment.

Special Setup

Python 3.6 on OS X does not include a trust store, so SSL connections do not work out of the box. This can be fixed by running:

/Applications/Python\ 3.6/Install\ Certificates.command

This installs a default trust store.

References:

You can’t perform that action at this time.