Data Wrangling With Python: NICAR 2017
This repository contains materials for "Data Wrangling With Python," a hands-on session presented at the 2017 Investigative Reporters and Editors NICAR conference in Jacksonville, Fla.
The code covers basic import, transformation, and export of data from CSV files. Output formats include JSON and SQL table creation statements.
What You'll Need
This code was written and taught on MacOS Sierra using Python 3.6; it should work fine on Python 2.7 as well as Python 3.5+. It's not been tested on Windows. I recommend setting up a virtual environment and then installing the following dependencies:
Here are the steps:
Setup for Jupyter Notebook Exercises
- Open your terminal and navigate to the Desktop:
- Clone this repository:
git clone https://github.com/anthonydb/data-wrangling-python-nicar-2017.git
- Change to the project directory you just cloned:
- Set up your virtual environment. We'll call it 'nicar':
- Install your dependencies:
pip install agate csvkit requests jupyter
- Now you can run Jupyter Notebook:
Jupyter Notebook will open in your default browser on
localhost:, and you will see a list of files in the directory. The most important file for this exercise is the Census 2010 data contained in
Next, using the "New" menu at the top right of the Jupyter Notebook app, create a new notebook matching the version of Python on your system.
Now, you can paste in the code found in the .ipynb file in this repository and run it to follow the tutorial.
When you're done, run
ctrl+cin the terminal to close Jupyter Notebook. Then
deactivateto exit your virtual environment.
Trying csvkit Commands
You can run the commands in the file csvkit-notes.md in your terminal. Make sure the virtual environment is activated.
Notes and Further Reading
This NICAR 2017 hands-on class was loosely inspired by the structure of a NICAR 2016 class taught by Jacqueline Kazil and Abraham Epton.
Suggested resources to learn more about Python and data: