Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Pythonic abstractions for data pipelines
Fetching latest commit...
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

datapypes - Pythonic abstractions for data pipelines

datapypes provides the building blocks to enable the following:

# Retrieve news from GoogleNews, store in MongoDB
# Create a post, send to Twitter & Tumblr
Post(text='I did something interesting').save(
  Twitter(username='tweet_user', password='tweet_pass'),
  Tumblr(username='tumblr_user', password='tumblr_pass'))
# Get SoundCloud, YouTube links for a track, post to Twitter, save to MongoDB
Track(artist="Meshuggah", title="New Millenium Cyanide Christ").get(
  SoundCloud(), Youtube()).save(
    Twitter(username='tweet_user', password='tweet_pass'),

See for implementation details.


Build your pipeline by subclassing these classes:

  • Attribute - A typed model attribute, i.e. title, author, date
  • Model - A data model, i.e. Article, Video
  • Set - A set of data models, i.e. News, Videos
  • Source - A data source, i.e. Google News, Twitter
  • Store - A data source & store, i.e. MongoDB, Twitter
  • SourcePype - Defines integration logic for Set & Source
  • StorePype - Defines integration logic for Set & Store


A Source defines these action methods:

  • one() - Returns a single data point
  • latest() - Returns a list of the most recent data
  • search() - Returns a list of data matching a query
  • stream() - Builds a list of data from a stream
  • all() - Returns a list of all available source data

A Store defines these action methods:

  • save() - Creates or updates data in a Store
  • update() - Updates data in a Store
  • delete() - Deletes data in a Store

Pype methods

A SourcePype defines these methods:

  • one() - Loads raw source data into a Model
  • latest() - Loads recent source data into a Set
  • search() - Loads found source data into a Set
  • stream() - Loads raw stream data into a Set
  • all() - Loads all source data into a Set

A StorePype defines these methods:

  • save_model() - Saves a single Model to a Store
  • save_set() - Saves an entire Set to a Store
  • update_model() - Updates a single Model in a Store
  • update_set() - Updates entire Set in a Store
  • delete_model() - Deletes a single Model from a Store
  • delete_set() - Deletes an entire Set from a Store


Mike Smathers <mikesmathers at gmail dot com>


All code is released under the MIT license. Please read the LICENSE.txt file for more details.

Something went wrong with that request. Please try again.