Permalink
Fetching contributors…
Cannot retrieve contributors at this time
115 lines (101 sloc) 11.8 KB

Pandas to Blaze

This page maps pandas constructs to blaze constructs.

Imports and Construction

import numpy as np
import pandas as pd
from blaze import data, by, join, merge, concat

# construct a DataFrame
df = pd.DataFrame({
   'name': ['Alice', 'Bob', 'Joe', 'Bob'],
   'amount': [100, 200, 300, 400],
   'id': [1, 2, 3, 4],
})

# put the `df` DataFrame into a Blaze Data object
df = data(df)
Computation Pandas Blaze
Column Arithmetic
df.amount * 2
df.amount * 2
Multiple Columns
df[['id', 'amount']]
df[['id', 'amount']]
Selection
df[df.amount > 300]
df[df.amount > 300]
Group By
df.groupby('name').amount.mean()
df.groupby(['name', 'id']).amount.mean()
by(df.name, amount=df.amount.mean())
by(merge(df.name, df.id),
   amount=df.amount.mean())
Join
pd.merge(df, df2, on='name')
join(df, df2, 'name')
Map
df.amount.map(lambda x: x + 1)
df.amount.map(lambda x: x + 1,
              'int64')
Relabel Columns
df.rename(columns={'name': 'alias',
                   'amount': 'dollars'})
df.relabel(name='alias',
           amount='dollars')
Drop duplicates
df.drop_duplicates()
df.name.drop_duplicates()
df.distinct()
df.name.distinct()
   
Reductions
df.amount.mean()
df.amount.value_counts()
df.amount.mean()
df.amount.count_values()
Concatenate
pd.concat((df, df))
concat(df, df)
Column Type Information
df.dtypes
df.amount.dtype
df.dshape
df.amount.dshape

Blaze can simplify and make more readable some common IO tasks that one would want to do with pandas. These examples make use of the odo library. In many cases, blaze will able to handle datasets that can't fit into main memory, which is something that can't be easily done with pandas.

from odo import odo
Operation Pandas Blaze
Load directory of CSV files
df = pd.concat([pd.read_csv(filename)
                for filename in
                glob.glob('path/to/*.csv')])
df = data('path/to/*.csv')
Save result to CSV file
df[df.amount < 0].to_csv('output.csv')
odo(df[df.amount < 0],
    'output.csv')
Read from SQL database
df = pd.read_sql('select * from t', con='sqlite:///db.db')

df = pd.read_sql('select * from t',
                 con=sa.create_engine('sqlite:///db.db'))
df = data('sqlite://db.db::t')