Big Data Frame on Spark
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.


DataFrames are useful constructs for data scientists, popularized by R and pandas.

Implementations of these packages are not natively distributed - their DataFrames struggle with big data. bigdf is a dataframe on top of Apache Spark. It is written as an internal Scala DSL with the look and feel of Pandas to make it easy for those familiar with Pandas to use it.

A gentle(but outdated) introduction is available at:

How to get it

  • Choose a repos directory (make one if it doesn't exist)
export $REPOS = ~/Documents/repos
  • Install scala and sbt (brew instructions below)
$ brew install scala
$ brew install sbt
  • clone the bigdf repo
$ cd $REPOS
$ git clone
  • sbt update, package, and test(optional) in the bigdf directory.
$ cd bigdf
$ sbt update
$ sbt test
$ sbt assembly
  • Start a shell and begin playing. You need spark 1.4.x
$ cd $REPOS
$ ./spark/bin/spark-shell --jars ./bigdf/target/scala-2.10/bigdf-assembly-*.jar 

Use with ipython notebook

$cd $REPOS
$git clone
$cd IScala
>++ 2.10.4
$ipython profile create scala

Add the following line to the three files ~/.ipython/profile_scala/ipython*

c.KernelManager.kernel_cmd = ["java", "-jar", "$ISCALA_PATH/lib/IScala.jar", "--connection-file", "{connection_file}", "--parent"]"

To start notebook

$cd <my notebooks directory>
$ipython notebook --profile scala

To install spark and bigdf to your local mvn repos

$brew install maven
$mvn install:install-file -Dfile=$REPOS/bigdf-transforms/lib/spark-assembly-1.2.0-SNAPSHOT-hadoop1.0.4.jar -DgroupId=org.apache -DartifactId=spark-assembly_2.10 -Dversion=1.2.0-SNAPSHOT-hadoop1.0.4 -Dpackaging=jar
$ mvn install:install-file -Dfile=$REPOS/bigdf-transforms/bigdf/target/scala-2.10/bigdf-assembly-0.2.jar -DgroupId=com.ayasdi -DartifactId=bigdf-assembly_2.10 -Dversion=0.2 -Dpackaging=jar

To start using bigdf in a notebook

%libraryDependencies += "org.apache" %% "spark-assembly" % "1.2.0-SNAPSHOT-hadoop1.0.4"
%libraryDependencies += "com.ayasdi" %% "bigdf-assembly" % "0.2"
%resolvers += "Local Maven Repository" at "file:///Users/mohit/.m2/repository"
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import com.ayasdi.bigdf._

val sc = new SparkContext("local[*]", "bigdf-intro")
val df = DF(sc, "/Users/mohit/Downloads/faredata2013/trip_fare_1.csv", ',', true)