A light weight, super fast, large scale machine learning library on spark .
Scala Java
Switch branches/tags
Nothing to show
Clone or download
takun2s Merge pull request #25 from zhanglistar/master
1. fix compile error on can't find symbols when using java and scala.
Latest commit 65cfedb Mar 21, 2018
Permalink
Failed to load latest commit information.
core 1. handle no found model file exception Mar 21, 2018
docs update docs Aug 2, 2017
img modify png Nov 29, 2016
ps 1. update version and add print weights Mar 18, 2018
spark 1. add load model interface. Mar 21, 2018
.gitignore init code Nov 14, 2016
LICENSE init code Nov 14, 2016
README.md Update README.md Sep 5, 2017
ReadMe.txt init code Nov 14, 2016
pom.xml 1. update version and add print weights Mar 18, 2018

README.md

Fregata: Machine Learning

GitHub license

  • Fregata is a light weight, super fast, large scale machine learning library based on Apache Spark, and it provides high-level APIs in Scala.

  • More accurate: For various problems, Fregata can achieve higher accuracy compared to MLLib.

  • Higher speed: For Generalized Linear Model, Fregata often converges in one data epoch. For a 1 billion X 1 billion data set, Fregata can train a Generalized Linear Model in 1 minute with memory caching or 10 minutes without it. Usually, Fregata is 10-100 times faster than MLLib.

  • Parameter Free: Fregata uses GSA SGD optimization, which dosen't require learning rate tuning, because we found a way to calculate appropriate learning rate in the training process. When confronted with super high-dimension problem, Fregata calculates remaining memory dynamically to determine the sparseness of the output, balancing accuracy and efficiency automatically. Both features enable Fregata to be treated as a standard module in data processing for different problems.

  • Lighter weight: Fregata just uses Spark's standard API, which allows it to be integrated into most business’ data processing flow on Spark quickly and seamlessly.

Architecture

This documentation is about Fregata version 0.1

  • core : mainly implements stand-alone algorithms based on GSA, including Classification Regression and Clustering
    • Classification: supports both binary and multiple classification
    • Regression: will release later
    • Clustering: will release later
  • spark : mainly implements large scale machine learning algorithms based on spark by wrapping core.jar and supplies the corresponding algorithms

Fregata supports spark 1.x and 2.x with scala 2.10 and scala 2.11 .

Algorithms

Installation

Two ways to get Fregata by Maven or SBT :

  • Maven's pom.xml
    <dependency>
       <groupId>com.talkingdata.fregata</groupId>
        <artifactId>core</artifactId>
        <version>0.0.3</version>
    </dependency>
    <dependency>
        <groupId>com.talkingdata.fregata</groupId>
        <artifactId>spark</artifactId>
        <version>0.0.3</version>
    </dependency>
  • SBT's build.sbt
    // if you deploy to local mvn repository please add
    // resolvers += Resolver.mavenLocal
    libraryDependencies += "com.talkingdata.fregata" % "core" % "0.0.3"
    libraryDependencies += "com.talkingdata.fregata" % "spark" % "0.0.3"

If you want to manual deploy to local maven repository , as follow :

git clone https://github.com/TalkingData/Fregata.git
cd Fregata
mvn clean package install

Quick Start

Suppose that you're familiar with Spark, the example below shows how to use Fregata's Logistic Regression, and experimental datas can be obtained on LIBSVM Data

  • adding Fregata into project by Maven or SBT referring to the Downloading part
  • importing packages
	import fregata.spark.data.LibSvmReader
	import fregata.spark.metrics.classification.{AreaUnderRoc, Accuracy}
	import fregata.spark.model.classification.LogisticRegression
	import org.apache.spark.{SparkConf, SparkContext}
  • loading training datas by Fregata's LibSvmReader API
    val (_, trainData)  = LibSvmReader.read(sc, trainPath, numFeatures.toInt)
    val (_, testData)  = LibSvmReader.read(sc, testPath, numFeatures.toInt)
  • building Logsitic Regression model by trainging datas
    val model = LogisticRegression.run(trainData)
  • predicting the scores of instances
    val pd = model.classPredict(testData)
  • evaluating the quality of predictions of the model by auc or other metrics
    val auc = AreaUnderRoc.of( pd.map{
      case ((x,l),(p,c)) =>
        p -> l
    })

Input Data Format

Fregata's training API needs RDD[(fregata.Vector, fregata.Num)], predicting API needs the same or RDD[fregata.Vector] without label

	import breeze.linalg.{Vector => BVector , SparseVector => BSparseVector , DenseVector => BDenseVector}
	import fregata.vector.{SparseVector => VSparseVector }

	package object fregata {
	  type Num = Double
	  type Vector = BVector[Num]
	  type SparseVector = BSparseVector[Num]
	  type SparseVector2 = VSparseVector[Num]
	  type DenseVector = BDenseVector[Num]
	  def zeros(n:Int) = BDenseVector.zeros[Num](n)
	  def norm(x:Vector) = breeze.linalg.norm(x,2.0)
	  def asNum(v:Double) : Num = v
	}
  • if the data format is LibSvm, then Fregata's LibSvmReader.read() API can be used directly
	// sc is Spark Context
	// path is the location of input datas on HDFS
	// numFeatures is the number of features for single instance
	// minPartitions is the minimum number of partitions for the returned RDD pointing the input datas
	read(sc:SparkContext, path:String, numFeatures:Int=-1, minPartition:Int=-1):(Int, RDD[(fregata.Vector, fregata.Num)])
  • else some constructions are needed

    • Using SparseVector
     	// indices is an 0-based Array and the index-th feature is not equal to zero
     	// values  is an Array storing the corresponding value of indices
     	// length  is the total features of each instance
     	// label   is the instance's label
    
     	// input datas with label
     	sc.textFile(input).map{
     		val indicies = ...
     		val values   = ...
     		val label    = ...
     		...
     		(new SparseVector(indices, values, length).asInstanceOf[Vector], asNum(label))
     	}
    
     	// input datas without label(just for predicting API)
     	sc.textFile(input).map{
     		val indicies = ...
     		val values   = ...
     		...
     		new SparseVector(indices, values, length).asInstanceOf[Vector]
     	}
    • Using DenseVector
     	// datas is the value of each feature
     	// label   is the instance's label
    
     	// input datas with label
     	sc.textFile(input).map{
     		val datas = ...
     		val label = ...
     		...
     		(new DenseVector(datas).asInstanceOf[Vector], asNum(label))
     	}
    
     	// input datas without label(just for predicting API)
     	sc.textFile(input).map{
     		val datas = ...
     		...
     		new DenseVector(indices, values, length).asInstanceOf[Vector]
     	}

MailList:

Contributors:

Contributed by TalkingData .