Home

Welcome to the hivemall wiki!

General tips

• Installation
• Install Hivemall UDFs as permanent functions
• Adding rowid for each row
• Hadoop tuning for Hivemall
• Compressing a large training table
• Run Hivemall on Amazon Elastic MapReduce
• Run Hivemall with Pig

Troubleshooting

• OutOfMemoryError in training
• SemanticException Generate Map Join Task Error: Cannot serialize object
• Asterisk argument for UDTF does not work
• The number of mappers is less than input splits in Hadoop 2.x

Advanced topics

• Ensemble learning for stable prediction
• Iterative training using distributed cache (#1 Logistic Regression on a9a dataset) (#2 Classification using Confidence Weight on news20 binary/multiclass dataset)

Particularly Important Tips

• Using explicit addBias() for a better prediction
• Use rand_amplify() to get better prediction results
• Mixing models for a better prediction convergence (MIX server)

Feature Engineering

• Feature Scaling
• Feature Hashing
• TF-IDF calculation

Evaluation

1. Statistical evaluation of a prediction model

Dataset generation

1. classification/logistic regression

Binary Classification

a9a binary classification

• Data preparation

1. Logistic Regression
2. Iterative training using distributed cache

news20 binary classification

• Data preparation

1. Perceptron, Passive Aggressive
2. CW, AROW, SCW
3. AdaGradRDA, AdaGrad, AdaDelta

KDD2010a/b binary classification

• Data preparation for KDD10a

1. PA/CW/AROW/SCW

• Data preparation for KDD10b

1. AROW

Webspam binary classification

• Data pareparation

1. PA1,AROW,SCW

Multiclass Classification

news20 multiclass classification

• Data preparation
• Data preparation for one vs the rest classifiers

1. PA
2. CW, AROW, SCW
3. Ensemble learning
4. one vs the rest classifier

Regression

E2006 tfidf regression

• Data preparation

1. Passive Aggressive, AROW

KDDCup 2012 track 2 CTR prediction

• Data preparation

1. Logistic Regression, Passive Aggressive
2. Logistic Regression with Amplifier
3. AdaGrad, AdaDelta

Recommendation

News20 multiclass related article recommendation

• Data preparation

1. LSH/Minhash

MovieLens movie recommendation

• Data preparation

1. Matrix Factorization
2. 10-fold Cross Validation (Matrix Factorization)

Nearest Neighbor

News20 multiclass similar article search

• Data preparation

1. LSH/Minhash and Brute-Force Search
2. kNN search using b-Bits Minhash