MADlib Benchmark Requirements

Intro

In order to well understand and be able to improve the performance of MADlib modules we need a proper benchmark framework. The purpose of this document is to lay out requirements for such a solution.

The main goals for MADlib benchmarks are:

• Competitive comparison

  Our initial comparison targets should be R and Mahout, as it should be easy to setup corresponding tests on the same HW configuration. Initial testing could be performed on a single host.

• Regression tests

  We should keep a log book of current run-times and rerun the appropriate tests after any substantial modifications to existing modules. The summary of it should be available on the wiki.

• Profiling & optimization

  MADlib benchmarks could potentially be very useful during ad-hoc profiling and optimization. Although it does not need to reinvent the performance metrics collection, as there are already tools for that.

Competitive Comparison

The following algorithms should be tested. Reference links to R and Mahout libraries are given below:

- Naive-Bayes Classification: [[R|http://finzi.psych.upenn.edu/R/library/e1071/html/naiveBayes.html]], [[Mahout|https://cwiki.apache.org/MAHOUT/bayesian.html]]
- Linear Regression: [[R|http://finzi.psych.upenn.edu/R/library/fRegression/html/regFit.html]], [[Mahout|https://cwiki.apache.org/MAHOUT/locally-weighted-linear-regression.html]]
- Logistic Regression: [[R|http://finzi.psych.upenn.edu/R/library/rms/html/lrm.html]], [[Mahout|https://cwiki.apache.org/MAHOUT/logistic-regression.html]]
- Decision Trees: [[R|http://finzi.psych.upenn.edu/R/library/RWeka/html/Weka_classifier_trees.html]], [[Mahout|https://cwiki.apache.org/MAHOUT/random-forests.html]]
- Support Vector Machines: [[R|http://finzi.psych.upenn.edu/R/library/kernlab/html/ksvm.html]], [[Mahout|https://cwiki.apache.org/MAHOUT/support-vector-machines.html]]
- Association Rules: [[R|http://finzi.psych.upenn.edu/R/library/arules/html/00Index.html]], [[Mahout|https://cwiki.apache.org/MAHOUT/parallel-frequent-pattern-mining.html]]
- k-means Clustering: [[R|http://finzi.psych.upenn.edu/R/library/stats/html/kmeans.html]], [[Mahout|https://cwiki.apache.org/MAHOUT/k-means-clustering.html]]
- SVD Matrix Factorisation: [[R|http://finzi.psych.upenn.edu/R/library/corpcor/html/fast.svd.html]], [[Mahout|https://cwiki.apache.org/MAHOUT/dimensional-reduction.html]]
- Latent Dirichlet Allocation: [[R|http://finzi.psych.upenn.edu/R/library/topicmodels/html/lda.html]], [[Mahout|https://cwiki.apache.org/MAHOUT/latent-dirichlet-allocation.html]]

General Requirements

In order to fulfill the above goals the MADlib benchmark tool should possess the following characteristics:

• Portability

  It must run on all OS and DB platforms supported by MADlib.

• Scalability

  It should be easy to scale the data size for a predefined test. This should apply to both the number of variables (table width) and the number of rows (table length).

• Repetitiveness

  It should be possible to rerun the performance test with identical starting conditions. This applies to both the data generation as well as the execution phase.

• Modularity

  It should be easy to add new performance tests for new or existing modules.

• Automation

  Execution of the benchmark (full or per module) should be easy to automate. Ideally this utility should be controlled from a single executable.

Data Generation

Because of the repetitive

#
# Table definition
#
- table: 
      name   : my_table     # table name
      distby : col_1        # column name for distributed by clause
    #  
    # Column definition
    #
    - colname   : col1
      datatype  : text|integer|float|boolean
      values    : 
            type : primary_key|random_list|random_number|foreign_key
            #
            # Types explained:
            #
            #   primary_key   - primary key, unique set of integers generated
            #                   using a sequence. No additional config needed.
            #   random_list   - list of values with frequency weights assigned.
            #                   Each value is drawn with probability of:
            #                   P(value_x) = weight_x / sum(all_weights)
            #   random_number - random number from a selected prob. distribution
            #                   with specified seed and distribution parameters.
            #   foreign_key   - randomly selected values from another table.column
            #
            # For random_list, random_number and foreign_key use seed = constant to make your
            # data set repeatable.
            #            
            list : 
                seed        : random|constant
                - value     : val_1  
                  weight    : 1
                - value     : val_2  
                  weight    : 3
                  ...
            random  : normal, chi-square, poisson, etc.
                parameters  : x,y,z,...
                seed        : random|constant
            fk      : some_other_table.some_column     
                seed        : random|constant
    ...  

API Requirements

For each MADlib module performance test one should be able to define (possibly in a common config file):

• Input Definition
• Run-time call arguments

Both data generation and test execution should be timed and all information should be saved in a database table located in MADlib schema, e.g.:

TABLE madlib.benchmark (
    module          TEXT
    (runtime_args)
    (input_defintion) 
    dgen_start      TIMESTAMP
    dgen_end        TIMESTAMP
    exec_start      TIMESTAMP
    exec_end        TIMESTAMP 
)        

Comments:

1. Because the benchmark tool requires connection to a database with MADlib extensions installed it is possible to think of it as an extension to madpack - the MADlib installer.

   madpack benchmark