MADlib Benchmark Requirements

Intro

In order to well understand and be able to improve the performance of MADlib modules we need a proper benchmark framework. The purpose of this document is to lay out requirements for such a solution.

The main goals for MADlib benchmarks are:

• Competitive comparison

  Our initial comparison targets should be R and Mahout, as it should be easy to setup corresponding tests on the same HW configuration. Initial testing could be performed on a single host.

• Regression tests

  We should keep a log book of current run-times and rerun the appropriate tests after any substantial modifications to existing modules. The summary of it should be available on the wiki.

• Profiling & optimization

  MADlib benchmarks could potentially be very useful during ad-hoc profiling and optimization. Although it does not need to reinvent the performance metrics collection, as there are already tools for that.

Competitive Comparison

The initial comparison tests can be performed on a single machine using the following plan:

• HW/OS platform: 64 bit Red Hat Enterprise Linux Server 5.5 with 16 CPU cores, 64GB RAM.
• Test environments:
  • MADlib on Greenplum 4.1
  • MADlib on PostgreSQL 9.0
  • Alpine Miner on Greenplum 4.1
  • R
  • Revolution
  • Hadoop/Mahout
• Algorithms:

  • Naive-Bayes Classification: R, Mahout

    Training data sets: 
       # of classes: 10, 100, ... 
       # of attributes: 10, 100, 1000, ... 
       # of rows: 1000, 10.000, 100.000, 1.000.000, 1.000.000.000
    Data to classify:   
       # of rows: 1000, 10.000, 100.000, 1.000.000, 1.000.000.000   
    Test 1: precompute class priors and feature probabilities, score the data.
    Test 2: score the data w/o pre-computation of class priors and feature probabilities.
    

  • Linear Regression: R, Mahout

    Data sets: 
       # of independent variables: 10, 100, 1.000, 10.000, ... 
       # of rows: 1000, 10.000, 100.000, 1.000.000, 1.000.000.000, ...
    Test: run the linear regression function.
    

  • Logistic Regression: R, Mahout

    Data sets: 
       # of variables: 10, 100, 1.000, ... 
       # of rows: 1000, 10.000, 100.000, 1.000.000, ...
    Test: run the logistic regression function.
    

  • Decision Trees: R, Mahout

  • Support Vector Machines: R, Mahout

  • Association Rules: R, Mahout

  • k-means Clustering: R, Mahout

  • SVD Matrix Factorisation: R, Mahout

  • Latent Dirichlet Allocation: R, Mahout

General Requirements

In order to fulfill the above goals the MADlib benchmark tool should possess the following characteristics:

• Portability

  It must run on all OS and DB platforms supported by MADlib. We can ensure this requirement by following the architecture of the MADlib installer (madpack), that is using db command-line utility to generate data sets as well as execute performance test calls.

• Scalability

  It should be easy to scale the data size for a predefined test. This should apply to both the number of variables (table width) and the number of rows (table length). Again, by using SQL to generate the data we can achieve a better scalability.

• Repetitiveness

  It should be possible to rerun the performance test with identical starting conditions. This applies to both the data generation as well as the execution phase.

• Modularity

  It should be easy to add new performance tests for new or existing modules.

• Automation

  Execution of the benchmark (full or per module) should be easy to automate. Ideally this utility should be controlled from a single executable.

Benchmark Configuration

In order to achieve scalable high performance data generation we should utilize (whenever possible) the parallel nature of the target database. This would suggest using SQL to prepare the test data according to desired specifications.

Definitions of test data and test runs should be stored in configuration file(s), e.g. under /madlib/src/perftest/<module>.yml

# 
# MADlib Performance test configuration file structure 
#

#
# Data definition:
#
DATA:
  - TABLE:          my_table     
    DISTRIBUTED_BY: column_name
    #  
    # Column definitions
    #
    COLUMNS:      
      - COLUMN:     column_name
        DATA_TYPE:  text | integer | float | boolean 
        VALUES:     primary_key | weighted_list | random_dist | foreign_key
        #
        # Type definition
        #
        #   primary_key   - primary key, unique set of integers generated
        #                   using a sequence. No additional config needed.
        #   weighted_list - list of values with frequency weights assigned.
        #                   Each value is drawn with probability of:
        #                   P(value_x) = weight_x / sum(all_weights)
        #   random_dist   - random number from a selected prob. distribution
        #                   with specified seed and distribution parameters.
        #   foreign_key   - randomly selected values from another table.column
        #
        # For random_list, random_number and foreign_key use SEED = constant 
        # to make your data set repeatable.
        #            
        WEIGHTED_LIST: 
           - VALUE:     val_1  
             WEIGHT:    1
           - VALUE:     val_2  
             WEIGHT:    3
        RANDOM_DIST:    
            FUNCTION:   normal | chi-square | poisson | etc.
            PARAMETERS: x,y,z,...
        FOREIGN_KEY:    some_other_table.some_column     
        SEED:           random | constant
        LIKE:           other_column    # of the same table

#  - TABLE:          ... 

# 
# Test run definition:
#
RUN:

  - TEST: 1
    # Standard SQL syntax
    SQL: "SELECT * FROM MADLIB_SCHEMA.lin_reg();"
    # Greenplum syntax
    GREENPLUM: "SELECT * FROM MADLIB_SCHEMA.lin_reg();"
    # Postgres syntax
    POSTGRES: "SELECT * FROM MADLIB_SCHEMA.lin_reg();"

#  - TEST: ...

Data Generation and Test Execution

Since performance benchmark requires connection to a database with MADlib extensions pre-installed it makes sense to use the MADlib installer (madpack) as the benchmark execution tool. This will

# madpack -p postgres -c <connection_string> benchmark <benchmark_arguments>

where benchmark_arguments are:

-r rowsize              number of rows to generate for each test 
-c module_prefix        prefix for configuration files from '/madlib/perftest/*.yml' 
                        to include in this benchmark (optional)

Benchmark Log

Each performance benchmark execution should be recorded for future reference.

Both data generation and test execution should be timed and all information should be saved in a database table located in MADlib schema, e.g.:

TABLE madlib.benchmark (
    name        TEXT
    config      TEXT
    rowsize     BIGINT
    dgen_start  TIMESTAMP
    dgen_end    TIMESTAMP
    run_names   TEXT[]      -- names/ids of all test runs 
    run_times   FLOAT[]     -- run lenght (in sec) for each test run
)