MADlib Benchmark Requirements
In order to well understand and be able to improve the performance of MADlib modules we need a proper benchmark framework. The purpose of this document is to lay out requirements for such a solution.
The main goals for MADlib benchmarks are:
-
Competitive comparison (Priority 1)
Our initial comparison targets should be R and Mahout, as it should be easy to setup corresponding tests on the same HW configuration. Initial testing could be performed on a single host.
-
Regression tests and other purposes (Priority 2)
We should keep a log book of current run-times and rerun the appropriate tests after any substantial modifications to existing modules. The summary of it should be available on the wiki.
In order to fulfill the above goals the MADlib benchmark tool should possess the following characteristics:
-
Portability
It must run on all OS and DB platforms supported by MADlib. We can ensure this requirement by following the architecture of the MADlib installer (madpack), that is using db command-line utility to generate data sets as well as execute performance test calls.
-
Scalability
It should be easy to scale the data size for a predefined test. This should apply to both the number of variables (table width) and the number of rows (table length). Again, by using SQL to generate the data we can achieve a better scalability.
-
Reproducibility
It should be possible to rerun the performance test with identical starting conditions. This applies to both the data generation as well as the execution phase.
-
Modularity
It should be easy to add new performance tests for new or existing modules.
-
Automation
Execution of the benchmark (full or per module) should be easy to automate. Ideally this utility should be controlled from a single executable.
The initial comparison tests can be performed on a single machine using the following plan:
- HW/OS platform: 64 bit Red Hat Enterprise Linux Server 5.5 with 16 CPU cores, 64GB RAM.
-
Test environments:
- MADlib on Greenplum 4.1
- MADlib on PostgreSQL 9.0
- Alpine Miner on Greenplum 4.1
- R
- Revolution
- Hadoop/Mahout
-
Algorithms:
-
Naive-Bayes Classification: R, Mahout
Scaling factor: number of classes, attributes, rows.Training set size factors: nr of classes nr of attributes nr of rows Test 1: precompute class priors and feature probabilities, then score the data. Test 2: score the data w/o pre-computation of class priors and feature probabilities. -
Data set size factors: nr of independent variables nr of rows Test: run the linear regression function. -
Logistic Regression: R, Mahout
Data set size factors: nr of variables nr of rows Test: run the logistic regression function. -
Training set size factors: nr of clases nr of features nr of rows Test: run the decision tree training and score the new data. -
Support Vector Machines: R, Mahout
Training set size factors: nr of classes/labels nr of features/dimensions nr of rows/points Test: run the SVM training and data scoring for each of the following kernel functions: (1) regression, (2) classification, (3) novelty detection -
Data set size factors: nr of unique items nr of transactions max number of items per transaction Test: run the association rules function. -
Data set size factors: nr of points/rows nr of dimensions density/sparsity of data Test: run the k-means clustering function. -
SVD Matrix Factorisation: R, Mahout
Data set size factors: matrix dimensions (rows, columns) density/sparsity of data Test: run the matrix factorisation function. -
Latent Dirichlet Allocation: R, Mahout
Data set size factors: nr of documents nr of words per document size of the dictionary (nr of unique words) Test: run the LDA function.
-
The benchmark tool shall:
- Whenever possible, generate relations in parallel.
The following class diagram shows the high-level design. Classes with a shaded background need to be implemented for each analytics tool and/or module. Classes with a plain background either lie outside the scope of the test framework and will not have to be implemented, or they are generic for all analytics tools and modules.
For each analytics tool:
- A test controller is implemented that drives data generation, data preprocessing/loading, and running the tests.
For each module:
- An arbitrary number of data generators exist, each of which is implemented as one or more relation ("table") generators.
- One or more loggers are implemented to process run-time statistics (usually, this means writing to persistant storage).
For each pair of analytics tool and module:
- A data loader is implemented that takes the output of each relation generator and converts it into the native format of the analytics tool. In PostgreSQL/Greenplum this would be loading into a table, whereas for other statistics software this would be writing out the data in the appropriate input file format (which might be a binary format).
- A test runner is implemented that drives the test. It collects run-time statistics and sends them to the logger.
In order to achieve scalable high performance data generation we should utilize (whenever possible) the parallel nature of the target database. This would suggest using SQL to prepare the test data according to desired specifications.
Definitions of test data and test runs should be stored in configuration file(s),
e.g. under /madlib/src/perftest/<module>.yml
#
# MADlib Performance test configuration file structure
#
#
# Data definition:
#
DATA:
- TABLE: my_table
DISTRIBUTED_BY: column_name
#
# Column definitions
#
COLUMNS:
- COLUMN: column_name
DATA_TYPE: text | integer | float | boolean
VALUES: primary_key | weighted_list | random_dist | foreign_key
#
# Type definition
#
# primary_key - primary key, unique set of integers generated
# using a sequence. No additional config needed.
# weighted_list - list of values with frequency weights assigned.
# Each value is drawn with probability of:
# P(value_x) = weight_x / sum(all_weights)
# random_dist - random number from a selected prob. distribution
# with specified seed and distribution parameters.
# foreign_key - randomly selected values from another table.column
#
# For random_list, random_number and foreign_key use SEED = constant
# to make your data set repeatable.
#
WEIGHTED_LIST:
- VALUE: val_1
WEIGHT: 1
- VALUE: val_2
WEIGHT: 3
RANDOM_DIST:
FUNCTION: normal | chi-square | poisson | etc.
PARAMETERS: x,y,z,...
FOREIGN_KEY: some_other_table.some_column
SEED: random | constant
LIKE: other_column # of the same table
# - TABLE: ...
#
# Test run definition:
#
RUN:
- TEST: 1
# Standard SQL syntax
SQL: "SELECT * FROM MADLIB_SCHEMA.lin_reg();"
# Greenplum syntax
GREENPLUM: "SELECT * FROM MADLIB_SCHEMA.lin_reg();"
# Postgres syntax
POSTGRES: "SELECT * FROM MADLIB_SCHEMA.lin_reg();"
# - TEST: ...Since performance benchmark requires connection to a database with MADlib extensions pre-installed it makes sense to use the MADlib installer (madpack) as the benchmark execution tool. This will
# madpack -p postgres -c <connection_string> benchmark <benchmark_arguments>
where benchmark_arguments are:
-r rowsize number of rows to generate for each test
-c module_prefix prefix for configuration files from '/madlib/perftest/*.yml'
to include in this benchmark (optional)
Each performance benchmark execution should be recorded for future reference.
Both data generation and test execution should be timed and all information should be saved in a database table located in MADlib schema, e.g.:
TABLE madlib.benchmark (
name TEXT
config TEXT
rowsize BIGINT
dgen_start TIMESTAMP
dgen_end TIMESTAMP
run_names TEXT[] -- names/ids of all test runs
run_times FLOAT[] -- run lenght (in sec) for each test run
)