Runtime - Program and run reproducible software experiments with the command line interface.
runtime -p PARAMETER_NAME parameter_value_1 .. parameter_value_n [-p PARAMETER2_NAME parameter2_value_1 .. parameter2_value_n] [-s extract_metric_script1 [-s extract_metric_script2 ...] [-m max_memory_usage (% total)] [ -t timeout value] [ -d (dryrun) ] -u using_expression -- command_line_template
Runtime is useful use to collect and format various statistics such as execution time or memory usage for large batches of program executions (runs). The collected statistics are stored as lines and columns into ascii files according to a format specified by the user. These files are meant to by read by humans or plotted with a tool such as gnuplot.
Runs are specified through a command line templated with parameter names (a.k.a. the command line template), and a list of parameter specifications which include for each parameter the name of the parameter and a list of possible values.
Given a command line template and a parameter list, Runtime produces a list of command lines to be executed by substituting each parameter name in the command line template by a one of its possible value.
By default, all the parameter values are combined to produce the complete list of command line to execute but the using expression can be used to control how the parameters values are combined together.
For each run at least the execution time and maximum memory usage are recorded and reported. More statistics can be added through metric extraction scripts.
Once all the command lines have been produced by Runtime, they are displayed and executed. The data collected during the execution as well as the output produced by the command line are stored into output files.
Finally a README file is created to record various meta data about the experimental setting. This includes the exact Runtime command that have been used to generate this particular experiment such that copying and pasting this command line will run the same experiment again. Other informations such as the host name and the date are also stored in the README file to improve reproducibility.
See the "Examples" section for concrete examples.
-p NAME value1 [value2 value3 ...] Declare a new parameter called NAME which can take the value value1 value2 value3 ... This options can be used multiple time as long as the declared parameters have different names. See "Parameters" for more details.
-s extraction_script_path Provide an extraction script path. Extraction script are fed with the output generated by each run and the output string is stored into an extra set of output files. Output files are named after the basename of the script. This option can be used multiple time as long as the basename of the script is different.
-m maximum_memory_usage Specify a memory limit for each run in % of the total memory available on the host machine. If the maximum is reached, the run is killed and the value ERR_MEM is reported for every metric.
-t timeout Specify a timeout in seconds. If the timeout is reached, the run is killed and the value ERR_TME is reported for every metrics.
-d Dry run: run as many tests as possible without running the experiments. Do not create any file.
-u using expression Specify the using expression. See "Using expression" for more details.
Each parameter is declared with the -p switch as follows. (Multiple parameters are declared with multiple -p switch.)
-p I<PARAMETER_NAME> value1 value2 value3 ...
PARAMETER_NAME is the parameter name in uppercase letters and is followed by a space separated list of all the parameters values. Possible examples:
-p COMPILER_NAME gcc clang
-p NUM_THREADS 1 2 3 4
Remark that this is also possible:
-p NUM_THREADS `seq 1 32`
The using expression serves multiple purposes, the most important ones are:
1. It describes how to combine the parameters values to build the command lines from the command line template. 2. It describes the format specification to write the results into files lines and columns with an adequate format.
The using expression is composed of parameters names, operators and format descriptors.
- Parameters names are the names (in capital letters) of the parameters formerly declared.
- Operators are 'x' or '='
'x' (carthesian product) combines all the values from the left operand with the values from the right operands
for example ('l' and 'c' are format descriptors, you can safely ignore them for now.):
runtime -p A a1 a2 -p B b1 b2 -u AlxBc -- echo A B C will program the execution of:
echo a1 b1 echo a1 b2 echo a2 b1 echo a2 b2 '=' maps all the values of the right operand to a value of the left operand with respect to the input value order.
runtime -p A a1 a2 -p B b1 b2 -u Al=Bc -- echo A B C
will program the execution of:
echo a1 b1 echo a2 b2
You can combine them, and use parenthesis:
runtime -p A a1 a2 -p B b1 b2 -p C c1 c2 -u "(Ac=Bl)xCf" -- echo A B C
Remark that in bash, you'll need to quote the using expressions so that the parenthesis are not captured by bash.
will program the execution of:
echo a1 b1 c1 echo a1 b1 c2 echo a2 b2 c1 echo a2 b2 c2
Note that if you introduce paranthesis, you must quote the using expression.
Format descriptor are associated with parameters to describe how the mesurements will be stored in the results files. They can be either 'f', 'l', or 'c'. - f stands for "one value per file" - l stands for "one value per line" - c stands for "one value per column"
runtime -p A a1 a2 -p B b1 b2 -p C c1 c2 -u AfxBcxCl -- echo A B C
Will create two files in the time output directory named: time_A.a1_B_C and time_A.a2_B_C
Each file contains times measurements laid out as follows:
# C B=b1 B=b2 c1 0.00 0.00 c2 0.00 0.00
i.e. One value per column for parameter B and one value per line for parameter C.
Skipping hopeless runs
If you know in advance that the execution time increase monotonically with the value of a parameter, you can tell runtime to skip the runs that have a higher value for this parameter. This can be specified with the '<' '>' parameter decoration.
E.g. the the expression:
runtime -t 10 -p A a1 a2 -p B b1 b2 -p C c1 c2 -u AfxBcxCl -- echo A B C
Will produce and run the following commands:
echo a1 b1 c1 echo a1 b1 c2 echo a1 b2 c1 echo a1 b2 c2 echo a2 b1 c1 echo a2 b1 c2 echo a2 b2 c1 echo a2 b2 c2
runtime -t 10 -p A a1 a2 -p B b1 b2 -p C c1 c2 -u "AfxBE<lt>cxCl" -- echo A B C
Remark that in bash you'll need to quote the using expression so that the '<lt>' symbol is not captured by bash.
Will not run echo a2 b2 c1 if echo a2 b1 c1 timed out (or went out of memory). When runs are skipped the value SKP is reported for all metrics.
Each execution of runtime creates a directory named after the current date. The directory contains - a time subdirectory, - a mem subdirectory, - a README file - a usr directory when a user script is provided.
Each subdirectory contains the reporting files except the output sub directory which contains the output of every execution.
Comparing compiler compilation times.
Let's say we want to compare the compilation time of various C compilers (e.g. gcc, clang) on different source code (e.g. quicksort.c, fibo.c).
runtime -p COMPILER gcc clang -p TEST_PROGRAM quicksort.c fibo.c -p OPTIMIZATION_LEVEL O1 O2 O3 Os -u TEST_PROGRAMfxCOMPILERcxOPTIMIZATION_LEVELl -- COMPILER -OPTIMIZATION_LEVEL TEST_PROGRAM
This declares three parameters called COMPILER TEST_PROGRAM and OPTIMIZATION_LEVEL. Each parameter can take any value in the list following the parameter name, for example COMPILER can take two values, either 'gcc' or 'clang'.
Running the previous Runtime command will produce and execute the following commands lines.
gcc -O1 quicksort.c gcc -O2 quicksort.c gcc -O3 quicksort.c gcc -Os quicksort.c clang -O1 quicksort.c clang -O2 quicksort.c clang -O3 quicksort.c clang -Os quicksort.c gcc -O1 fibo.c gcc -O2 fibo.c gcc -O3 fibo.c gcc -Os fibo.c clang -O1 fibo.c clang -O2 fibo.c clang -O3 fibo.c clang -Os fibo.c
... and create the following output file structure:
2014-05-11_152521/ |-- README |-- time/ | |-- time_TEST_PROGRAM.fibo.c_COMPILER_OPTIMIZATION_LEVEL | `-- time_TEST_PROGRAM.quicksort.c_COMPILER_OPTIMIZATION_LEVEL |-- mem/ | |-- mem_TEST_PROGRAM.fibo.c_COMPILER_OPTIMIZATION_LEVEL | `-- mem_TEST_PROGRAM.quicksort.c_COMPILER_OPTIMIZATION_LEVEL `-- output/ |-- TEST_PROGRAM.fibo.c_COMPILER.clang_OPTIMIZATION_LEVEL.O1.out |-- TEST_PROGRAM.fibo.c_COMPILER.clang_OPTIMIZATION_LEVEL.O2.out |-- TEST_PROGRAM.fibo.c_COMPILER.clang_OPTIMIZATION_LEVEL.O3.out |-- TEST_PROGRAM.fibo.c_COMPILER.clang_OPTIMIZATION_LEVEL.Os.out |-- TEST_PROGRAM.fibo.c_COMPILER.gcc_OPTIMIZATION_LEVEL.O1.out |-- TEST_PROGRAM.fibo.c_COMPILER.gcc_OPTIMIZATION_LEVEL.O2.out |-- TEST_PROGRAM.fibo.c_COMPILER.gcc_OPTIMIZATION_LEVEL.O3.out |-- TEST_PROGRAM.fibo.c_COMPILER.gcc_OPTIMIZATION_LEVEL.Os.out |-- TEST_PROGRAM.quicksort.c_COMPILER.clang_OPTIMIZATION_LEVEL.O1.out |-- TEST_PROGRAM.quicksort.c_COMPILER.clang_OPTIMIZATION_LEVEL.O2.out |-- TEST_PROGRAM.quicksort.c_COMPILER.clang_OPTIMIZATION_LEVEL.O3.out |-- TEST_PROGRAM.quicksort.c_COMPILER.clang_OPTIMIZATION_LEVEL.Os.out |-- TEST_PROGRAM.quicksort.c_COMPILER.gcc_OPTIMIZATION_LEVEL.O1.out |-- TEST_PROGRAM.quicksort.c_COMPILER.gcc_OPTIMIZATION_LEVEL.O2.out |-- TEST_PROGRAM.quicksort.c_COMPILER.gcc_OPTIMIZATION_LEVEL.O3.out `-- TEST_PROGRAM.quicksort.c_COMPILER.gcc_OPTIMIZATION_LEVEL.Os.out
- 2014-05-11_152521/ is the root directory for this experiment, named after the date in the YYYY-mm-dd_hhmmss format.
- README is the README file described earlier.
- time/ is the directory containing all the data points for execution time.
- time/time_TEST_PROGRAM.fibo.c_COMPILER_OPTIMIZATION_LEVEL one file. Remark that in the filename TEST_PROGRAM is the only parameter that is valued because TEST_PROGRAM file format is 'f' which stands for 'one value per file', other parameters such as compiler take different values in this file so the value is not reported in the filename.
- mem/ is the directory containing all the data points for maximum memory usage. Files are named following the same principle
- ouput/ is the directory containing the output (stdout and stderr) generated by the runs. There is one file per run so all the parameters are valued in the filename.
The content of the file time/time_TEST_PROGRAM.fibo.c_COMPILER_OPTIMIZATION_LEVEL is the following and should speak for itself.
# Experiment started on: 2014-05-11 15:25:22. # Machine hostname: neb. # Timout for each run -1 s. # Maximum memory usage allowed -0.0009765625 MiB. # # Reporting: Wall clock time (in seconds). # # OPTIMIZATION_LEVEL COMPILER=gcc COMPILER=clang O1 0.04 0.05 O2 0.05 0.05 O3 0.08 0.05 Os 0.05 0.05
See INSTALL file.
Benjamin Negrevergne, <email@example.com>
Copyright and license
Copyright (C) 2010-2013 by Benjamin Negrevergne
This library is free software; you can redistribute it and/or modify it under the terms of the GPLv3.