Benchy

Simple benching tool written in zig. The first goal of this project is to learn a bit more about zig. The application runs commands given in a configuration file multiple times and gives metrics on the execution time.

It is thought to compare different version of a given program.

Installation

$ zig build -Doptimize=ReleaseSafe

The executable is located in zig-out/bin/

Usage

The basic usage is

benchy [path/to/config/file]

List of all the available options

benchy --help
    -h, --help
            Display this help and exit

        --no-csv
            Don't write a csv file of the results

 
        --no-script
            Don't write a gnuplot script template (automatically selected if no csv is requested)       
        
        --no-stdout
            Don't print the results on the standard output

        --cmd-output
            Print measured program standard output

    -o, --csv-filename <str>
            Name to give to the output csv

The program reads a YAML configuration file.

Here is all possible options:

nb_runs: 2                          # Number of runs per program
warmup: 2                           # Number of warm_ups to do
names: [ test1, my_super_test ]     # names of the tests
argvs: [ "./a.out 1", "./my_test" ] # Commands to run

By default, benchy reads a benchy.yml file that must be located in the current working directory when invoking the program. An alternative configuration file can be specified as an argument.

The program outputs the results to the standard output. It also generates a csv file in ./benchy-output/. The file is timestamped by default. By default, a simple gnuplot script is generated to plot the csv.

TO DO

Features

• Take a proper configuration file as input (yaml, JSON, zon ?)
• Take the number of time the programs will be launch
• Return the data (csv)
• Possibility to give a path to the config file we want
• Possibility to do warm-up runs
• Generate a script to plot the results
• Pretty prints during measure
• Options
  • Change the csv name
  • Don't generate csv or stdout
  • Don't generate script
  • Choose script format ? (gnuplot? python? ...?)
  • Change script, plot name
  • Display help
  • Throw the stdout of the measured programs
• Add other metrics
  • memory usage
  • binary size

Debug

• Enhance (memory management, idiomatic zig, builtins... )
• Do not print YAML parsing debug info
• Script named after the csv if given with option
• Add tests
• Fix size retrieval on non-local file (which and absolute path ?)
• Make the timestamp human-readable (DD-MM-YYYY-hh-mm-ss)

Ideas

Configuration file

We use zig-yaml to parse the input yaml file.

Zon

This solution needs us to find a deserializer or make one

The zon will look like the input structure:

Option 1

.{
    .name = "name of the bench",
    .nb_run = number_of_runs,
    .names = .{ "name of", "the programs" },
    .argvs = .{ 
        .{"./prog1"}, 
        .{"./prog2", "arg1", "arg2"}
    },
}

Option 2

.{
    .name = "name of the bench",
    .nb_run = number_of_runs,
    .tests = .{
        .{
            .name = "name of",
            .argv = .{"./prog1"},
        },
        .{
            .name = "the program",
            .argv = .{ "./prog1", "arg1", "arg2" },
        },
    },
}

We may use eggzon or find the standard one or write our own.

JSON

Call directly gnuplot

We can use gnuzplot to avoid using an intermediate script.

Alternatives

You may be interested in more advanced projects such as

• hyperfine
• bench