Using custom report formats with embedded tests

[kimh]

Is it possible to report with json format with embedded tests such as cpu ? I found that I can use json reporter with sysbench.hooks.report_cumulative = sysbench.report_json in a custom lua script but then I have to define my own event function.

Is it possible to only change a formatter with embedded tests?

[akopytov]

No, it's not currently possible. For the future major release, I'm planning to convert all embedded benchmarks to Lua to allow all benchmarks use custom reports and other features, but at the same time have the event loop itself implemented in C to avoid LuaJIT overhead for those benchmarks where it's critical, like the CPU one.

I'm going to keep this issue open as a feature request.

[ghost]

@kimh came here with exactly the same question. Did you end up implementing your own event function implementing the cpu test? Or did you manage to use the FFI to call the C code from src/tests/cpu/sb_cpu.c?

[akopytov]

@TiagoCH unfortunately most functions in sb_cpu.c are static, so using FFI to call them would not work without modifications to C code.

It would probably be not too hard to reimplement cpu_execute_event() as an event() function in a Lua script.

Actually, just having an empty event() function would also be a nice CPU benchmark, albeit a different one. In which case, it would also be possible to use report hooks.

[ghost]

Thanks for the tip, I'll try that the empty event function together with CSV output. I love the ease of use of sysbench, but manually scraping the data from the output into something I can use in Excel is a major PITA.

[ghost]

I create the following empty_test.lua using an empty event function:

#!/usr/bin/env sysbench

sysbench.cmdline.options = {
    ["cpu-max-prime"] = {"CPU maximum prime", 10000},
    ["threads"] = {"Number of threads", 2},
    ["histogram"] = {"Show histogram", "on"},
    ["report-interval"] = {"Report interval", 1}
}

function event()
end

function sysbench.hooks.report_intermediate(stat)
    local seconds = stat.time_interval
    print(string.format("%.0f;%u;%4.2f",
        stat.time_total, 
        stat.threads_running,
        stat.events / seconds))
end

I can then run it like this:

./empty-test.lua --cpu-max-prime=20000 --threads=8 --histogram --report-interval=1 run

[ghost]

Unfortunately, if I try to run it with the default arguments specified in sysbench.cmdline.options, the default values are ignored. What am I doing wrong?

[ghost]

By the way, this is my attempt to convert the prime test from cpu_execute_test() into lua:

#!/usr/bin/env sysbench

sysbench.cmdline.options = {
    ["cpu-max-prime"] = {"CPU maximum prime", 10000},
    ["threads"] = {"Number of threads", 2},
    ["histogram"] = {"Show histogram", "on"},
    ["report-interval"] = {"Report interval", 1}
}

function event()
    n = 0
    for c = 3, 9999 do
        t = math.sqrt(c)
        l = 2
        for l = 2, t do
            if c % l == 0 then
                break
            end
        end
        if l > t then
            n = n + 1
        end
    end
end

function sysbench.hooks.report_intermediate(stat)
    local seconds = stat.time_interval
    print(string.format("%.0f;%u;%4.2f",
        stat.time_total, 
        stat.threads_running,
        stat.events / seconds))
end

[akopytov]

Unfortunately, if I try to run it with the default arguments specified in sysbench.cmdline.options, the default values are ignored. What am I doing wrong?

That is currently not possible, sorry. I created #151 to fix it.

[akopytov]

By the way, this is my attempt to convert the prime test from cpu_execute_test() into lua:

Looks good! sysbench.opt.cpu_max_prime might be used instead of a hard-coded limit.

[ghost]

Great, I was wondering what the syntax for accessing the arguments was - this is my very first hour doing lua :-)

In the meantime I tried comparing the sysbench cpu test with this first lua implementation, and was surprised by the results. The report shows two times as many events for the lua version as for the C version. Since I don't think lua can be two times faster than C, there must be a catch somewhere. Any ideas?

Here's the C test:

$ sysbench cpu --cpu-max-prime=20000 --threads=8 run
sysbench 1.0.7 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 8
Initializing random number generator from current time


Prime numbers limit: 20000

Initializing worker threads...

Threads started!


General statistics:
    total time:                          10.0033s
    total number of events:              23909

Latency (ms):
         min:                                  3.33
         avg:                                  3.35
         max:                                 28.53
         95th percentile:                      3.36
         sum:                              80002.27

Threads fairness:
    events (avg/stddev):           2988.6250/2.69
    execution time (avg/stddev):   10.0003/0.00

And here is the lua test:

$ ./prime-test.lua --cpu-max-prime=20000 --threads=8 run
sysbench 1.0.7 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 8
Initializing random number generator from current time


Initializing worker threads...

Threads started!


General statistics:
    total time:                          10.0015s
    total number of events:              44917

Latency (ms):
         min:                                  1.77
         avg:                                  1.78
         max:                                  5.10
         95th percentile:                      1.79
         sum:                              79968.59

Threads fairness:
    events (avg/stddev):           5614.6250/1.58
    execution time (avg/stddev):   9.9961/0.00

(EDIT: I was ignoring the sysbench.opt.cpu_max_prime argument, as you noted. Once I fixed that, the lua version is just two times faster than the C version. Still surprising.)

[akopytov]

OK, 3 things:

1. I spotted a minor error in the script. Loop variables are always local to the loop body in Lua. So in this snippet:

        l = 2
        for l = 2, t do
            if c % l == 0 then
                break
            end
        end
        if l > t then
            n = n + 1
        end

The if statement after the loop will always see 2 as the value of l. Because the preceding for loop creates its own variable named l, whose value is discarded after leaving the loop.

But that is a minor thing, it verified it does not affect performance.

2. I think there may be multiple reasons for the 2x performance difference between LuaJIT and C :

• embedded benchmarks and Lua scripts use different implementations of the event generation loop. This is mostly for historic reasons, and I'm planning to fix it in the next major version

• the event code itself is not the same. In Lua all numbers have the floating point type. In C, we use 64-bit integers, but there is also floating point calculations involved because of sqrt(). So there is some conversion between integers and floating point numbers and vice versa.

3. LuaJIT can indeed be faster than C in some cases. Mostly because it is a tracing JIT compiler. I.e. it detects frequently executed operations at runtime, and then compiles them to native code. Which sometimes allows more efficient loop unrolling and inlining than a C compiler would be able to achive. I'm not precisely sure if this applies here, but it's also worth mentioning.

The master sysbench branch support the --luajit-cmd option, which can be used to either disable JIT compiler and use the Lua interpreter instead (--luajit-cmd=off) or dump some verbose information about JIT compilation (--luajit-cmd=dump).

[evandhoffman]

+1 for this issue. I'd like to automate running cpu/mem tests across thousands of boxes weekly/at provision time and post the result json to a central DB/Solr for analysis. Being able to emit native test results in json would make this way easier.