diff --git a/docs/src/manual.md b/docs/src/manual.md index df2755cf..337343cf 100644 --- a/docs/src/manual.md +++ b/docs/src/manual.md @@ -86,7 +86,18 @@ You can pass the following keyword arguments to `@benchmark`, `@benchmarkable`, - `time_tolerance`: The noise tolerance for the benchmark's time estimate, as a percentage. This is utilized after benchmark execution, when analyzing results. Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.time_tolerance = 0.05`. - `memory_tolerance`: The noise tolerance for the benchmark's memory estimate, as a percentage. This is utilized after benchmark execution, when analyzing results. Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.memory_tolerance = 0.01`. -To change the default values of the above fields, one can mutate the fields of `BenchmarkTools.DEFAULT_PARAMETERS`, for example: +The following keyword arguments relate are experimental and subject to change, see [Running custom benchmarks](@ref) for further details: + +- `run_customizable_func_only`: If `true`, only the customizable benchmark. Defaults to `BenchmarkTools.DEFAULT_PARAMETERS..run_customizable_func_only = false`. +- `enable_customizable_func`: If `:ALL` the customizable benchmark runs on every sample, if `:LAST` the customizable benchmark runs on the last sample, if `:FALSE` the customizable benchmark is never run. Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.enable_customizable_func = :FALSE` +- `customizable_gcsample`: If `true`, runs `gc()` before each sample of the customizable benchmark. Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.customizable_gcsample = false` +- `setup_prehook`: Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.teardown_posthook = _nothing_func`, which returns nothing. +- `teardown_posthook`: Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.teardown_posthook = _nothing_func`, which returns nothing. +- `sample_result`: Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.teardown_posthook = _nothing_func`, which returns nothing. +- `prehook`: Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.teardown_posthook = _nothing_func`, which returns nothing. +- `posthook`: Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.teardown_posthook = _nothing_func`, which returns nothing. + +To change the default values of the above fields, one can mutate the fields of `BenchmarkTools.DEFAULT_PARAMETERS` (this is not supported for `prehook` and `posthook`), for example: ```julia # change default for `seconds` to 2.5 @@ -347,10 +358,20 @@ BenchmarkTools.Trial gcsample: Bool false time_tolerance: Float64 0.05 memory_tolerance: Float64 0.01 + run_customizable_func_only: Bool false + enable_customizable_func: Symbol FALSE + customizable_gcsample: Bool false + setup_prehook: _nothing_func (function of type typeof(BenchmarkTools._nothing_func)) + teardown_posthook: _nothing_func (function of type typeof(BenchmarkTools._nothing_func)) + sample_result: _nothing_func (function of type typeof(BenchmarkTools._nothing_func)) + prehook: _nothing_func (function of type typeof(BenchmarkTools._nothing_func)) + posthook: _nothing_func (function of type typeof(BenchmarkTools._nothing_func)) times: Array{Float64}((10000,)) [26549.0, 26960.0, 27030.0, 27171.0, 27211.0, 27261.0, 27270.0, 27311.0, 27311.0, 27321.0 … 55383.0, 55934.0, 58649.0, 62847.0, 68547.0, 75761.0, 247081.0, 1.421718e6, 1.488322e6, 1.50329e6] gctimes: Array{Float64}((10000,)) [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 … 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.366184e6, 1.389518e6, 1.40116e6] memory: Int64 16752 allocs: Int64 19 + customizable_result: Nothing nothing + customizable_result_for_every_sample: Bool false ``` As you can see from the above, a couple of different timing estimates are pretty-printed with the `Trial`. You can calculate these estimates yourself using the `minimum`, `maximum`, `median`, `mean`, and `std` functions (Note that `median`, `mean`, and `std` are reexported in `BenchmarkTools` from `Statistics`): @@ -1008,3 +1029,51 @@ This will display each `Trial` as a violin plot. - BenchmarkTools attempts to be robust against machine noise occurring between *samples*, but BenchmarkTools can't do very much about machine noise occurring between *trials*. To cut down on the latter kind of noise, it is advised that you dedicate CPUs and memory to the benchmarking Julia process by using a shielding tool such as [cset](http://manpages.ubuntu.com/manpages/precise/man1/cset.1.html). - On some machines, for some versions of BLAS and Julia, the number of BLAS worker threads can exceed the number of available cores. This can occasionally result in scheduling issues and inconsistent performance for BLAS-heavy benchmarks. To fix this issue, you can use `BLAS.set_num_threads(i::Int)` in the Julia REPL to ensure that the number of BLAS threads is equal to or less than the number of available cores. - `@benchmark` is evaluated in global scope, even if called from local scope. + +## Experimental - Running custom benchmarks + +If you want to run code during a benchmark, e.g. to collect different metrics, say using perf, you can configure a custom benchmark. +A custom benchmark runs in the following way, where `benchmark_function` is the function we are benchmarking: +```julia +local setup_prehook_result +try + setup_prehook_result = setup_prehook(params) + $(setup) + prehook_result = prehook() + for _ in 1:evals + benchmark_function() + end + posthook_result = posthook() + return sample_result(params, setup_prehook_result, prehook_result, posthook_result) +finally + $(teardown) + teardown_posthook(params, setup_prehook_result) +end +``` +The result from `sample_result` is collected and can be accessed from the `customizable_result` field of `Trial`, which is the type of a benchmark result. + +Note that `prehook` and `posthook` should be as simple and fast as possible, moving any heavy lifting to `setup_prehook`, `sample_result` and `teardown_posthook`. + +As an example, these are the hooks to replicate the normal benchmarking functionality +```julia +setup_prehook(_) = nothing +samplefunc_prehook() = (Base.gc_num(), time_ns()) +samplefunc_posthook = samplefunc_prehook +function samplefunc_sample_result(params, _, prehook_result, posthook_result) + evals = params.evals + sample_time = posthook_result[2] - prehook_result[2] + gcdiff = Base.GC_Diff(posthook_result[1], prehook_result[1]) + + time = max((sample_time / evals) - params.overhead, 0.001) + gctime = max((gcdiff.total_time / evals) - params.overhead, 0.0) + memory = Int(Base.fld(gcdiff.allocd, evals)) + allocs = Int( + Base.fld( + gcdiff.malloc + gcdiff.realloc + gcdiff.poolalloc + gcdiff.bigalloc, + evals, + ), + ) + return time, gctime, memory, allocs +end +teardown_posthook(_, _) = nothing +``` diff --git a/src/BenchmarkTools.jl b/src/BenchmarkTools.jl index 37102cbe..766fe6c9 100644 --- a/src/BenchmarkTools.jl +++ b/src/BenchmarkTools.jl @@ -8,7 +8,7 @@ using Statistics using UUIDs: uuid4 using Printf using Profile -using Compat +using Compat: pkgversion, @noinline ############## # Parameters # diff --git a/src/execution.jl b/src/execution.jl index a9c3e25b..43aafc52 100644 --- a/src/execution.jl +++ b/src/execution.jl @@ -16,6 +16,7 @@ end mutable struct Benchmark samplefunc + customizable_func quote_vals params::Parameters end @@ -109,21 +110,61 @@ end function _run(b::Benchmark, p::Parameters; verbose=false, pad="", warmup=true, kwargs...) params = Parameters(p; kwargs...) @assert params.seconds > 0.0 "time limit must be greater than 0.0" - if warmup - b.samplefunc(b.quote_vals, Parameters(params; evals=1)) #warmup sample + if warmup #warmup sample + params.run_customizable_func_only && + b.samplefunc(b.quote_vals, Parameters(params; evals=1)) + !params.run_customizable_func_only && + b.customizable_func(b.quote_vals, Parameters(params; evals=1)) end trial = Trial(params) + if params.enable_customizable_func == :ALL + trial.customizable_result = [] + trial.customizable_result_for_every_sample = true + end params.gctrial && gcscrub() start_time = Base.time() - s = b.samplefunc(b.quote_vals, params) - push!(trial, s[1:(end - 1)]...) - return_val = s[end] + + return_val = nothing + if !params.run_customizable_func_only + s = b.samplefunc(b.quote_vals, params) + push!(trial, s[1:(end - 1)]...) + return_val = s[end] + end + if params.enable_customizable_func == :ALL + params.customizable_gcsample && gcscrub() + s = b.customizable_func(b.quote_vals, params) + push!(trial.customizable_result, s[1]) + + if params.run_customizable_func_only + return_val = s[end] + end + end + iters = 2 while (Base.time() - start_time) < params.seconds && iters ≤ params.samples - params.gcsample && gcscrub() - push!(trial, b.samplefunc(b.quote_vals, params)[1:(end - 1)]...) + if !params.run_customizable_func_only + params.gcsample && gcscrub() + push!(trial, b.samplefunc(b.quote_vals, params)[1:(end - 1)]...) + end + + if params.enable_customizable_func == :ALL + params.customizable_gcsample && gcscrub() + push!(trial.customizable_result, b.customizable_func(b.quote_vals, params)[1]) + end + iters += 1 end + + if params.enable_customizable_func == :LAST + params.customizable_gcsample && gcscrub() + s = b.customizable_func(b.quote_vals, params) + trial.customizable_result = s[1] + + if params.run_customizable_func_only + return_val = s[end] + end + end + return trial, return_val end @@ -506,6 +547,24 @@ macro benchmarkable(args...) end end +samplefunc_prehook() = (Base.gc_num(), time_ns()) +samplefunc_posthook = samplefunc_prehook +function samplefunc_sample_result(__params, _, prehook_result, posthook_result) + __evals = __params.evals + __sample_time = posthook_result[2] - prehook_result[2] + __gcdiff = Base.GC_Diff(posthook_result[1], prehook_result[1]) + + __time = max((__sample_time / __evals) - __params.overhead, 0.001) + __gctime = max((__gcdiff.total_time / __evals) - __params.overhead, 0.0) + __memory = Int(Base.fld(__gcdiff.allocd, __evals)) + __allocs = Int( + Base.fld( + __gcdiff.malloc + __gcdiff.realloc + __gcdiff.poolalloc + __gcdiff.bigalloc, + __evals, + ), + ) + return __time, __gctime, __memory, __allocs +end # `eval` an expression that forcibly defines the specified benchmark at # top-level in order to allow transfer of locally-scoped variables into # benchmark scope. @@ -519,6 +578,7 @@ function generate_benchmark_definition( @nospecialize corefunc = gensym("core") samplefunc = gensym("sample") + customizable_func = gensym("customizable") type_vars = [gensym() for i in 1:(length(quote_vars) + length(setup_vars))] signature = Expr(:call, corefunc, quote_vars..., setup_vars...) signature_def = Expr( @@ -562,32 +622,66 @@ function generate_benchmark_definition( @noinline function $(samplefunc)( $(Expr(:tuple, quote_vars...)), __params::$BenchmarkTools.Parameters ) - $(setup) - __evals = __params.evals - __gc_start = Base.gc_num() - __start_time = time_ns() - __return_val = $(invocation) - for __iter in 2:__evals - $(invocation) - end - __sample_time = time_ns() - __start_time - __gcdiff = Base.GC_Diff(Base.gc_num(), __gc_start) + $BenchmarkTools.@noinline $(setup) + # Isolate code so that e.g. setup doesn't cause different code to be generated by e.g. changing register allocation + # Unfortunately it still does, e.g. if you define a variable in setup then it's passed into invocation adding a few instructions + __prehook_result, __posthook_result, __return_val = $BenchmarkTools.@noinline ( + function (__evals) + prehook_result = $BenchmarkTools.samplefunc_prehook() + $BenchmarkTools.@noinline __return_val_2 = $(invocation) + for __iter in 2:__evals + $BenchmarkTools.@noinline $(invocation) + end + posthook_result = $BenchmarkTools.samplefunc_posthook() + return prehook_result, posthook_result, __return_val_2 + end + )( + __params.evals + ) $(teardown) - __time = max((__sample_time / __evals) - __params.overhead, 0.001) - __gctime = max((__gcdiff.total_time / __evals) - __params.overhead, 0.0) - __memory = Int(Base.fld(__gcdiff.allocd, __evals)) - __allocs = Int( - Base.fld( - __gcdiff.malloc + - __gcdiff.realloc + - __gcdiff.poolalloc + - __gcdiff.bigalloc, - __evals, + return $BenchmarkTools.samplefunc_sample_result( + __params, nothing, __prehook_result, __posthook_result + )..., + __return_val + end + @noinline function $(customizable_func)( + $(Expr(:tuple, quote_vars...)), __params::$BenchmarkTools.Parameters + ) + local __setup_prehook_result + try + __setup_prehook_result = $BenchmarkTools.@noinline __params.setup_prehook( + __params + ) + $BenchmarkTools.@noinline $(setup) + __prehook_result, __posthook_result, __return_val = $BenchmarkTools.@noinline ( + function (__evals) + prehook_result = __params.prehook() + # We'll run it evals times. + $BenchmarkTools.@noinline __return_val_2 = $(invocation) + for __iter in 2:__evals + $BenchmarkTools.@noinline $(invocation) + end + posthook_result = __params.posthook() + return prehook_result, posthook_result, __return_val_2 + end + )( + __params.evals + ) + return __params.sample_result( + __params, + __setup_prehook_result, + __prehook_result, + __posthook_result, ), - ) - return __time, __gctime, __memory, __allocs, __return_val + __return_val + finally + $(teardown) + __params.teardown_posthook(__params, __setup_prehook_result) + end end - $BenchmarkTools.Benchmark($(samplefunc), $(quote_vals), $(params)) + $BenchmarkTools.Benchmark( + $(samplefunc), $(customizable_func), $(quote_vals), $(params) + ) end, ) end diff --git a/src/parameters.jl b/src/parameters.jl index ff1bc615..142663f0 100644 --- a/src/parameters.jl +++ b/src/parameters.jl @@ -5,7 +5,7 @@ const RESOLUTION = 1000 # 1 μs = 1000 ns # Parameters # ############## -mutable struct Parameters +mutable struct Parameters{A,B} seconds::Float64 samples::Int evals::Int @@ -15,9 +15,133 @@ mutable struct Parameters gcsample::Bool time_tolerance::Float64 memory_tolerance::Float64 + run_customizable_func_only::Bool + enable_customizable_func::Symbol + customizable_gcsample::Bool + setup_prehook + teardown_posthook + sample_result + prehook::A + posthook::B + + function Parameters{A,B}( + seconds, + samples, + evals, + evals_set, + overhead, + gctrial, + gcsample, + time_tolerance, + memory_tolerance, + run_customizable_func_only, + enable_customizable_func, + customizable_gcsample, + setup_prehook, + teardown_posthook, + sample_result, + prehook::A, + posthook::B, + ) where {A,B} + if enable_customizable_func ∉ (:FALSE, :ALL, :LAST) + throw( + ArgumentError( + "invalid value $(enable_customizable_func) for enable_customizable_func which must be :FALSE, :ALL or :LAST", + ), + ) + end + if run_customizable_func_only && enable_customizable_func == :FALSE + throw( + ArgumentError( + "run_customizable_func_only is set to true, but enable_customizable_func is set to :FALSE", + ), + ) + end + return new( + seconds, + samples, + evals, + evals_set, + overhead, + gctrial, + gcsample, + time_tolerance, + memory_tolerance, + run_customizable_func_only, + enable_customizable_func, + customizable_gcsample, + setup_prehook, + teardown_posthook, + sample_result, + prehook, + posthook, + ) + end +end + +# https://github.com/JuliaLang/julia/issues/17186 +function Parameters( + seconds, + samples, + evals, + evals_set, + overhead, + gctrial, + gcsample, + time_tolerance, + memory_tolerance, + run_customizable_func_only, + enable_customizable_func, + customizable_gcsample, + setup_prehook, + teardown_posthook, + sample_result, + prehook::A, + posthook::B, +) where {A,B} + return Parameters{A,B}( + seconds, + samples, + evals, + evals_set, + overhead, + gctrial, + gcsample, + time_tolerance, + memory_tolerance, + run_customizable_func_only, + enable_customizable_func, + customizable_gcsample, + setup_prehook, + teardown_posthook, + sample_result, + prehook, + posthook, + ) end -const DEFAULT_PARAMETERS = Parameters(5.0, 10000, 1, false, 0, true, false, 0.05, 0.01) +_nothing_func(args...) = nothing +DEFAULT_PARAMETERS = Parameters( + 5.0, + 10000, + 1, + false, + 0, + true, + false, + 0.05, + 0.01, + # customizable Parameters + false, + :FALSE, + false, + # customizable functions + _nothing_func, + _nothing_func, + _nothing_func, + _nothing_func, + _nothing_func, +) function Parameters(; seconds=DEFAULT_PARAMETERS.seconds, @@ -29,6 +153,14 @@ function Parameters(; gcsample=DEFAULT_PARAMETERS.gcsample, time_tolerance=DEFAULT_PARAMETERS.time_tolerance, memory_tolerance=DEFAULT_PARAMETERS.memory_tolerance, + run_customizable_func_only=DEFAULT_PARAMETERS.run_customizable_func_only, + enable_customizable_func=DEFAULT_PARAMETERS.enable_customizable_func, + customizable_gcsample=DEFAULT_PARAMETERS.customizable_gcsample, + setup_prehook=DEFAULT_PARAMETERS.setup_prehook, + teardown_posthook=DEFAULT_PARAMETERS.teardown_posthook, + sample_result=DEFAULT_PARAMETERS.sample_result, + prehook=DEFAULT_PARAMETERS.prehook, + posthook=DEFAULT_PARAMETERS.posthook, ) return Parameters( seconds, @@ -40,6 +172,14 @@ function Parameters(; gcsample, time_tolerance, memory_tolerance, + run_customizable_func_only, + enable_customizable_func, + customizable_gcsample, + setup_prehook, + teardown_posthook, + sample_result, + prehook, + posthook, ) end @@ -48,24 +188,83 @@ function Parameters( seconds=nothing, samples=nothing, evals=nothing, + evals_set=nothing, overhead=nothing, gctrial=nothing, gcsample=nothing, time_tolerance=nothing, memory_tolerance=nothing, + run_customizable_func_only=nothing, + enable_customizable_func=nothing, + customizable_gcsample=nothing, + setup_prehook=nothing, + teardown_posthook=nothing, + sample_result=nothing, + prehook=nothing, + posthook=nothing, ) - params = Parameters() - params.seconds = seconds != nothing ? seconds : default.seconds - params.samples = samples != nothing ? samples : default.samples - params.evals = evals != nothing ? evals : default.evals - params.overhead = overhead != nothing ? overhead : default.overhead - params.gctrial = gctrial != nothing ? gctrial : default.gctrial - params.gcsample = gcsample != nothing ? gcsample : default.gcsample - params.time_tolerance = + params_seconds = seconds != nothing ? seconds : default.seconds + params_samples = samples != nothing ? samples : default.samples + params_evals = evals != nothing ? evals : default.evals + params_evals_set = evals_set != nothing ? evals_set : default.evals_set + params_overhead = overhead != nothing ? overhead : default.overhead + params_gctrial = gctrial != nothing ? gctrial : default.gctrial + params_gcsample = gcsample != nothing ? gcsample : default.gcsample + params_time_tolerance = time_tolerance != nothing ? time_tolerance : default.time_tolerance - params.memory_tolerance = + params_memory_tolerance = memory_tolerance != nothing ? memory_tolerance : default.memory_tolerance - return params::BenchmarkTools.Parameters + params_run_customizable_func_only = if run_customizable_func_only != nothing + run_customizable_func_only + else + default.run_customizable_func_only + end + params_enable_customizable_func = if enable_customizable_func != nothing + enable_customizable_func + else + default.enable_customizable_func + end + params_customizable_gcscrub = if customizable_gcsample != nothing + customizable_gcsample + else + default.customizable_gcsample + end + params_setup_prehook = if setup_prehook != nothing + setup_prehook + else + default.setup_prehook + end + params_teardown_posthook = if teardown_posthook != nothing + teardown_posthook + else + default.teardown_posthook + end + params_sample_result = if sample_result != nothing + sample_result + else + default.sample_result + end + params_prehook = prehook != nothing ? prehook : default.prehook + params_posthook = posthook != nothing ? posthook : default.posthook + return Parameters( + params_seconds, + params_samples, + params_evals, + params_evals_set, + params_overhead, + params_gctrial, + params_gcsample, + params_time_tolerance, + params_memory_tolerance, + params_run_customizable_func_only, + params_enable_customizable_func, + params_customizable_gcscrub, + params_setup_prehook, + params_teardown_posthook, + params_sample_result, + params_prehook, + params_posthook, + )::BenchmarkTools.Parameters end function Base.:(==)(a::Parameters, b::Parameters) @@ -76,7 +275,15 @@ function Base.:(==)(a::Parameters, b::Parameters) a.gctrial == b.gctrial && a.gcsample == b.gcsample && a.time_tolerance == b.time_tolerance && - a.memory_tolerance == b.memory_tolerance + a.memory_tolerance == b.memory_tolerance && + a.run_customizable_func_only == b.run_customizable_func_only && + a.enable_customizable_func == b.enable_customizable_func && + a.customizable_gcsample == b.customizable_gcsample && + a.setup_prehook == b.setup_prehook && + a.teardown_posthook == b.teardown_posthook && + a.sample_result == b.sample_result && + a.prehook == b.prehook && + a.posthook == b.posthook end function Base.copy(p::Parameters) @@ -90,6 +297,14 @@ function Base.copy(p::Parameters) p.gcsample, p.time_tolerance, p.memory_tolerance, + p.run_customizable_func_only, + p.enable_customizable_func, + p.customizable_gcsample, + p.setup_prehook, + p.teardown_posthook, + p.sample_result, + p.prehook, + p.posthook, ) end diff --git a/src/serialization.jl b/src/serialization.jl index 7bec2c8d..3afabead 100644 --- a/src/serialization.jl +++ b/src/serialization.jl @@ -16,6 +16,8 @@ const SUPPORTED_TYPES = Dict{Symbol,Type}( ) # n.b. Benchmark type not included here, since it is gensym'd +customizable_result_recover(::Nothing) = nothing + function JSON.lower(x::Union{values(SUPPORTED_TYPES)...}) d = Dict{String,Any}() T = typeof(x) @@ -23,6 +25,7 @@ function JSON.lower(x::Union{values(SUPPORTED_TYPES)...}) name = String(fieldname(T, i)) field = getfield(x, i) ft = typeof(field) + field = ft <: Function ? nothing : field value = ft <: get(SUPPORTED_TYPES, nameof(ft), Union{}) ? JSON.lower(field) : field d[name] = value end @@ -50,19 +53,32 @@ function recover(x::Vector) for i in 1:fc ft = fieldtype(T, i) fn = String(fieldname(T, i)) - if ft <: get(SUPPORTED_TYPES, nameof(ft), Union{}) - xsi = recover(fields[fn]) - else - xsi = if fn == "evals_set" && !haskey(fields, fn) - false - elseif fn in ("seconds", "overhead", "time_tolerance", "memory_tolerance") && - fields[fn] === nothing - # JSON spec doesn't support Inf - # These fields should all be >= 0, so we can ignore -Inf case - typemax(ft) + xsi = if fn == "customizable_result" + customizable_result_recover(fields[fn]) + elseif ft <: get(SUPPORTED_TYPES, nameof(ft), Union{}) + recover(fields[fn]) + elseif fn in ( + "setup_prehook", "teardown_posthook", "sample_result", "prehook", "posthook" + ) + getfield(BenchmarkTools.DEFAULT_PARAMETERS, Symbol(fn)) + elseif fn == "evals_set" && !haskey(fields, fn) + false + elseif fn in ("seconds", "overhead", "time_tolerance", "memory_tolerance") && + fields[fn] === nothing + # JSON spec doesn't support Inf + # These fields should all be >= 0, so we can ignore -Inf case + typemax(ft) + elseif fn == "enable_customizable_func" + if !haskey(fields, fn) + :FALSE else - convert(ft, fields[fn]) + Symbol(fields[fn]) end + elseif fn in ("run_customizable_func_only", "customizable_gcsample") && + !haskey(fields, fn) + getfield(BenchmarkTools.DEFAULT_PARAMETERS, Symbol(fn)) + else + convert(ft, fields[fn]) end if T == BenchmarkGroup && xsi isa Dict for (k, v) in copy(xsi) diff --git a/src/trials.jl b/src/trials.jl index 67382813..9cb55399 100644 --- a/src/trials.jl +++ b/src/trials.jl @@ -8,6 +8,28 @@ mutable struct Trial gctimes::Vector{Float64} memory::Int allocs::Int + customizable_result + customizable_result_for_every_sample::Bool + + function Trial( + params, + times, + gctimes, + memory, + allocs, + customizable_result=nothing, + customizable_result_for_every_sample=false, + ) + return new( + params, + times, + gctimes, + memory, + allocs, + customizable_result, + customizable_result_for_every_sample, + ) + end end Trial(params::Parameters) = Trial(params, Float64[], Float64[], typemax(Int), typemax(Int)) @@ -21,7 +43,18 @@ function Base.:(==)(a::Trial, b::Trial) end function Base.copy(t::Trial) - return Trial(copy(t.params), copy(t.times), copy(t.gctimes), t.memory, t.allocs) + return Trial( + copy(t.params), + copy(t.times), + copy(t.gctimes), + t.memory, + t.allocs, + if t.customizable_result_for_every_sample + copy(t.customizable_result) + else + t.customizable_result + end, + ) end function Base.push!(t::Trial, time, gctime, memory, allocs) @@ -40,9 +73,33 @@ end Base.length(t::Trial) = length(t.times) function Base.getindex(t::Trial, i::Number) - return push!(Trial(t.params), t.times[i], t.gctimes[i], t.memory, t.allocs) + return Trial( + t.params, + [t.times[i]], + [t.gctimes[i]], + t.memory, + t.allocs, + if t.customizable_result_for_every_sample + [t.customizable_result[i]] + else + t.customizable_result + end, + ) +end +function Base.getindex(t::Trial, i) + return Trial( + t.params, + t.times[i], + t.gctimes[i], + t.memory, + t.allocs, + if t.customizable_result_for_every_sample + t.customizable_result[i] + else + t.customizable_result + end, + ) end -Base.getindex(t::Trial, i) = Trial(t.params, t.times[i], t.gctimes[i], t.memory, t.allocs) Base.lastindex(t::Trial) = length(t) function Base.sort!(t::Trial) @@ -98,10 +155,19 @@ mutable struct TrialEstimate gctime::Float64 memory::Int allocs::Int + customizable_result + + function TrialEstimate( + params, times, gctime, memory, allocs, customizable_result=nothing + ) + return new(params, times, gctime, memory, allocs, customizable_result) + end end function TrialEstimate(trial::Trial, t, gct) - return TrialEstimate(params(trial), t, gct, memory(trial), allocs(trial)) + return TrialEstimate( + params(trial), t, gct, memory(trial), allocs(trial), trial.customizable_result + ) end function Base.:(==)(a::TrialEstimate, b::TrialEstimate) @@ -113,7 +179,9 @@ function Base.:(==)(a::TrialEstimate, b::TrialEstimate) end function Base.copy(t::TrialEstimate) - return TrialEstimate(copy(t.params), t.time, t.gctime, t.memory, t.allocs) + return TrialEstimate( + copy(t.params), t.time, t.gctime, t.memory, t.allocs, t.customizable_result + ) end function Base.minimum(trial::Trial) diff --git a/test/CustomizableBenchmarkTests.jl b/test/CustomizableBenchmarkTests.jl new file mode 100644 index 00000000..e66ca38f --- /dev/null +++ b/test/CustomizableBenchmarkTests.jl @@ -0,0 +1,93 @@ +module CustomizableBenchmarkTests + +using BenchmarkTools +using Test + +x = Ref(0) +setup_prehook(_) = x[] += 1 +prehook() = x[] += 1 +posthook() = x[] += 1 +function sample_result(_, setup_prehook_result, preehook_result, posthook_result) + @test setup_prehook_result == 1 + @test preehook_result == 2 + @test posthook_result == 3 + @test x[] == 3 + return x[] += 1 +end +function teardown_posthook(_, setup_prehook_result) + @test setup_prehook_result == 1 + @test x[] == 4 + return x[] += 1 +end + +@testset "Disabled custom benchmarking" begin + x[] = 0 + res = @benchmark nothing setup_prehook = setup_prehook prehook = prehook posthook = + posthook sample_result = sample_result teardown_posthook = teardown_posthook run_customizable_func_only = + false + @test res.customizable_result === nothing + @test !res.customizable_result_for_every_sample +end + +@testset "custom benchmarking last" begin + for run_customizable_func_only in (true, false) + x[] = 0 + res = @benchmark nothing setup_prehook = setup_prehook prehook = prehook posthook = + posthook sample_result = sample_result teardown_posthook = teardown_posthook enable_customizable_func = + :LAST run_customizable_func_only = run_customizable_func_only + if run_customizable_func_only + @test isempty(res.times) + @test isempty(res.gctimes) + @test res.memory == typemax(Int) + @test res.allocs == typemax(Int) + end + @test !res.customizable_result_for_every_sample + @test res.customizable_result === 4 + end +end + +@testset "custom benchmark every sample, independent of iterations" begin + for run_customizable_func_only in (true, false) + x[] = 0 + setup_prehook(_) = x[] = 1 + res = @benchmark nothing setup_prehook = setup_prehook prehook = prehook posthook = + posthook sample_result = sample_result teardown_posthook = teardown_posthook enable_customizable_func = + :ALL run_customizable_func_only = run_customizable_func_only samples = 1000 + if run_customizable_func_only + @test isempty(res.times) + @test isempty(res.gctimes) + @test res.memory == typemax(Int) + @test res.allocs == typemax(Int) + end + @test res.customizable_result_for_every_sample + @test res.customizable_result == fill(4, 1000) + end +end + +@testset "custom benchmark every sample with iteration dependence" begin + for run_customizable_func_only in (true, false) + x[] = 0 + setup_prehook(_) = x[] += 1 + prehook() = x[] += 1 + posthook() = x[] += 1 + function sample_result(_, setup_prehook_result, preehook_result, posthook_result) + return x[] += 1 + end + function teardown_posthook(_, setup_prehook_result) + return x[] += 1 + end + res = @benchmark nothing setup_prehook = setup_prehook prehook = prehook posthook = + posthook sample_result = sample_result teardown_posthook = teardown_posthook enable_customizable_func = + :ALL run_customizable_func_only = run_customizable_func_only samples = 1000 + if run_customizable_func_only + @test isempty(res.times) + @test isempty(res.gctimes) + @test res.memory == typemax(Int) + @test res.allocs == typemax(Int) + end + @test res.customizable_result_for_every_sample + @test res.customizable_result == collect(5 * (1:1000) .- 1) + end +end + +end # module diff --git a/test/ParametersTests.jl b/test/ParametersTests.jl index 9fa07027..167fce77 100644 --- a/test/ParametersTests.jl +++ b/test/ParametersTests.jl @@ -17,6 +17,7 @@ BenchmarkTools.DEFAULT_PARAMETERS.gctrial = p.gctrial BenchmarkTools.DEFAULT_PARAMETERS.seconds = oldseconds BenchmarkTools.DEFAULT_PARAMETERS.gctrial = oldgctrial +f(x) = x p = Parameters(; seconds=1, gctrial=false, @@ -26,6 +27,14 @@ p = Parameters(; gcsample=false, time_tolerance=0.043, memory_tolerance=0.15, + # customizable Parameters + run_customizable_func_only=true, + enable_customizable_func=:ALL, + customizable_gcsample=true, + # customizable functions + setup_prehook=f, + teardown_posthook=f, + sample_result=f, ) oldseconds = BenchmarkTools.DEFAULT_PARAMETERS.seconds oldgctrial = BenchmarkTools.DEFAULT_PARAMETERS.gctrial @@ -35,6 +44,16 @@ oldsamples = BenchmarkTools.DEFAULT_PARAMETERS.samples oldevals = BenchmarkTools.DEFAULT_PARAMETERS.evals oldoverhead = BenchmarkTools.DEFAULT_PARAMETERS.overhead oldgcsample = BenchmarkTools.DEFAULT_PARAMETERS.gcsample +old_run_customizable_func_only = + BenchmarkTools.DEFAULT_PARAMETERS.run_customizable_func_only +old_enable_customizable_func = BenchmarkTools.DEFAULT_PARAMETERS.enable_customizable_func +old_customizable_gcsample = BenchmarkTools.DEFAULT_PARAMETERS.customizable_gcsample +old_setup_prehook = BenchmarkTools.DEFAULT_PARAMETERS.setup_prehook +old_teardown_posthook = BenchmarkTools.DEFAULT_PARAMETERS.teardown_posthook +old_sample_result = BenchmarkTools.DEFAULT_PARAMETERS.sample_result +old_prehook = BenchmarkTools.DEFAULT_PARAMETERS.prehook +old_posthook = BenchmarkTools.DEFAULT_PARAMETERS.posthook + BenchmarkTools.DEFAULT_PARAMETERS.seconds = p.seconds BenchmarkTools.DEFAULT_PARAMETERS.gctrial = p.gctrial BenchmarkTools.DEFAULT_PARAMETERS.time_tolerance = p.time_tolerance @@ -43,6 +62,13 @@ BenchmarkTools.DEFAULT_PARAMETERS.samples = p.samples BenchmarkTools.DEFAULT_PARAMETERS.evals = p.evals BenchmarkTools.DEFAULT_PARAMETERS.overhead = p.overhead BenchmarkTools.DEFAULT_PARAMETERS.gcsample = p.gcsample +BenchmarkTools.DEFAULT_PARAMETERS.run_customizable_func_only = p.run_customizable_func_only +BenchmarkTools.DEFAULT_PARAMETERS.enable_customizable_func = p.enable_customizable_func +BenchmarkTools.DEFAULT_PARAMETERS.customizable_gcsample = p.customizable_gcsample +BenchmarkTools.DEFAULT_PARAMETERS.setup_prehook = p.setup_prehook +BenchmarkTools.DEFAULT_PARAMETERS.teardown_posthook = p.teardown_posthook +BenchmarkTools.DEFAULT_PARAMETERS.sample_result = p.sample_result + @test p == Parameters() @test p == Parameters(p) BenchmarkTools.DEFAULT_PARAMETERS.seconds = oldseconds @@ -53,5 +79,31 @@ BenchmarkTools.DEFAULT_PARAMETERS.samples = oldsamples BenchmarkTools.DEFAULT_PARAMETERS.evals = oldevals BenchmarkTools.DEFAULT_PARAMETERS.overhead = oldoverhead BenchmarkTools.DEFAULT_PARAMETERS.gcsample = oldgcsample +BenchmarkTools.DEFAULT_PARAMETERS.run_customizable_func_only = + old_run_customizable_func_only +BenchmarkTools.DEFAULT_PARAMETERS.enable_customizable_func = old_enable_customizable_func +BenchmarkTools.DEFAULT_PARAMETERS.customizable_gcsample = old_customizable_gcsample +BenchmarkTools.DEFAULT_PARAMETERS.setup_prehook = old_setup_prehook +BenchmarkTools.DEFAULT_PARAMETERS.teardown_posthook = old_teardown_posthook +BenchmarkTools.DEFAULT_PARAMETERS.sample_result = old_sample_result + +for vals in (false, true, :ARST, :TRUE, :false, :ON) + @test_throws ArgumentError Parameters(p; enable_customizable_func=vals) + @test_throws ArgumentError Parameters(; enable_customizable_func=vals) +end + +@test_throws ArgumentError Parameters(; + enable_customizable_func=:FALSE, run_customizable_func_only=true +) +@test_nowarn Parameters(; enable_customizable_func=:FALSE, run_customizable_func_only=false) +for run_customizable_func_only in (false, true) + @test_nowarn Parameters(; + enable_customizable_func=:ALL, run_customizable_func_only=run_customizable_func_only + ) + @test_nowarn Parameters(; + enable_customizable_func=:LAST, + run_customizable_func_only=run_customizable_func_only, + ) +end end # module diff --git a/test/SerializationTests.jl b/test/SerializationTests.jl index e24314a1..bafe6377 100644 --- a/test/SerializationTests.jl +++ b/test/SerializationTests.jl @@ -6,7 +6,7 @@ using Test function eq(x::T, y::T) where {T<:Union{values(BenchmarkTools.SUPPORTED_TYPES)...}} return all(i -> eq(getfield(x, i), getfield(y, i)), 1:fieldcount(T)) end -eq(x::T, y::T) where {T} = isapprox(x, y) +eq(x::T, y::T) where {T} = x == y function withtempdir(f::Function) d = mktempdir() @@ -99,22 +99,78 @@ end @test_throws ArgumentError BenchmarkTools.recover([1]) end -@testset "Backwards Comppatibility with evals_set" begin +@testset "Backwards Compatibility with evals_set" begin json_string = "[{\"Julia\":\"1.11.0-DEV.1116\",\"BenchmarkTools\":\"1.4.0\"},[[\"Parameters\",{\"gctrial\":true,\"time_tolerance\":0.05,\"samples\":10000,\"evals\":1,\"gcsample\":false,\"seconds\":5.0,\"overhead\":0.0,\"memory_tolerance\":0.01}]]]" json_io = IOBuffer(json_string) - @test BenchmarkTools.load(json_io) == - [BenchmarkTools.Parameters(5.0, 10000, 1, false, 0.0, true, false, 0.05, 0.01)] + @test BenchmarkTools.load(json_io) == [ + BenchmarkTools.Parameters( + 5.0, + 10000, + 1, + false, + 0.0, + true, + false, + 0.05, + 0.01, + false, + :FALSE, + false, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + ), + ] json_string = "[{\"Julia\":\"1.11.0-DEV.1116\",\"BenchmarkTools\":\"1.4.0\"},[[\"Parameters\",{\"gctrial\":true,\"time_tolerance\":0.05,\"evals_set\":true,\"samples\":10000,\"evals\":1,\"gcsample\":false,\"seconds\":5.0,\"overhead\":0.0,\"memory_tolerance\":0.01}]]]" json_io = IOBuffer(json_string) - @test BenchmarkTools.load(json_io) == - [BenchmarkTools.Parameters(5.0, 10000, 1, true, 0.0, true, false, 0.05, 0.01)] + @test BenchmarkTools.load(json_io) == [ + BenchmarkTools.Parameters( + 5.0, + 10000, + 1, + true, + 0.0, + true, + false, + 0.05, + 0.01, + false, + :FALSE, + false, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + ), + ] end @testset "Inf in Paramters struct" begin - params = BenchmarkTools.Parameters(Inf, 10000, 1, false, Inf, true, false, Inf, Inf) + params = BenchmarkTools.Parameters( + Inf, + 10000, + 1, + false, + Inf, + true, + false, + Inf, + Inf, + false, + :FALSE, + false, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + BenchmarkTools._nothing_func, + ) io = IOBuffer() BenchmarkTools.save(io, params) diff --git a/test/runtests.jl b/test/runtests.jl index 6f58393a..82fd4c8f 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -34,3 +34,7 @@ println("done (took ", took_seconds, " seconds)") print("Testing serialization...") took_seconds = @elapsed include("SerializationTests.jl") println("done (took ", took_seconds, " seconds)") + +print("Testing custom benchmarking...") +took_seconds = @elapsed include("CustomizableBenchmarkTests.jl") +println("done (took ", took_seconds, " seconds)")