Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add experimental support for customisable benchmarking #347

Open
wants to merge 29 commits into
base: master
Choose a base branch
from

Conversation

Zentrik
Copy link
Contributor

@Zentrik Zentrik commented Dec 28, 2023

Replaces #325 (closes #325)

This pr adds the ability to run a custom benchmarking function which has hooks to inject in custom functions. The current design supports running perf on a benchmark (see https://github.com/Zentrik/BenchmarkToolsPlusLinuxPerf.jl) and profiling benchmarks (excluding the setup, teardown and gcscrub which the current bprofile includes).

@Zentrik

This comment was marked as outdated.

@Zentrik

This comment was marked as resolved.

@Zentrik

This comment was marked as resolved.

@Zentrik Zentrik changed the title [WIP] Add experimental support for perf (via LinuxPerf.jl) Add support for perf (via LinuxPerf.jl) Dec 30, 2023
@Zentrik Zentrik changed the title Add support for perf (via LinuxPerf.jl) Add experimental support for perf (via LinuxPerf.jl) Dec 30, 2023
Zentrik added a commit to Zentrik/LinuxPerf.jl that referenced this pull request Dec 30, 2023
@Zentrik

This comment was marked as outdated.

src/trials.jl Show resolved Hide resolved
@Zentrik Zentrik marked this pull request as ready for review December 30, 2023 14:44
@DilumAluthge
Copy link
Member

I haven't added tests, but given the CI doesn't have perf available, not sure how useful it would be.

We might need to set up Buildkite CI on this repo. @vchuravy @staticfloat

src/trials.jl Show resolved Hide resolved
@Zentrik Zentrik force-pushed the linux-perf branch 2 times, most recently from f27a5de to bbfb733 Compare January 7, 2024 23:31
@Zentrik

This comment was marked as outdated.

src/execution.jl Outdated Show resolved Hide resolved
@Zentrik
Copy link
Contributor Author

Zentrik commented Jun 20, 2024

I'd like to mark this feature as experimental or something to that effect so that we can make breaking changes to it without making a breaking change to BenchmarkTools.
The changes to samplefunc are largely just so that it matches customisable_func (theoretically it reduces overhead by a couple of instructions but that's not noticeable) and also so the hooks for samplefunc are exposed.

I'll make a separate pr removing Buildkite.

I've moved the LinuxPerf stuff to https://github.com/Zentrik/BenchmarkToolsPlusLinuxPerf.jl.

I do have a version of bprofile that uses this functionality to only profile the relevant stuff, but because samples are generally quite short (on the order of microseconds) not much gets profiled making it pretty useless.

bprofile_setup_prehook(_) = Profile.check_init()
function bprofile_prehook()
    results = samplefunc_prehook()
    status = ccall(:jl_profile_start_timer, Cint, ())
    if status < 0
        error(Profile.error_codes[status])
    end
    return results
end
function bprofile_posthook()
    Profile.stop_timer()
    return samplefunc_posthook()
end

# """
#     @bprofile expression [other parameters...]

# Run `@benchmark` while profiling. This is similar to

#     @profile @benchmark expression [other parameters...]

# but the profiling is applied only to the main
# execution (after compilation and tuning).
# The profile buffer is cleared prior to execution.

# View the profile results with `Profile.print(...)`.
# See the profiling section of the Julia manual for more
# information.
# """
macro bprofile(args...)
    _, params = prunekwargs(args...)
    if !haskw(args, :gctrial)
        args = (args..., Expr(:kw, :gctrial, false))
    end
    if !haskw(args, :gcsample)
        args = (args..., Expr(:kw, :gcsample, false))
    end
    tmp = gensym()
    return esc(
        quote
            local $tmp = $BenchmarkTools.@benchmarkable $(args...)
            $(
                if hasevals(params)
                    :(run(
                        $tmp, $BenchmarkTools.Parameters($tmp.params; evals=1); warmup=false
                    ))
                else
                    :($BenchmarkTools.tune!($tmp))
                end
            )
            $BenchmarkTools.Profile.stop_timer()
            $BenchmarkTools.Profile.clear()
            $BenchmarkTools.run(
                $tmp;
                setup_prehook=$BenchmarkTools.bprofile_setup_prehook,
                prehook=$BenchmarkTools.bprofile_prehook,
                posthook=$BenchmarkTools.bprofile_posthook,
                sample_result=$BenchmarkTools.samplefunc_sample_result,
                enable_customisable_func=:ALL,
                run_customisable_func_only=true,
            )
        end,
    )
end

@Zentrik Zentrik changed the title Add experimental support for perf (via LinuxPerf.jl) Add experimental support for most customisable benchmarking Jun 20, 2024
This should theoretically allow FunctionWrappers or callable structs to be used, though serialization may be an issue.
@Zentrik Zentrik changed the title Add experimental support for most customisable benchmarking Add experimental support for customisable benchmarking Jun 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants