Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What code does @benchmark actually benchmark? #285

Open
benninkrs opened this issue Jun 18, 2022 · 0 comments
Open

What code does @benchmark actually benchmark? #285

benninkrs opened this issue Jun 18, 2022 · 0 comments
Labels

Comments

@benninkrs
Copy link

(Apologies if this not the proper place/way for me to ask this question.)

I recently discovered that @benchmark does not work the way I thought it did--and hence is not benchmarking what I thought it was. I would be very grateful if one of the authors (perhaps @jrevels?) could take a few moments to clarify how it works and the rationale behind it.

Suppose we have defined a function f and run the following:

x = 456;
y = 789;
@btime f(123, $x, y)

I read somewhere that the aim of BenchmarkTools is to estimate the cost of executing a given expression in typical use; in Julia the scope is usually a function. The manual also gives the impression that interpolation replaces subexpressions with their values (as it does in string and macro interpolation). Thus I imagined that @btime f(123, $x, y) would expand to something like

wrapper(y_) = f(123, 456, y_);
wrapper(y);   # get compilation out of the way
start_timer()
wrapper(y)
stop_timer()

That is, (1) the expression in question is wrapped in a function, (2) literal constants remain as they are, (3) interpolated variables are replaced by their (globally evaluated) values, and (4) uninterpolated arguments in the original expression become parameters of the wrapping function.

However, from looking at generate_benchmark_definition(), it seems that the expanded code is something more like

wrapper(x_) = f(123, x_, y);
wrapper(x);   # get compilation out of the way
start_timer()
wrapper(x)
stop_timer()

That is, (3') interpolated arguments become parameters of the wrapping function, and (4') uninterpolated arguments remain as global-scope bindings (evaluated during the benchmark).

Is this second description accurate (modulo details like gc and repetition)?
If so, what are/were the reasons for implementing it that way instead of how I originally imagined?

Thank you for your insights.

@gdalle gdalle added this to the v2.0 milestone Sep 18, 2023
@gdalle gdalle removed this from the v2.0 milestone Jan 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants