Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bench modules (ExUnit-style benchmarks definitions) (DSL) #236

Open
hauleth opened this issue Jul 27, 2018 · 5 comments

Comments

@hauleth
Copy link

commented Jul 27, 2018

I am thinking about writing support for ExUnit-style benchmarks (if there is no work toward such thing yet). And I am having something like that in mind:

defmodule SampleBench do
  use Benchee.Unit

  bench do
    input "Small (1 Thousand)", do: Enum.to_list(1..1_000)
    input "Middle (100 Thousand)", do: Enum.to_list(1..100_000)

    setup_all input do
      {Enum.shuffle(input), fn i -> [i, i + 1] end}
    end

    func "flat_map", {input, func} do
      Enum.flat_map(input, func)
    end
    
    func "map.flatten", {input, func} do
      input |> Enum.map(func) |> List.flatten
    end
  end
end

Any suggestions or other propositions for the syntax?

@PragTob PragTob changed the title Bench modules (ExUnit-style benchmarks definitions) Bench modules (ExUnit-style benchmarks definitions) (DSL) Jul 27, 2018

@PragTob

This comment has been minimized.

Copy link
Member

commented Jul 27, 2018

👋

Hi there and thanks for checking in :) Sometimes people ask about this, so it's also somewhere on my "fun" agenda for after 1.0. Hence I've given this some thought in the past :)

  • I'd keep the naming closer to what the options in benchee are named, for familarity sake :) What is called func there we usually call job. I'm not sure what setup_all maps to in benchee. I'd think that it would be executed once before all of them but I guess it maps to what we call before_scenario
  • Would be interesting how to deal with all the other config options (time, warmup, precheck etc.). At best the mechanism would be fairly generic so that not every time "benchee core" adds an option the DSL would need to be adapted.
  • Specifying formatters might also be very interesting, keep in mind that we're changing that API soon (bring arguments closer to the formatter themselves)
  • also how to incorporate hooks that are specific for just one job might be interesting or left out

Otherwise looks good to me on a first view :)

I added DSL to the title, as that's always how I thought about this as Benchee.DSL. Of course as you do it it's your choice just my input 😁

On another not - I think it's interesting how you add the anonymous function in the setup_all - it avoids duplication but at first confused the hell out of me 😄 (mainly because I usually put it as a variable at the top and then reuse it in the job definitions)

Hope I was helpful 👋

@devonestes

This comment has been minimized.

Copy link
Collaborator

commented Jul 27, 2018

@hauleth What about the DSL do you prefer to the current API? Is there some functionality that you think the DSL could offer that the current API does not? Or is there something confusing about the current API that is made clearer with this DSL?

I'm totally open to the idea, but I would love some further input as to what users would gain by having a second API available to them. It would likely be a significant maintenance burden, so I think we should go down this road with care.

@PragTob

This comment has been minimized.

Copy link
Member

commented Jul 28, 2018

@devonestes that's a good question! Imo lots of people just prefer the DSL style because it looks nicer and looks more like ExUnit as far as I can see :)

Regarding maintenance one way or another I'd do it as a separate project - with no API completeness guarantees. However, if the "general" options (time, warmup etc.) are implemented in a generic way it could catch quite some stuff. Even if there's a fallback like options or configuration that just takes a key word list or a map. Only if we introduce new per job functionality would it really need adjustment... or so optimistic me hopes.

I can see a beauty of a DSL. But our working with data structures I like - I have one benchmark where I programmaticly build the map. Quite fun.

@hauleth

This comment has been minimized.

Copy link
Author

commented Jul 28, 2018

@devonestes I was thinking about making it separate library though (named Benchamin), however I wanted to discuss the DSL and all pros and cons there as it is IMHO the best place for such.

Why DSL? As @PragTob said, it looks more like ExUnit so it makes it more familiar for the development and allows to use it in similar way the ExUnit is used. So we could have directory containing performance tests and run them in CI for sake of comparing them with the future changes. Check out Rails PerfTest for what I have in mind.

@PragTob this could be written using my proposed syntax via meta programming. However for simple benches like these it would be infeasible to use DSL, DSL is meant for running benches for whole application, save them and track performance changes through time. So use case is a little bit different.

@PragTob

This comment has been minimized.

Copy link
Member

commented Jul 28, 2018

@hauleth I don't understand why one needs a DSL to do these kind of benchmark comparisons. It does the same - it's just a different way to define them. Or am I missing something?

If you're looking for a project to do what you described look no further than elixir-bench which has the same purpose (aka run benchmarks on a CI like infrastructure, save results, compare over time). It still needs some work but we're currently working on it in GSoC (cc: @tallysmartins). Currently it uses the JSON formatter to put the JSON in a specified directory but we'll probably write a custom formatter for it.

And of course it can be written using meta programming, that's just more complicated and not as easy as building up a Map :)

Don't get me wrong, I'm not against a DSL I just want to understand it better. I can totally see how the biggest benefit might just be "It looks more like ExUnit so it's not a thing I run once but all the time and I maintain them more dilligently".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.