help: estimate BigO for multiple functions?

[yitang]

Hi, I'd like to use this package to estimate the BigO for multiple functions. I wonder what's the practical way to implement it. Currently I can get the benchmark statistics for one input size, and I have to manually change the input size and run it again to get a curve for function v.s. input size.

the code is like this

size = 100
x = np.random.randn(size)

def test_f1(benchmark):
    benchmark(f1)

def test_f2(benchmark):
    benchmark(f2)

def test_f3(benchmark):
    benchmark(f3)

Thanks.

[yitang]

I tried to pass the input size from shell command, but py.test doesn't pass it through.

[ionelmc]

Ok, there are two things being conflated here:

• Estimating O(). If you want to compare timings on a gradient of inputs you might want to use parametrization. See: http://pytest.org/latest/parametrize.html?highlight=parametrize

• Getting input from commandline. This is possible by using a plugin (eg: implement one of these http://pytest.org/latest/writing_plugins.html?highlight=hooks#initialization-command-line-and-configuration-hooks). Not sure if that's possible to do from a conftest.py ...

  Alternatively, you can get the configuration for the inputs from env var, not exactly nice but you can still pass it on command line.

[yitang]

I found few scripts which use parametrization in your tests/ folder, and managed to get it work. Thank you very much.

[yitang]

hello again, would you consider adding this type of plots additional to histogram?

[ionelmc]

Currently you can only do it by either using internals from pytest-benchmark (not recommended, internals aren't stable) or reading the json files yourself.

At the very least, I'm open to having some sort of plugin system, so you can extend pytest-benchmark with other kinds of plots.

Having some builtin alternative to the svg histogram needs some discussion first.

Can you explain first what your needs are and what you want to get from these charts?

[yitang]

Good to hear.

I have a couple of algorithms which solve the same problem and of course I want to use the fasted one. So I could do a benchmark using pytest-benchmark, and pick the best one. But some algorithms only works well with small dataset, and I don't want to apply it to the full dataset. To void choosing this type of algorithms, I could vary the input size using parametrization, get a nice curve for each algorithm, and then use it to estimate the run time for the full dataset.

So a nice plots shows the runtime v.s. input size for each algorithms would be really help. The runtime could be min, or median. We could also add error bar (Q3-Q1 or something) to indicate uncertainty.

Y

[ionelmc]

One way I see this is with an overlay of lines like this: http://bl.ocks.org/rkirsling/33a9e350516da54a5d4f

• Each color would be an algorithm
• The X would be the parametrization (assuming that all algorithms take same inputs)
• The Y would be durations.

That would do basically the same thing as the current chart but more compressed. And in addition this would work well with the grouping features (you would group by test function name and each group would get a color).

[ionelmc]

To put this in different words, it would be the same as now, but instead of generating multiple svg files (as it's now with the parametrization) the result would be a single svg but everything in it.

But of course someone needs to invest some time in building this. Are you interested? :)

Regarding the charting library, for context, I picked pygal because it has no dependencies (as opposed to matplotlib of whatever which have very heavy dependencies).

[yitang]

that's exactly what i am after. I'd try to implement it using pygal and see how far I can go.

[ionelmc]

Sure. I guess you can start by plotting min as a line chart (eg: http://www.pygal.org/en/latest/documentation/types/line.html) to first get something working, and after that you could look at how to make pygal plot areas.