feat(benchmark): add util to benchmark functions

[kkharji]

Trying to add a way in which we can tests the performance and improve it. I'd like to have this be part of busted run test output somehow. maybe through a global variable 'bench_run', 'bench_compare'

example:

becnh_run("predicting y", {
  warmup = 4, -- number of runs before start calculating
  runs = 100, -- number of runs 
  fun = function()  -- function to test
    vim.wait(400, function() end)
   end
})

output

("predicting y") Benchmark:
  - total elapsed time: 492591167.000ms
  - runs: 100
  - min: 5031917.000s, max: 492584084.000s, mean: 247768180.710s, median: 246534042.000000s, std: 142929375.485669s

not sure how to do compare yet

[kkharji]

This I had for awhile because I want to attach stuff to the test buffer. So maybe just act as if it isn't there :D

[kkharji]

Update

bench("Predicting Y", {
  runs = 10,
  fun = {
    base = function()
      vim.wait(200, function() end)
    end,
    ench = function()
      vim.wait(100, function() end)
    end,
  },
})

[[Predicting Y]] Benchmark:
  - total elapsed time: 1542816208.000ms
  - runs: 5
  - [[ench]] min: 103352833.000s, max: 521638208.000s, mean: 312692466.400s, median: 313483416.000000s, std: 165187575.603357s
  - [[base]] min: 722736083.000s, max: 1542671541.000s, mean: 1132657666.200s, median: 1132462583.000000s, std: 324067377.174429s

[Conni2461]

I can't push (no idea why)

New interface after my changes

bench {
  warmup = 3,
  runs = 50,
  fun = {
    {
      "uv",
      function()
        local fd = assert(vim.loop.fs_open("README.md", "r", 438))
        local stat = assert(vim.loop.fs_fstat(fd))
        local data = assert(vim.loop.fs_read(fd, stat.size, 0))
        assert(vim.loop.fs_close(fd))
        return data
      end,
    },
    {
      "io",
      function()
        local fp = assert(io.open "README.md", "r")
        local contents = fp:read "*all"
        fp:close()
        return contents
      end,
    },
  },
}

patch.diff

[Conni2461]

example output. name is formatted in the benchmark script

[kkharji]

Great work conni, thank you. I'm about to reach home, but don't you agree you should have name of the benchmark as first param to keep consistent with busted style, like it and describe

[Conni2461]

I removed it because i didnt know what to do with it. I mostly stole the format from hyperfine because its a good format. For now i just needed benchmarks between fzf-native, fzy-native and fzf-lua for the telescope wiki. If you wanna add it back. Thats fine too.

Right now i am more concerned with the wiki changes which i wanna get done today

[kkharji]

new output

Benchmark Group: 'IO Vs LIBUV' -----------------------
Benchmark #1: 'uv'
  Time(mean ± σ):    0.0 ms ± 0.0 ms
  Range(min … max):  0.0 ms … 0.1 ms  80 runs
Benchmark #2: 'io'
  Time(mean ± σ):    0.0 ms ± 0.0 ms
  Range(min … max):  0.0 ms … 0.1 ms  80 runs
Summary
  'io' ran
  2.1 ± 1.3 times faster than 'uv'

Interface

bench("IO Vs LIBUV", {
  warmup = 3,
  runs = 80,
  fun = {
    {
      "uv",
      function()
        local fd = assert(vim.loop.fs_open("README.md", "r", 438))
        local stat = assert(vim.loop.fs_fstat(fd))
        local data = assert(vim.loop.fs_read(fd, stat.size, 0))
        assert(vim.loop.fs_close(fd))
        return data
      end,
    },
    {
      "io",
      function()
        local fp = assert(io.open "README.md", "r")
        local contents = fp:read "*all"
        fp:close()
        return contents
      end,
    },
  },
})

[kkharji]

@Conni2461 if it looks good to you, I think we should merge, otherwise tag me

[Conni2461]

Can we also get some simple tests for the stat module. After that i would be fine with merging.

Thanks :)

[kkharji]

Can we also get some simple tests for the stat module. After that i would be fine with merging.

Thanks :)

you mean stat.lua. 🤔 not sure how I might do that or if it's going to be modified anytime in the future since it's pretty static and standard

[Conni2461]

Yeah Thanks :)