Support custom workflow benchmark

[artikell]

This PR implements a custom benchmark method. The ability to generate workload through Lua. Using Lua to generate requests will enable more capabilities:

• Quickly support new commands
• Customizable key distribution patterns, such as [NEW] Enhance valkey-benchmark with Multi-Language Support and Advanced Testing Capabilities #1884
• Support for variable value sizes
• ... ...

It can be considered as a continuation of this issue.

The first step is to write a workload script(workflow.lua)：

-- `request_id` represents the unique identifier of the request
return "LPUSH " .. tostring(request_id) .. " " .. tostring(request_id)

Execute through the following command:

./src/valkey-benchmark --workload ./src/workload.lua

Tip：

• workload mode cannot be combined with other commands

Todo：

• Support generating requests based on return redisReply
• Support workload benchmark in cluster mode
• Support multi thread workload benchmark
• Add workloads for common scenarios

[codecov]

Codecov Report

Attention: Patch coverage is 12.22222% with 79 lines in your changes missing coverage. Please review.

Project coverage is 70.89%. Comparing base (d56a1f7) to head (bf4f2b6).
Report is 3 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/valkey-benchmark.c	12.22%	79 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1898      +/-   ##
============================================
- Coverage     71.06%   70.89%   -0.17%     
============================================
  Files           123      123              
  Lines         65671    65760      +89     
============================================
- Hits          46669    46623      -46     
- Misses        19002    19137     +135     

Files with missing lines	Coverage Δ
src/valkey-benchmark.c	58.49% <12.22%> (-3.57%)	⬇️

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[zuiderkwast]

Cool idea. It's very customizable, but maybe overkill?

For key distribution patterns, it seems more practical to use a normal distribution and add a regular parameter to specify the desired variance or std deviation. If it's specified, then normal distribution is enabled, otherwise the legacy uniform distribution is used.

Key sizes distribution and cluster slot distribution can use a similar distributions and parameters.

We should think about performance too. Lua is quite slow. Valkey-benchmark already has a hard time generating traffic fast enough. It needs to use multiple threads to be able to keep valkey-server busy on a single thread.

[artikell]

@zuiderkwast
Based on the previous discussion, it requires some complex workflow bench (running multiple commands in parallel). From some issues, there are some customized requirements for benchmarks (customized value size, fcall benchmark).

I chose Lua because Valkey itself relies on Lua.

From a performance perspective, on the one hand, this is mainly due to the consumption of generating commands, which may not necessarily result in significant losses compared to network interactions (depending on the complexity of the workflow). On the other hand, it is not strongly dependent on Lua. In the future, you can refer to the module pattern or use LD_PRELOAD to replace generateCommand methods.

At least compared to the benchmarks of workflow implemented in different languages (with even worse performance), it's better for us to unify them.

[arthurkiller]

Interesting idea! I have noticed that the benchmark tool currently lacks scalability, and using Lua is a great approach

[asafpamzn]

@arthurkiller , cool idea.
I think that it is a complementary to #1884 which can also be used to create custom commands using code in the language to their choice.

https://github.com/asafpamzn/valkey-benchmark-node
https://github.com/asafpamzn/valkey-benchmark-java
https://github.com/asafpamzn/valkey-benchmark-go
https://github.com/asafpamzn/valkey-benchmark-python

I think that both should work, and some users will prefer the above and others will prefer to use

./src/valkey-benchmark --workflow ./src/workflow.lua

[zuiderkwast]

Based on the previous discussion, it requires some complex workflow bench (running multiple commands in parallel). From some issues, there are some customized requirements for benchmarks (customized value size, fcall benchmark).

So is the main purpose that users can simulate their own application's workload to design their own benchmarks that are more realistic for their workload?

I chose Lua because Valkey itself relies on Lua.

From a performance perspective, on the one hand, this is mainly due to the consumption of generating commands, which may not necessarily result in significant losses compared to network interactions (depending on the complexity of the workflow).

Yeah, I get it and I think it's a good idea. Currently I need 3-4 threads in valkey-benchmark to keep one singe-threaded valkey-server instance busy (run near 100% CPU) locally with pipelining enabled. You can always use more threads though. How much more CPU does it take to to generate the same load with Lua? If it's not more than twice as much, then I think it's still fine. You can always run valkey-benchmark with more threads.