CacheBench: changing number of keys for provided test configs

[kevins981]

Hi CacheLib developers,

In the provided cachebench configurations such as this one (thank you for making them open source!), if there is a specified popDistFile, would I still be able to modify the numKeys field?

For example, the original hit_ratio CDN config liked above has numKeys == 8935378. If I double the number of keys and use the same popDistFile, is this still a valid benchmark?

Some additional info:
I ran a quick experiment by doing exactly that and ran ./opt/cachelib/bin/cachebench --json_test_config cachelib/cachebench/test_configs/ecosys/cdn/config.json
It seems that only keys from the lower half of the 8935378*2 keys are being accessed. Is this the expected behavior?

There is a scalingFactor_ variable here, but I am not sure if it was intended to adjust for this difference in the number of keys.

Thank you,
Kevin

[therealgymmy]

@kevins981: sorry for the late response. Yes you can adjust numKeys. This is often used to scale the workload up or down.

For example, the original hit_ratio CDN config liked above has numKeys == 8935378. If I double the number of keys and use the same popDistFile, is this still a valid benchmark?

Yes it is still valid. You will need to size your cache proportionally larger if you want to get the same result we evaluated internally. But if you wanted to see the effect of a larger working-set on the same cache size, scaling up the key count is one way to achieve that.

[kevins981]

Hi CacheLib developers,

I am reopening this thread because I am still unsure about CacheBench's behavior when using popDistFile with an increased numKeys.

@therealgymmy mentioned above that we are allowed to increase numKeys using the same popDistFile file. I did the following to test this:

I am using cachelib/cachebench/test_configs/hit_ratio/cdn/config.json as an example. The original config had numKeys == 8935378 and I increased it to 80 million. For context, we wanted to do this in order to scale the workload up (increasing cache size, numKeys etc).

For the Workload Generator (which is used by default here), my understanding is that in getReq(), the idx generated should cover than entire range of the key space (from 0 to numKeys, based on the popularity distribution). Thus, if we increase numKeys, say from 8935378 to 80 million, I expect to see many idx larger than 8935378.

I went ahead and printed out all idxs generated in getReq(). However, CacheBench seems to be only generating idxs smaller than 8935378. Is this the expected behavior? Does this mean CacheBench is in fact only accessing the first 8935378 items instead of 80 million?

I am using the latest CacheLib main branch (as of March 12 2023). The only changes I made to cachelib/cachebench/test_configs/hit_ratio/cdn/config.json are increasing numKeys and using numThreads = 1 (so I can see the prints properly).
The command I used was ./opt/cachelib/bin/cachebench --json_test_config ./cachelib/cachebench/test_configs/hit_ratio/cdn/config.json --progress=5 > popdist_test

Please correct me if any of my understanding is wrong. Any insights would be appreciated!

Thank you for your time,
Kevin

[jaesoo-fb]

Hi @kevins981, thanks for reporting the bug; I can confirm that key popularity scaling is not being supported.

Basically, there is something needs to be done to support the scaling in FastDiscreteDistribution. The problem is that we are not considering the key scaling factor, i.e., size_total / sum(sizes), when user wants to scale the total number objects to size_total from sum(sizes). We will put it in our backlog.

[kevins981]

Thank you!

[jaesoo-fb]

This has been fixed in c90dd1d (WorkloadGenerator: fixed a bug in scaling key popularity distribution)