pbench-fio: needs a --cache-drop feature

[bengland2]

For distributed storage, there has to be a way for pbench-fio to drop cache on storage servers prior to a performance test. Otherwise read performance data will not be valid.

Part of the problem is that pbench-fio is intended to run multiple test types in sequence with the -t option. That's great functionality, but it means that there is no opportunity for the user to drop cache on storage servers prior to running a read test. For example, if the pbench-fio user specifies -t write,read (a common and reasonable test sequence), the cache has to be dropped after the write but before the read.

A possible solution to this is to add a "--cache-drop-cmd" option that specifies a call-out to an external script. For example:

 --cache-drop-cmd='ansible -i ceph-hosts -m shell -a /usr/local/bin/drop-cache.sh osds'

This does not require pbench to know anything about the underlying storage architecture - all that is handled in the call-out command. But it is vital for Ceph, Gluster, NFS, SMB, and iSCSI performance testing for example.

If there are no objections to this proposal I'll submit a pull request.

Unless there are objections to this

[ndokos]

How do I specify where exactly in the control flow the command is going to be executed? Is pbench-fio supposed to know that the next test is going to be a read test, which will then trigger the command?

[bengland2]

Great question. Since fio has a finite set of test types which are predetermined to be read,write,rw,randread,randwrite,randrw, then it should be possible to tell which tests involve reading data. So we could do it that way.

But on the other hand there can be cases where you want to drop cache before a randwrite test - for example, to prevent Ceph from already having cached metadata about RADOS objects on disk - of course it will load and cache that data after the test starts, but at least you have deterministic behavior - the cache state is not dependent on history of the system.

So IMHO after further thought, we should just do it for every test type and make life simple and make results repeatable. If the user wants to test cache behavior, they can either a) run a long enough test that caching comes into play, or b) don't provide the --cache-drop option in the command.

opinions?

[portante]

@bengland2, can you submit the pull request to see how this will play out? I am inclined to think that just dropping cache between all iterations would be fine for starters.

[ndokos]

PR #386 was merged.