Add fio tests to the dm-snapshot test suite #52
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Increase the buffer size used by blktrace for event extraction to eliminate dropped events for high IOPS devices like null_blk. Signed-off-by: Nikos Tsironis <ntsironis@arrikto.com>
Until now we ran blkparse using backticks. This method stores the whole output of blkparse in memory before we get to process it. This is inefficient both memory and performance-wise. Run blkparse with IO.popen so we can process its output incrementally through a pipe. In a test, where we processed 4.5G of blktrace traces, this change reduced Ruby's memory usage from 12G to 3.7G and the test's run time from 10m19.409s to 5m31.387s. Signed-off-by: Nikos Tsironis <ntsironis@arrikto.com>
blkparse() parses the whole blktrace output and generates an events array with all the relevant events. For large traces this can result in heavy memory consumption. Moreover, taking into account how Ruby's GC works, calling blkparse() many times in a row can result in fork() failing with ENOMEM, because the memory consumed by the events array is never released back to the system. Also, there are use cases, e.g. the IOHistogram class, where it's more efficient to process the blkparse events incrementally rather than storing them all in memory and then iterating over the events array. Allow for such an incremental processing by introducing new helper functions and use these functions in IOHistogram. In a test, where we processed 4.5G of blktrace traces, this change reduced Ruby's memory usage from 3.7G to 26M and the test's run time from 5m31.387s to 4m17.346s. Signed-off-by: Nikos Tsironis <ntsironis@arrikto.com>
We are only interested in either Queue or IO completion events so there is no need to capture all possible events with blktrace. Filter the events captured by blktrace using an appropriate filter. This helps reduce the size of blktrace's output and therefore decrease its processing time with blkparse. Signed-off-by: Nikos Tsironis <ntsironis@arrikto.com>
Add class whichs records the per CPU IO distribution, i.e., records how many reads/writes each CPU has queued to the underlying device. Signed-off-by: Nikos Tsironis <ntsironis@arrikto.com>
Add various fio tests that: * Measure random read/write throughput, as the number of worker processes increases. * Measure random read/write latency, as the number of worker processes increases. * Measure sequential read/write throughput, as the number of worker processes increases. * Trace the per CPU IO distribution for random and sequential writes, as the number of worker processes increases. Signed-off-by: Nikos Tsironis <ntsironis@arrikto.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello,
This PR adds various fio tests to the dm-snapshot test suite. It also includes some fixes/improvements for the blktrace module.