New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trying to evaluate the performance for dm-writeboost #194

Closed
allenjhan opened this Issue Jun 8, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@allenjhan

allenjhan commented Jun 8, 2018

Hi,

I am trying to evaluate the performance of dm-writeboost for a school project. Basically, I am trying to show the performance of dm-writeboost is better (more speed) when the size of the cache increases. I am manually setting the wb->nr_segments variable to do this.

I have noticed that the speed of the cache slows down when urge_writeback becomes true. (That is what I think is happening.) I have nr_batch set to 4, which may be why it is slow. When urge_writeback becomes true, dm-writeboost becomes slower than the backing device. Therefore, I have tried to evaluate dm-writeboost so that the entire workload fits in the cache device. That way, the cache does not need for the writeback daemon to finish writeback in the middle of the workload.

My test setup is as follows. I use Postmark, but modified so that the open() system call is used with the O_DIRECT and O_SYNC option. I choose the buffering setting to be false. I also deactivated the write cache of my hard drive with the hdparm -W 0 option. This guarantees that the only cache being evaluated is dm-writeboost. There should be no interference from another cache. I first allow the system to run a workload for about five minutes, which fills the cache device. This gives the best chance for hits when I run another workload. I then make the cache writeback all dirty caches to the backing device with drop_caches, to make sure writeback will not interrupt the next workload. I then run a smaller workload that fits completely in the cache device and observe the speed and hit rate.

I have two questions:

  1. I have a strange result I cannot explain. dm-writeboost speeds up the backing device by about 15%. However, it reaches 15% speed increase at cache size that gives 20% hit rate. From 20% hit rate to 100% hit rate, there is no further improvement in the speed of dm-writeboost. This is strange since when the cache has a high hit rate, the backing device is not accessed, and so dm-writeboost should run at the speed of the cache device. There should be tremendous speed improvement when the backing device is not accessed. For example, even when I observe 100% hit rate, I only see the 15% speed improvement.

  2. I have used O_DIRECT | OSYNC to turn off kernel caching and used hdparm to turn off the hard drive's write cache. However, the speed of the hard drive (without dm-writeboost) is still around 30 MB/s. If I include the -o sync option during mount, the speed of the HDD drops to 10 KB/s. That is a huge difference. Is there another cache in the system I am not aware of? Besides the hard drive write cache and kernel caching, is there another source for caching?

I understand if you are busy and do not have time to consider my question for very long. If you even read this far, I am already grateful.

@allenjhan

This comment has been minimized.

allenjhan commented Jun 15, 2018

I have read some discussions online about O_DIRECT, and I am not sure what is happening. After more measurements, I am seeing that dm-writeboost has no effect on the inherent speed of the backing device when O_DIRECT is used. After running for a few minutes at the speed of the backing device, the cache slows down, just like it would if the writeback daemon turns on when urge_writeback is true. The cache statistics are updated, just like it would be if the cache worked normally.

Discussion online seems to indicate that O_DIRECT bypasses the kernel and causes IO to happen via DMA. If true, that would explain why dm-writeboost has no change with increasing cache size: the cache is not being used. However, this is contradicted by the fact that the cache statistics are being updated normally. I am still not sure how to interpret my results, but my solution is to stop using the O_DIRECT flag for my measurements. I will share the results of my measurements if they make sense.

@akiradeveloper

This comment has been minimized.

Owner

akiradeveloper commented Jun 16, 2018

@allenjhan Thank you for evaluating Writeboost. But, this is not a school for you. I will close this issue.

One comment: Your understanding of O_DIRECT and how Writeboost works when it's enabled is wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment