-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
When I use numjobs=8, the perfoemance is lower then nubnerjob=1 #1046
Comments
I think it's unreasonable, but you should provide more details, please read how to report bugs |
Thanks~ |
I would like to echo @Martins3 comment about need for the minimum information. Taking a glance I will note that you have a high depth of 128 for one job alone and cranking up numjobs means you are effectively asking for a depth of 1024. If your backend doesn't actually have a "depth" that deep then all you are doing is creating contention for the depth it can service and the more contention there is the worse the speeds you achieve will be as latency goes up. |
Let's make the next action clearer: @ZhenLiu94 what is the output of
|
My rbd0 is a cloud drive, so I don't have /sys/block/rbd0/device/ on my system, but I can cd /sys/block/rbd0, then use "ls", The output information is
I thought a lot last night, Here's my idea: |
@ZhenLiu94 Depending on your machine it could be threads but it could just as well be in-flight I/Os. I would be more likely to suspect the later. There will be some maximum number of I/Os can be in-flight and then there will be some maximum number of I/Os that the kernel is then willing to queue up internally. Doing a Google search (https://www.google.com/search?q=linux+rbd+queue_depth ) leads you to https://docs.ceph.com/docs/mimic/man/8/rbd/ which mentions about a default queue depth of 128 which feels like a hint :-). Did you check what happened if you used just two jobs and then 4? Did you check what iostat said the queue depth of your rbd device was while fio was running with all these permutations? |
I will also note my greps above were wrong and should have been:
obviously you don't have a device in your case but queue should be there... |
@ZhenLiu94 any follow up on this one? |
There is going to be some maximum number of I/O that can be kept in flight by your hardware at any given time. If we assume that the I/Os reaching the disk are a uniform size then once you have hit that limit future I/Os are going to have to sit and wait in a queue before they can be serviced. The deeper that queue becomes the higher the latency for I/Os (because the latency now has to include the time it takes to get through that queue). Thus you can hit a throughput plateaux but see latency steadily rise. BtFS is a copy on write filesystem and this workload will likely cause rapid fragmentation (a google search for btrfs fragmentation quickly got me to https://btrfs.wiki.kernel.org/index.php/Gotchas#Fragmentation ) so I don't think that result is unexpected either. Having said all that I suspect there was more in fio job which you didn't post (I notice the chart mentions However from what's been posted I can't see something that looks like a bug in fio and the questions seem more of the form "How do I/Why is?" (and this has now attracted two different setups so it's becoming more unfocussed). Such questions are better aimed at the fio mailing list (note that the list only accepts plain text emails). I strongly recommend continuing any further discussion there so I'll close the issue here. |
When I use fio testing my ceph cluster, Some results are not as expected. For example, the command I use is
fio --filename=/dev/rbd0 -iodepth=128 -rw=read -ioengine=libaio -bs=4k -size=100G -numjobs=8 -group_reporting -direct=1 -name=iops_write -runtime=600
but the cluster's iops result is lower then numjobs=1. So ,I want to ask when I use fio testing a ceph cluster, how should I set my fio's parameters?
thx~
The text was updated successfully, but these errors were encountered: