Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
bonnie++ meets LizardFS: drastic I/O error #662
LizardFS is under evaluation on my machine. I have setup some VirtualBox VMs with the same settings:
Three chunkserver VMs are running. Each one has one dedicated 8 GiB hard drive
When I invoke bonnie++ on a folder of the mounted LizardFS, bonnie++ fails:
See attached screenshots for more details!
Do you have any idea what went wrong exactly? Are able to reproduce it?
Thank in advance again!
Um, bonnie++ met LizardFS a long while ago and always worked fine. I just ran it on your setup though, and indeed I saw this
Still, can you try changing some mount options related to metadata caching and see if your problem disappears? By some options I mean these:
Hey. I reproduced your problem and also made some interesting observations about the bug bonnie++ stumbled upon in LizardFS.
As @onlyjob previously mentioned mfsmount parameters for disabling cache do not make any difference.
The "drastic I/O" error arises in the
and it's not present in the
which does not use file listing via
It means that Lizard's
Thus I think the implementation of Lizardfs's
An interesting insight here is that this
I'll keep you updated on progress in this issue.
About those 8167 files left in the test directory after benchmark from the third @greensquirrel's screenshot:
@onlyjob what commands did you use to delete the files?
I can confirm the problems with directory deleting using
"Large enough" technically: when the number of directory entries (
I haven't yet managed to reproduce this error using
I have, however, found the part of code causing this trouble. The problem lies in the master, in iteration by index over direntries when interleaving
Thus each subsequent
This bug is not connected to the
TL;DR: I found the cause of this bug ;)
I started working on a fix. I'll post here on updates.
Update on the issue
I have already come up with a couple of possible solutions and have planned to test and compare them "in the real code" to choose one offering best performance and(/or) smallest memory consumption - both of which are important for code run on a master machine.
It may take a week or two before we push the fix into master (git branch) as we want to make sure it doesn't lower master and client performance more than is necessary for them to work correctly and doesn't break anything else.
Short, much less technical issue description:
The problem with "holey" dirent iteration stems from the lack of a true dirent index on master. This index is currently created artificially (in the easiest way possible, which is fine) and becomes inconsistent when the dirent container is modified during iteration over it. At this point, the dirent indices on master and client stop matching and we get holes in iteration.