Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Improve MockVault performance #740
From what I have seen, the vaults will write each chunk to its own file on disk.
Would it be possible to redesign the MockVault implementation, to be closer to the actual implementation, so that chunks are written to individual files? It wouldn’t be very complicated or invasive it seems to me, and the change would be rather well contained, with few implications if any.
It would give a persistence mechanism that is as closely resembling the actual implementation as possible, and I think would increase the value of the tests.
The advantage of the single MockVault file is that it can be easily shared between machines, but the individual files could for example all be stored in a folder which can easily be zipped up.
Thanks for the submission! It sounds like a reasonable idea to me.
I think this will require benchmarks to make sure we don't introduce a performance regression (e.g. if there are thousands of chunks then there could be thousands of files, which could inadvertently introduce new performance problems if not implemented properly).
Yes, that's true. I was hoping that maybe we could get so close to the real implementation, that if we actually do find performance problems, it could perhaps be valuable insight for the real implementation as well.
The performance problem with single file is mostly about the contention with heavy writing. And as you say, the performance can be negatively affected when reading many files. I'm not all that familiar with how the access pattern will look like in a real vault, and how it might differ from a mock vault. I need to read up more on what's going on there.