Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve MockVault performance #740

Open
oetyng opened this Issue Feb 12, 2019 · 3 comments

Comments

Projects
None yet
2 participants
@oetyng
Copy link

oetyng commented Feb 12, 2019

From what I have seen, the vaults will write each chunk to its own file on disk.
MockVault however, writes to a single file. This is impacting IO performance significantly, and the testing utility for it when storing anything above a couple of thousand stored values, diminishes.

Would it be possible to redesign the MockVault implementation, to be closer to the actual implementation, so that chunks are written to individual files? It wouldn’t be very complicated or invasive it seems to me, and the change would be rather well contained, with few implications if any.

It would give a persistence mechanism that is as closely resembling the actual implementation as possible, and I think would increase the value of the tests.

The advantage of the single MockVault file is that it can be easily shared between machines, but the individual files could for example all be stored in a folder which can easily be zipped up.

@m-cat m-cat added help wanted question and removed question labels Feb 12, 2019

@m-cat

This comment has been minimized.

Copy link
Contributor

m-cat commented Feb 12, 2019

Thanks for the submission! It sounds like a reasonable idea to me.

I think this will require benchmarks to make sure we don't introduce a performance regression (e.g. if there are thousands of chunks then there could be thousands of files, which could inadvertently introduce new performance problems if not implemented properly).

@oetyng

This comment has been minimized.

Copy link
Author

oetyng commented Feb 12, 2019

Yes, that's true. I was hoping that maybe we could get so close to the real implementation, that if we actually do find performance problems, it could perhaps be valuable insight for the real implementation as well.

The performance problem with single file is mostly about the contention with heavy writing. And as you say, the performance can be negatively affected when reading many files. I'm not all that familiar with how the access pattern will look like in a real vault, and how it might differ from a mock vault. I need to read up more on what's going on there.

@m-cat

This comment has been minimized.

Copy link
Contributor

m-cat commented Feb 13, 2019

Yep, agreed. I meant to mention above that I think more benchmarks would be really useful to have anyway, especially since we don't have too many in Safe Client Libs yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.