New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Store temporary files in memory [$40] #2816
Comments
I'm currently tinkering on this one a bit. Nothing too serious yet but nevertheless I have a question regarding the encryption and decryption. There are multiple places in the code that read data from a temporary file, de-/encrypt the data and write it back to a new temporary file. Notably:
If I want to store the tempory file data in memory, then it seems like the usage of temporary files should be removed here as well. The only hint I found, is a comment (see second link) that mentions something about not being able to differentiate in which part of the stack an exception resides. Why is this an issue and is this even still relevant? |
The reason for not using a stream, is that I have stumbled on different cases where this fails. It is my understanding that there is something in the .Net Crypto API (padding?) that does not work with non-seekable streams. The part with the exception is a temporary fix to do some encryption in parallel. For the |
No reason we can't have a fully seekable in-memory stream |
There's now a bounty of $40 |
Most of the changes originally mentioned are implemented by now. Proof of concept implementation: Jojo-1000:mem-temporary-files
Missing:
The speed is not significantly higher compared to NVME SSD (probably due to RAM caching on the SSD). After discussion with @gpatel-fr, it was decided to postpone work on this for now. |
As the temporary files are usually small (say 50-200MB) they can easily fit into memory on most machines (small ARM devices like RPI or Synology excluded).
Experiments with setting a RAM-disk as the temporary folder shows significant speedup and will reduce write-wear on SSD disks.
This issue suggests that the temporary files for
dblock
,dindex
anddlist
are stored in memory, if the--volume-size
is less than 10% of the memory. This should of course be configurable, for users who want to control what happens.This change requires that the
ICompressionModule
interface is changed slightly such that it accepts aSystem.IO.Stream
as input instead of the current path-string
.The
BackendManager
needs to be changed a bit as it relies on the files being present. and it should also support streams.An optional, and more extensive solution, would be to replace calls to
System.IO.File.Open
that work on temporary files, such that a special kind of path, like:tempfile://xyz
can be detected, and redirect to some internal storage dictionary with temporary files being mapped to (memory-)streams.The text was updated successfully, but these errors were encountered: