Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store temporary files in memory [$40] #2816

Open
kenkendk opened this issue Oct 14, 2017 · 5 comments
Open

Store temporary files in memory [$40] #2816

kenkendk opened this issue Oct 14, 2017 · 5 comments

Comments

@kenkendk
Copy link
Member

As the temporary files are usually small (say 50-200MB) they can easily fit into memory on most machines (small ARM devices like RPI or Synology excluded).

Experiments with setting a RAM-disk as the temporary folder shows significant speedup and will reduce write-wear on SSD disks.

This issue suggests that the temporary files for dblock, dindex and dlist are stored in memory, if the --volume-size is less than 10% of the memory. This should of course be configurable, for users who want to control what happens.

This change requires that the ICompressionModule interface is changed slightly such that it accepts a System.IO.Stream as input instead of the current path-string.

The BackendManager needs to be changed a bit as it relies on the files being present. and it should also support streams.

An optional, and more extensive solution, would be to replace calls to System.IO.File.Open that work on temporary files, such that a special kind of path, like: tempfile://xyz can be detected, and redirect to some internal storage dictionary with temporary files being mapped to (memory-)streams.

@TekkiWuff
Copy link
Contributor

@kenkendk

I'm currently tinkering on this one a bit. Nothing too serious yet but nevertheless I have a question regarding the encryption and decryption.

There are multiple places in the code that read data from a temporary file, de-/encrypt the data and write it back to a new temporary file. Notably:

If I want to store the tempory file data in memory, then it seems like the usage of temporary files should be removed here as well.
Now I wonder why the data is not simply passed through the encryption layer via Streams. The IEncryption interface seems to have the nessecary Stream Encrypt(Stream input) and Stream Decrypt(Stream input) methods that both the AESEncryption and GPGEncryption classes implement with actual code.

The only hint I found, is a comment (see second link) that mentions something about not being able to differentiate in which part of the stack an exception resides. Why is this an issue and is this even still relevant?

@kenkendk
Copy link
Member Author

kenkendk commented Nov 3, 2017

The reason for not using a stream, is that I have stumbled on different cases where this fails. It is my understanding that there is something in the .Net Crypto API (padding?) that does not work with non-seekable streams.

The part with the exception is a temporary fix to do some encryption in parallel. For the concurrent_processing branch which I am working on, the compressed volume is just encrypted by itself. This code will go away, so no need to handle that part.

@BMurri
Copy link

BMurri commented Mar 12, 2019

No reason we can't have a fully seekable in-memory stream

@MrElliwood
Copy link

There's now a bounty of $40

@warwickmm warwickmm changed the title Store temporary files in memory Store temporary files in memory [$40] Nov 28, 2021
@Jojo-1000
Copy link
Contributor

Most of the changes originally mentioned are implemented by now.

Proof of concept implementation: Jojo-1000:mem-temporary-files

  • Interface to allow temporary files on disk or in memory
  • Some backends require filenames (rsync), convert to actual file by writing to disk in that case or if --disable-streaming-transfers is set

Missing:

  • Detection of available memory
  • Spillover to disk if memory becomes too full
  • Configurable threshold
  • Check whether the backend allows it when creating the temp file instead of when reading

The speed is not significantly higher compared to NVME SSD (probably due to RAM caching on the SSD). After discussion with @gpatel-fr, it was decided to postpone work on this for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants