-
Notifications
You must be signed in to change notification settings - Fork 70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Debug self-encryption of small files #385
Comments
@RolandSherwin how exactly do you envision this to work, when you say small files what's the range of sizes you actually mean here? is it between 3 * MIN_CHUNK_SIZE and 3 * MAX_CHUNK_SIZE ? if so, for a file of size let's say 2 * MAX_CHUNK_SIZE we'll need to generate datamap + 2 chunks, right? and that'd be a different mechanism as SE needs at least three chunks to work... |
I don't know if this is a good place to raise this, but it would be nice to have an API for self encrypting a block of memory. Currently it only allows self encryption of a file on disk and its not good to force data to be written to disk just for this purpose. |
Hey @bochaco. Right, I see! Also, we were having troubles when actually trying to upload those extra chunks. We fetched store cost + paid + sent over chunk+payment, but they were never accepted. |
These may be two separate bugs. Firstly uploading individual small files does work (ie But uploading a directory (ie |
After digging in, the issue with small files ending up in 1 chunk (plus the data-map chunk) is due to the fact that files that are smaller than 3 * MAX_CHUNK_SIZE are split into 3 chunks, thus if the content of the file is by chance something like "hello\nhello\nhello\n" it ends up being split into three equal chunks: "hello\n". |
@bochaco That sounds like the issue riddim isolated as he was just adding the same text to a file to see what happened to the upload cost. It doesn't explain the issue I was experiencing and reported to Qi Ma. Is anyone looking into that? |
Closing this because the behaviour is as intended. |
Currently, trying to call self encryption on a small file leads to undesirable results.
We should just have 1 chunk + data map for a small file, instead we get 3 chunks. + data map.
It is from this function
get_num_chunks
inside self encryption.Also, earlier we had a method to encrypt small files separately, but now it is removed, maybe that is the cause? Anyways, move that functionality into self-encryption crate.
Also add logging to self encryption.
The text was updated successfully, but these errors were encountered: