New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with large files #82

Closed
Kuurusch opened this Issue Sep 9, 2015 · 34 comments

Comments

Projects
None yet
@Kuurusch
Copy link

Kuurusch commented Sep 9, 2015

I have a NAS and try to create encrypted folders on the NAS. So, small files can I store, but when I try to store large files around 4GB, it allways interrupts somewhere in the middle. And that with different files. It says, it can't read the file. On Mac- and Windows-machines the same behaviour!

@tobihagemann

This comment has been minimized.

Copy link
Member

tobihagemann commented Sep 9, 2015

Thanks for reporting! #72 could be related. Seems like that this can't be fixed on Windows. We'll look into this regarding Mac!

@Kuurusch

This comment has been minimized.

Copy link

Kuurusch commented Sep 9, 2015

Hm, but there tey are talking about WebDav. But I don't use WebDav. It's on a normal mounted Volume over the LAN. It seems more that maybe the connection over the LAN is less stable than an intern connection. As soon there are some delays or so, it interrupts. Because its not everytime on the same point in the file. After the error the volume is also somehow dismounted in the system, even when it shows mounted in the cryptomator itself.

@tobihagemann

This comment has been minimized.

Copy link
Member

tobihagemann commented Sep 9, 2015

Oh, I should've explained that. Cryptomator is using the WebDAV protocol to mount a "virtual" client-side drive. So the unlocked Cryptomator vault you're seeing is using WebDAV. Therefore Cryptomator is dependent on the WebDAV client implementation of the respective operating system. We decided using WebDAV, because every major operating system supports WebDAV.

As explained in the related issue, this could lead to a problem on Windows with files larger than 4 GB (and we still have to check this on Mac). We'll stick to WebDAV for now, because we wanted a multi-platform approach from the get-go, but this could be a subject to change in the long run.

Edit: I have to check with @totalvoidness when he's back from vacation. We'll also check the error-case so that the vault doesn't dismount unexpectedly.

@Kuurusch

This comment has been minimized.

Copy link

Kuurusch commented Sep 10, 2015

Aha, okdoky. Then I think the problem lies definitly there ;) So maybe it would be good to implement differen protokolls for different platforms. SMB for mac usw.

@jaorueta

This comment has been minimized.

Copy link

jaorueta commented Nov 3, 2015

Try to split the file before copy to the vault

@tobihagemann

This comment has been minimized.

Copy link
Member

tobihagemann commented Nov 3, 2015

Could be a possibility, but splitting files could lead to synchronization issues with the cloud storage service, and therefore more likely lead to data loss. Because of that I don't think splitting is a viable option. Thanks for your suggestion though!

@gstkenpo

This comment has been minimized.

Copy link

gstkenpo commented Dec 8, 2015

Ha Ha, same problem here. I do mount my volume on local SD card. But the problem persist. Finally it shows error message "The Finder can't complete the operation because some data in 'xxx' can't be read or written. (Error code 36)"

Any update for this issue? Thanks

@overheadhunter

This comment has been minimized.

Copy link
Member

overheadhunter commented Dec 8, 2015

@gstkenpo Well the 4 GiB limit is related to the target file system, of course. This is something the user is used to from unencrypted volumes, too. WebDAV doesn't have this limit (I've tried it with 8 GiB files). This isn't high priority, as Cryptomator doesn't cause the problem but rather propagates it from the physical file system to the virtual one.

@gstkenpo

This comment has been minimized.

Copy link

gstkenpo commented Dec 8, 2015

@totalvoidness thanks for your reply. But my file size is just 1.9GB.mov file. It shd be within FAT32 limitation.

@overheadhunter

This comment has been minimized.

Copy link
Member

overheadhunter commented Dec 8, 2015

@gstkenpo please describe your setup, I probably didn't understand you correctly. You have a FAT32 drive on your NAS, which is mounted via (what protocol?) on your Mac and your vault resides inside this mounted drive? From your other issue I understand that the "error code 36" problem was caused by some ._ files. How does this relate to the NAS problem?

@hotisnottheword

This comment has been minimized.

Copy link

hotisnottheword commented Feb 2, 2016

I also have a problem with large files. My PC is running Windows 10 64bit all file systems are NTFS which supports large files no problem. If encrypt a 7.64GB video file it appears to complete but the file size is in the virtual drive its only 80MB.

@markuskreusch markuskreusch added this to the 1.x milestone Feb 26, 2016

@germanstudent

This comment has been minimized.

Copy link

germanstudent commented Jun 10, 2016

I tried to use Cryptomator for a bigger archive today and I noticed the total used disk space was off, after the copy/encryption process finished.

The resulting, copied file was way smaller than the original, hence corrupted. Cryptomator mounts the vault as FAT by default, though the host file system uses NTFS, which might be a problem.

1
Host file system: NTFS, Cryptomator mount: FAT

2
Copy process looks like the right amount of data is transferred.

3
File size is off after the copy process finishes, without an error message triggered

I think this kind of a big issue, because no error message pops up and data could potentially be lost, if the file size difference keeps unnoticed by the user.

System: Windows 10 Pro (64 Bit), Cryptomator 1.1.0

@overheadhunter

This comment has been minimized.

Copy link
Member

overheadhunter commented Jun 11, 2016

This is indeed a severe issue, however this needs to be reported to Microsoft. Cryptomator doesn't limit the file size, but the WebDAV client under Windows has a 4GiB limit. Windows knows this, but doesn't warn the user. This is a problem.

We could do something like if windows and file size > 4GiB then fail, however this would even prevent 3rd party WebDAV clients, that DO support large files... Another option would be to add a servlet filter, that fails with error 413 when the user agent is Microsoft-WebDAV-MiniRedir and the file size is too large for Windows to handle correctly. However this doesn't work with chunked requests...

@sergeevabc

This comment has been minimized.

Copy link

sergeevabc commented Nov 8, 2016

@overheadhunter, let me express it from an average user perspective:

  • well-known EncFS(MP) is also multi-platform, however does not use WebDav
  • EncFS possible drawbacks are not related to how files are accessed
  • so there are other ways to implement I/O without fight with Microsoft over 4GiB limit of WebDav
@tobihagemann

This comment has been minimized.

Copy link
Member

tobihagemann commented Nov 8, 2016

That's why we're planning to move on to FUSE #252 and Dokany #207. 😁 However, it doesn't change the fact that Microsoft messed up their WebDAV client in Windows. 😉

Edit: Ah, interesting. EncFSMP uses PFM, has also been mentioned in #315. But we're still more likely to implement FUSE/Dokany.

@sergeevabc

This comment has been minimized.

Copy link

sergeevabc commented Nov 8, 2016

@tobihagemann, +1 for smooth and solid PFM (Pismo File Mount) or WinFSP (FUSE for Windows).

@zhupgh

This comment has been minimized.

Copy link

zhupgh commented Dec 22, 2016

Hello,
When you are going to solve the problem of large files in Windows?

@pilavdzic

This comment has been minimized.

Copy link

pilavdzic commented Jan 5, 2017

Some cloud storage providers won't store files larger than 2GB for the same reason. But why not have cryptomator split files into 2gb chunks if they are larger before encrypting and then append them back together after decrypting? I think this might be fairly easy to do - the built in windows copy command for example can append files back together.

Waiting for MS to fix webdav won't ever happen. We should be working around webdav limitations not be constrained by them. Webdav has a lot of bugs by the way - did you test other languages and character sets? Some programs have files that are names really edge-case things.

@tobihagemann

This comment has been minimized.

Copy link
Member

tobihagemann commented Jan 5, 2017

Not sure if you followed the whole thread, but our plan is to replace WebDAV with Dokany #207 in the future. And: We won't split files as explained in #299.

@pilavdzic

This comment has been minimized.

Copy link

pilavdzic commented Jan 5, 2017

Thanks, yes I missed your earlier comment. That sounds like a good plan!

I can't say I understand what you mean in #299 though, by 'kill' did you just mean it will be slow or?

Boxcryptor or another similar software does do something like that. If this were my project I'd simply add a layer that compresses, splits then decompresses/reconsitutes after transparently to the cloud provider layer so it can't tell that it wasn't always multiple files.

I could be wrong, I just started using this product today and don't know enough but this is the first thing I ran into. So, right now I manually 7zip the files into chunks then place them into my cryptovault.

It would just be nice if this was automated as one of the layers built into the app instead of me having to do it separately.

@tobihagemann

This comment has been minimized.

Copy link
Member

tobihagemann commented Jan 5, 2017

Let's say we would split a file in two or more files and they'd be eventually synchronized (uploaded one by one!). Keep in mind that we don't have any control over synchronization. Think of this edge case: Multiple people work on the same file (e.g., because you share a vault with your colleague) and now you have multiple uploads at the same time. This actually doesn't even have to be such an extreme case. Synchronization conflicts can also happen if you have multiple devices and work with synchronization clients that are doing an awful job at keeping things in sync.

In that way you could end up with a splitted file that has different origins. Let's say we have file.part1 and file.part2. There is a possibility(!) that file.part1 ends up in "another version" than file.part2. Heck, why don't we just throw in file.part1 (1) and file.part2 (1) in there, because they just popped up as synchronization conflicts! Have fun putting everything together. 😂

I'm exaggerating, but hopefully you get the idea why there is a small possibility that could destroy(!) your data or make it unreadable. This would be fatal and that's why we prefer atomic operations. This is only guaranteed by keeping the file "as a whole".

I don't really know how Boxcryptor (do they?) or similar software are dealing with this sort of problem. Maybe they don't deal with it at all or they don't have to deal with it. Either because they think that the possibility is extremely low or because they have full control of the synchronization.

Edit: Okay, I'd like to point out that there certainly are technically sophisticated ways to prevent these sync conflict issues I've mentioned. But they would create a conflict with the user story of #236 that we wanted to solve with #336. There's just a whole lot to think about, we can't just "split" files without negative effects.

@pilavdzic

This comment has been minimized.

Copy link

pilavdzic commented Jan 12, 2017

@sergeevabc

This comment has been minimized.

Copy link

sergeevabc commented Jan 12, 2017

In sum, Cryptomator is too raw to use in production, since it has problems with integral part of daily routine, the large files. Workaround is to use EncFSMP, EncFS4Win, or VeraCrypt. And set a e-mail reminder to come back here one day.

to:      nudge@nudgemail.com
subject: 01/01/2018

How is Cryptomator doing about large files?
Changelog is here https://github.com/cryptomator/cryptomator/releases
@devarni

This comment has been minimized.

Copy link

devarni commented Jan 24, 2017

Same problem here....
1.) Not the correct disk size is used, the encrypted files are on a 1TB disk but after mounted with WebDav the available space on the netdrive is displayed with the same space as the system partition.
2.) Large files are not supported, so it's unusable as a secure backup solution. I've tried to copy a 10GB file which doesn't work.

Sorry guys, but this is a mistake by design. If WebDav has such known limitation you cannot use WebDav and you should put a large note on the website about the limitations.

@sveeke

This comment has been minimized.

Copy link

sveeke commented Mar 29, 2017

I actually agree with @devarni. There should be a large warning when the Windows version is downloaded that this limitation exists. I advised a lot of people to use Cryptomator, but more and more of them run in to this problem now.

I find it remarkable that there is still no entry of this issue here: https://cryptomator.freshdesk.com/support/solutions/folders/16000030172 and when you download the app.

Well, back to VeraCrypt it is. The concept of Cryptomator is rather nice (especially its simplicity), but there are too many issues and problems with it to consider seriously as an alternative to the more robust options around. What might help is an advice to use another open source WebDav client instead of the crappy Windows implementation, but that would negate a lot of the user friendlyness of the current Cryptomator client. On the other hand, it would not be broken :).

@ibleedbinari

This comment has been minimized.

Copy link

ibleedbinari commented Dec 9, 2017

A temporary, unsupported workaround is to mount the Cryptomator vault using Cyberduck/Mountain Duck (either directly using the integration, or by mounting the WebDAV URL provided by Cryptomator). Using this method mounts the folder as NTFS, therefore being able to handle the large files. However, be warned that Cyberduck doesn't properly track date modified metadata in all situations as of now (https://trac.cyberduck.io/ticket/10171) so it cannot be used with a folder syncing program that relies on the date modified field (such as rsync or FreeFileSync).

@overheadhunter overheadhunter modified the milestones: 1.x, 1.4.0 Jun 20, 2018

@overheadhunter

This comment has been minimized.

Copy link
Member

overheadhunter commented Jun 20, 2018

Depends on #207

@overheadhunter

This comment has been minimized.

Copy link
Member

overheadhunter commented Jul 12, 2018

Good news, everyone! We've released version 1.4.0-beta2, which introduces Dokany support. You now have the choice between Dokany and WebDAV based virtual hard drives.

Please retest this issue with Dokany enabled.

@devarni

This comment has been minimized.

Copy link

devarni commented Jul 12, 2018

It seems the encoding performance is very slow... a 24GB files need about >5 hours

@overheadhunter

This comment has been minimized.

Copy link
Member

overheadhunter commented Jul 12, 2018

@devarni beta2 is still single threaded and has a lot of debugging code. performance is a non-goal of this release. so please stick with the original issue for now

@NTWarrior

This comment has been minimized.

Copy link

NTWarrior commented Jul 14, 2018

Still having issues dealing with >4GB files using Dokany on 1.4.0-beta2 on a vault in Google Drive accessed through File Stream, though I know File Stream could be the problem rather than Cryptomator.
I moved one large file out of the vault and onto a local drive and it was saved as a 0B file, then trying to copy (which I had meant to do the first time) another one threw up "The file 'abc.xyz' is too large for the destination file system" even though the file was just slightly over 4GB and the destination drive was on NTFS with plenty free space. Also getting very high CPU usage that persists after transfers have finished.

@infeo

This comment has been minimized.

Copy link
Member

infeo commented Sep 6, 2018

will be fixed when the newest https://github.com/cryptomator/dokany-nio-adapter library version is included.

@infeo

This comment has been minimized.

Copy link
Member

infeo commented Oct 11, 2018

Fixed with e08c2b7 .

@infeo infeo closed this Oct 11, 2018

@tobihagemann

This comment has been minimized.

Copy link
Member

tobihagemann commented Oct 15, 2018

Please retest with 1.4.0-beta3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment