Copy very slow on Ubuntu 17.04 #451

giosuel · 2017-02-08T13:01:34Z

When I start cryptomator, everything is ok, the vault is created and open. But when I try to copy data, it says, that is takes over 500'000 hours.

I'm running Cryptomator on: Ubuntu 17.04
I'm using Cryptomator in version: 1.2.3

Description

When I start cryptomator, everything is ok, the vault is created and open. But when I try to copy data, it says, that is takes over 500'000 hours.
How can I increase that speed? Please help!

2017-02-08 13:36:09,937 INFO  [Cryptomator:39] Starting Cryptomator 1.2.3 on Linux 4.9.0-15-generic (amd64)
2017-02-08 13:36:09,945 INFO  [SingleInstanceManager:300] no running instance found
2017-02-08 13:36:10,391 INFO  [MainApplication:45] JavaFX application started
2017-02-08 13:36:10,437 INFO  [Localization:44] Detected language "en" and region "US"
2017-02-08 13:36:10,440 INFO  [Localization:49] Attempting to load localization from: /localization/en_US.txt
2017-02-08 13:36:10,441 INFO  [Localization:54] Attempting to load localization from: /localization/en.txt
2017-02-08 13:36:10,529 INFO  [SettingsProvider:88] Failed to load settings, creating new one.
2017-02-08 13:37:09,926 INFO  [SettingsProvider:113] Settings saved to /home/giosuel/.Cryptomator/settings.json
2017-02-08 13:37:32,062 INFO  [log:186] Logging initialized @82875ms
2017-02-08 13:37:32,104 INFO  [Server:345] jetty-9.3.3.v20150827
2017-02-08 13:37:32,127 INFO  [ContextHandler:775] Started o.e.j.s.ServletContextHandler@31ecb3c3{/,null,AVAILABLE}
2017-02-08 13:37:32,133 INFO  [ServerConnector:270] Started ServerConnector@741434d0{HTTP/1.1,[http/1.1]}{0.0.0.0:42427}
2017-02-08 13:37:32,134 INFO  [Server:397] Started @82948ms
2017-02-08 13:37:32,139 INFO  [WebDavServer:83] Cryptomator is running on port 42427
2017-02-08 13:37:32,295 INFO  [WebDavServer:119] Servlet available under http://localhost:42427/[...]/Code
2017-02-08 13:37:32,298 INFO  [AbstractWebdavServlet:169] authenticate-header = Basic realm="Jackrabbit Webdav Server"
2017-02-08 13:37:32,300 INFO  [AbstractWebdavServlet:174] csrf-protection = null
2017-02-08 13:37:32,300 INFO  [AbstractWebdavServlet:181] createAbsoluteURI = true
2017-02-08 13:37:32,301 INFO  [ContextHandler:775] Started o.c.f.w.FontendIdHidingServletContextHandler@3882002a{/[...]/Code,null,AVAILABLE}
2017-02-08 13:37:32,317 INFO  [WebDavMounterProvider:40] Using LinuxGvfsDavMounter
2017-02-08 13:57:14,564 INFO  [SettingsProvider:113] Settings saved to /home/giosuel/.Cryptomator/settings.json

The text was updated successfully, but these errors were encountered:

markuskreusch · 2017-02-08T15:12:27Z

Can you enable the debug mode (in the settings) and then restart Cryptomator and try to copy a file again. This will add additional output to the logfile which you may attach here afterwards so we can have a deeper look at the issue. You may use testdata to do that because the logfile will contain cleartext pathnames etc.

markuskreusch · 2017-02-28T09:11:35Z

Closing this as we got no response now for some time. If you come back to us we may repoen this.

beanstalkblue · 2017-03-11T21:35:55Z

I am experiencing the same issue on ubuntu 16.10.

I mount the davfs volume like so:

sudo mount -t davfs http://localhost:42427/my-hidden-id/cryptomator /mymountpoint/cryptomator -o username=me,rw,uid=me,file_mode=700,dir_mode=700

I rsync to the directory like so:

sudo rsync --info=progress2 -a --no-links --no-whole-file /home/me/Test/ /mymountpoint/cryptomator/Test/

For this test I am transferring a large javascript project (babylon.js).

I ran cryptomator in debug mode. I have attached the log.

(I had to trim the log to a smaller size because of github upload limits.)
cryptomator.log.txt

markuskreusch · 2017-03-13T15:15:14Z

Will try this on a test system and come back to you. I currently assume the problem may be that the babylon.js sources contain many small files. With such files WebDAV is not very fast.

beanstalkblue · 2017-03-13T21:19:23Z

Copying basically stops. It's not just slow, its 20kb/s after a certain amount of time.

markuskreusch · 2017-03-14T09:10:32Z

Tried this on debian 8.7 and ubuntu 16.04. rsync copyied with around 21MB/s (faster start, slowed down later). One interresting thing I noticed is, that copying completed before any data was encrypted. It seems all data was copyied to the davfs cache first and then send to cryptomator. Encryption speed reached 90MB/s (peak). You may check if the davfs cache on Ubuntu 17.04 is configured differently. Maybe that causes problems but in general I have no clue.

beanstalkblue · 2017-03-14T19:13:32Z

I just tried again using only the cli. I notice the slowdown happens right after messages like these in the console:

skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/sshpk-conv" skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/sshpk-sign" skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/sshpk-verify" skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/strip-bom" skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/strip-indent" skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/svgo" skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/tsc" skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/tsserver" skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/uglifyjs" skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/user-home" skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/uuid" skipping non-regular file "test/Babylon.js/Tools/Gulp/node_modules/.bin/webpack"

You don't have these files because you didn't do npm install on your Babylon.js.

Could this be the cause?

markuskreusch · 2017-03-14T20:12:16Z

Will try again with these files, just to be sure. But to tell the truth, if skipping those files cause issues we may not be able to fix this but it maybe an incompatibility of rsync and webdav shares or similar stuff.

Will have a look at the cryptomator log if I can reproduce the problem to check if anything weird happens regarding webdav requests/responses. One thing you could try to confirm this, is to use another webdav server and try rsyncing the same data to that location.

markuskreusch · 2017-03-15T11:37:46Z

Could reproduce the problem described and got a speed of around 140kB/s. Copying does not stop but just takes very long. The Cryptomator logfile contained many requests but no errors and after waiting for some time everything was copied.

I still assume this is caused by the huge amount of small files. Average file size inside the node_modules dir is ~6KiB. Inside this directory are tens of thounsands of files. A short test with some generated directories containing 18k 6KiB files lead to similar results. So Webdav is just really slow I assume.

Closing this as I really do not think this is a bug but only a side effect of the webdav speed limitations and the requests produced by rsync and davfs2. I suggest to use a thrid party webdav client to transfer the files into the vault - this in general leads to much more speed with many small files.

Looking forward to #252 because this may improve the situation.

jhuebner79 · 2017-05-29T12:26:30Z

How could we rsync stuff into Cryptomator containers without the webdav/davfs2 overhead then?

RubyShnol · 2018-02-28T22:55:34Z

Can confirm the issue. Basically, useless software under Ubuntu.

overheadhunter · 2018-03-01T12:12:24Z

How could we rsync stuff into Cryptomator containers without the webdav/davfs2 overhead then?

FUSE support is scheduled for the next major version. 😉

markuskreusch added misc:help-wanted Extra attention is needed state:awaiting-response We need further input from the issue author labels Feb 8, 2017

markuskreusch closed this as completed Feb 28, 2017

markuskreusch reopened this Mar 13, 2017

markuskreusch mentioned this issue Mar 13, 2017

Very slow copy speed in Ubuntu 16.10. #467

Closed

markuskreusch self-assigned this Mar 13, 2017

markuskreusch removed the state:awaiting-response We need further input from the issue author label Mar 13, 2017

markuskreusch closed this as completed Mar 15, 2017

This was referenced Mar 15, 2017

rsync stalled, filesize 0 #244

Closed

Encryption speed low #66

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Copy very slow on Ubuntu 17.04 #451

Copy very slow on Ubuntu 17.04 #451

giosuel commented Feb 8, 2017 •

edited by overheadhunter

Loading

markuskreusch commented Feb 8, 2017

markuskreusch commented Feb 28, 2017

beanstalkblue commented Mar 11, 2017 •

edited

Loading

markuskreusch commented Mar 13, 2017 •

edited

Loading

beanstalkblue commented Mar 13, 2017 •

edited

Loading

markuskreusch commented Mar 14, 2017

beanstalkblue commented Mar 14, 2017 •

edited

Loading

markuskreusch commented Mar 14, 2017

markuskreusch commented Mar 15, 2017

jhuebner79 commented May 29, 2017 •

edited

Loading

RubyShnol commented Feb 28, 2018

overheadhunter commented Mar 1, 2018

Copy very slow on Ubuntu 17.04 #451

Copy very slow on Ubuntu 17.04 #451

Comments

giosuel commented Feb 8, 2017 • edited by overheadhunter Loading

Description

markuskreusch commented Feb 8, 2017

markuskreusch commented Feb 28, 2017

beanstalkblue commented Mar 11, 2017 • edited Loading

markuskreusch commented Mar 13, 2017 • edited Loading

beanstalkblue commented Mar 13, 2017 • edited Loading

markuskreusch commented Mar 14, 2017

beanstalkblue commented Mar 14, 2017 • edited Loading

markuskreusch commented Mar 14, 2017

markuskreusch commented Mar 15, 2017

jhuebner79 commented May 29, 2017 • edited Loading

RubyShnol commented Feb 28, 2018

overheadhunter commented Mar 1, 2018

giosuel commented Feb 8, 2017 •

edited by overheadhunter

Loading

beanstalkblue commented Mar 11, 2017 •

edited

Loading

markuskreusch commented Mar 13, 2017 •

edited

Loading

beanstalkblue commented Mar 13, 2017 •

edited

Loading

beanstalkblue commented Mar 14, 2017 •

edited

Loading

jhuebner79 commented May 29, 2017 •

edited

Loading