Copying files is slow and slows to a crawl over time for large numbers of files [$200]

[elementaryBot]

Copying a lot of files via Pantheon Files becomes slower and slower over time.

I've created 250,000 100-byte files on tmpfs for testing, and kicked off copying to another tmpfs. It started off at speeds over 100Kb/s but halfway through it's just 4Kb/s (!) and dropping.

Profiling with sysprof shows that all this time is spent in g_list_last(), which probably means that we're abusing a linked list somewhere and that it has to walk the entire list of already copied files, one by one, for each next file copied.

Testcase:
mkdir ~/created-files ~/copy-here
sudo mount -t tmpfs -o size=1G,mode=0777 tmpfs ~/created-files
sudo mount -t tmpfs -o size=1G,mode=0777 tmpfs ~/copy-here
cd created-files
split -b 100 SOME-BIG-FILE

open Pantheon Files and copy "created-files" folder into "copy-here"

This is a synthetic test case, but I had over 250,000 files during my last backup for OS reinstallation, so this is a real-life scenario.

ProblemType: Bug
DistroRelease: elementary OS 0.3
Package: pantheon-files 0.1.5.1+r1680+pkg35~ubuntu0.3.1 [origin: LP-PPA-elementary-os-daily]
ProcVersionSignature: Ubuntu 3.13.0-43.72-generic 3.13.11.11
Uname: Linux 3.13.0-43-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.6
Architecture: amd64
CrashDB: pantheon_files
CurrentDesktop: Pantheon
Date: Sun Dec 21 04:42:10 2014
ExecutablePath: /usr/bin/pantheon-files
GsettingsChanges:

InstallationDate: Installed on 2014-12-10 (10 days ago)
InstallationMedia: elementary OS 0.3 "Freya" - Daily amd64 (20141209)
SourcePackage: pantheon-files
UpgradeStatus: No upgrade log present (probably fresh install)

Launchpad Details: #LP1404588 Sergey "Shnatsel" Davidoff - 2014-12-21 01:53:22 +0000

[elementaryBot]

pantheon-files-daemon has a lot of memory used too (250Mb)

Launchpad Details: #LPC Sergey "Shnatsel" Davidoff - 2014-12-21 04:41:05 +0000

[elementaryBot]

In addition, such state makes any operations in Files very slow. Even the startup process of pantheon-files while pantheon-files-daemon is in such state is very slow.

Launchpad Details: #LPC Sergey "Shnatsel" Davidoff - 2014-12-21 04:42:04 +0000

[elementaryBot]

Replacing GList with GSequence data structure might be a way to hotfix this without changing huge amounts of code.

Launchpad Details: #LPC Sergey "Shnatsel" Davidoff - 2014-12-24 21:55:20 +0000

[elementaryBot]

A bounty of 100$ has been placed on this bug

Launchpad Details: #LPC Jeremy Wootten - 2015-02-18 14:55:15 +0000

[elementaryBot]

Might this be related with a similar issue concerning very slow file transfer to USB stick?

Launchpad Details: #LPC Giulio Sant - 2015-02-23 16:46:20 +0000

[elementaryBot]

I have changed the bug description to clarify that the bounty relates to obtaining significant improvement in file copying performance in general, not just for file numbers of the order of 100,000. Even with comparatively small numbers of files (100 - 1000) Files is very much slower than other well known file managers. I have increased the bounty to reflect the widened scope.

Launchpad Details: #LPC Jeremy Wootten - 2015-03-01 08:04:25 +0000

[elementaryBot]

Note solved but it was improved some so I'm bumping it from the milestone

Launchpad Details: #LPC Cody Garver - 2015-03-25 11:33:10 +0000

[elementaryBot]

I assume both source and destination were open in a Files view during the copy? Different tabs or different windows? Icon View or other?

Launchpad Details: #LPC Jeremy Wootten - 2016-04-09 10:22:46 +0000

[elementaryBot]

So I ran a couple benchmarks to see if I could figure out what the problems might be here. Wrote a simple program, basically a "cp" clone using g_file_copy to benchmark copy speeds against "cp" itself. What I found is that g_file_copy has very similar performance to "cp" (copied 10,000 small files at about 160kB/sec), so no problems there. Seems more like this has to do with all the queuing and locking going on in the file manager. Been swapping out various data structures and benchmarking and seeing some small performance increases. Removing some locking from the deep counter and switching out the marlin file queue for a thread-safe GAsyncQueue improved things a bit. I've been getting between 40kB/sec to 60kB/sec with those changes. It might also be worth swapping out the GIOScheduler stuff since that is deprecated. Not sure if that will bring any speed increase with it.

Launchpad Details: #LPC Matt Spaulding - 2016-10-12 20:53:15 +0000

[elementaryBot]

Matt: Thanks for having a go at this. Just for clarity, the target is to get Files to be at least comparable to other popular filemanagers in performance in this aspect, say within 75%? This assumes that other features like "undo" that might affect speed are also comparable.

Launchpad Details: #LPC Jeremy Wootten - 2016-10-13 10:46:49 +0000

[elementaryBot]

Okay, thank you for the clarification. Which file managers should I run comparisons against? At least in my tests with Nautilus it's copy speeds with large numbers of files is very poor, comparable to what we're seeing with Files.

Launchpad Details: #LPC Matt Spaulding - 2016-10-13 15:09:16 +0000

[elementaryBot]

Matt: I was thinking of Thunar and PCFman primarily although I have not done a comparison recently I admit. I assumed Nautilus was was superior at time of filing of the bug but perhaps things have changed. It is a fairly old bug now.

If Files is (now) already comparable to the best file managers under the conditions quoted in the bug then I would be willing to change the target to a more modest improvement and/or fixing of associated memory leakages.

Launchpad Details: #LPC Jeremy Wootten - 2016-10-13 17:34:41 +0000

[elementaryBot]

Since the code seems to be on github now, I've got a patch if anyone would like to try and comment? See vjr@c972549

Launchpad Details: #LPC Vishal Rao - 2017-06-20 05:41:26 +0000

[elementaryBot]

Hello Vishal, thank you for your work. I see you have created a pull request on GitHub. We will be concentrating on getting integrated testing working and more complete before merging any major changes (we have only just migrated) but I should be able to test your fix fairly soon. It would be good to get this issue fixed. There is a general desire to replace GList with another structure where possible in elementary code. There are quite a lot of GLists inherited in Files. If you have any test conditions and comparative performance timings please include them in your PR.

Launchpad Details: #LPC Jeremy Wootten - 2017-06-20 08:22:22 +0000