Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Syncing slow when web gui open #867

Closed
mmack opened this issue Oct 16, 2014 · 56 comments
Closed

Syncing slow when web gui open #867

mmack opened this issue Oct 16, 2014 · 56 comments

Comments

@mmack
Copy link

@mmack mmack commented Oct 16, 2014

Hey Guys,
i have syncthing installed on my synology nas. And i have a really strange performance issue.

I can't better describe the issue other than this: If i have the web ui of the nas open in my browser, syncing slows down to a few bytes per second. If i close it it raises immediately to about 50Mbit.

Maybe this helps you guys to improve this great product! :)

Max

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Oct 16, 2014

Slows down network speed wise?
Can you check the CPU load with and without the Web UI running?

@mmack
Copy link
Author

@mmack mmack commented Oct 16, 2014

CPU usage of the process is about 30% higher if i open the webui.

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Oct 16, 2014

But is it hitting the CPU limit, hence why it could be slowing down?

@mmack
Copy link
Author

@mmack mmack commented Oct 17, 2014

Nope. It's not hitting any limit. That's why i posted here...

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Oct 17, 2014

I have no clue why this could happen then. The GUI runs in a separate routine, so I don't see how it would be interfering with the transfer speed apart from taking a slice of the CPU's time. But if the CPU is not maxed out, it makes very little sense.

@mmack
Copy link
Author

@mmack mmack commented Oct 18, 2014

I know it does not make sense. But it is happening.. Any debug info i can provide?

@calmh
Copy link
Member

@calmh calmh commented Oct 18, 2014

It'll be holding database snapshots and doing database reads, which is kind of expensive. Shouldn't block syncing as such, but will certainly drive up CPU usage...

@andybalholm
Copy link

@andybalholm andybalholm commented Oct 22, 2014

I'm using Syncthing to synchronize about 100 GB of files between two computers on my local network. If I open the web GUI on either machine, the transfer rate drops to about 1 KB per second. When the GUI is closed, the transfer rate is about 20 MB per second.

The CPU usage on the destination computer is about 160% when the GUI is closed, and 260% when it is open (4-core CPU). If I do a CPU profile on it when the GUI is open, the only functions that show up are garbage collection, and functions called by ServeHTTP. So I assume that means that generating the data for the GUI is using the whole 260% CPU, and starving the actual transfer process.

Most of the usage shown in the profile involves leveldb, so there is probably disk contention and/or database lock contention keeping the transfer process from using the remaining 140% CPU.

@rostchri
Copy link

@rostchri rostchri commented Oct 22, 2014

I can confirm this behaviour. I'm currently syncing (using v0.10.2 on both endpoints) 80 gb from solaris to osx. If the web-frontend at the osx-endpoint is open, the transfer-speed is about x KB/s where x < 100. As soon as the web-frontend at the osx-endpoint is closed, the speed increases to 5 MB/s (My osx-machine is rather old and slow).

It doesnt matter if i open the web-frontend at the solaris-endpoint ...

@calmh calmh added the bug label Oct 24, 2014
@cweimann
Copy link

@cweimann cweimann commented Dec 24, 2014

I have been waiting days for a sync to finish. I just found this problem report and closed the browser tab with the web gui in it. Now the ifstat traffic has gone from about 0.003Mb/s to 28Mb/s. Wow. After closing the web interface it takes it a few seconds to get back up to speed. Opening the tab up again and it promptly slows down.

Here is the output of netstat showing the effect of starting the opening web interface.

# netstat -I re1 1
            input          (re1)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
      2177     0     0    3289084       2365     0     163100     0
      2157     0     0    3261380       2355     0     162795     0
      2177     0     0    3006715       2706     0     815069     0
      1438     0     0    2166373       1564     0     105777     0
        12     0     0       6509          9     0       1118     0
        17     0     0       8534         19     0       7675     0
         4     0     0        258          5     0       3881     0
         4     0     0        290          4     0        382     0

I'm running FreeBSD to FreeBSD and FreeBSD to Windows.

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Dec 24, 2014

Perhaps having the GUI open is IO intensive?
I cannot reproduce this as the device I am running on is fairly powerful.

@cweimann
Copy link

@cweimann cweimann commented Dec 24, 2014

The sending side is a Xeon E5507 2.27Ghz and the receiving side is an i7 920 2.67Ghz. Hardly the latest and greatest but we aren't talking about a Raspberry Pi either.

More detail. Having the gui open on the sending side has no effect. Gui on the receiving side has a huge effect. Neither machine is cpu bound or disk bound. Cpu is certainly much busier with the gui open when it is actively transferring but it is not cpu bound.

I think calmh's suggesting of some sort of database contention makes the most sense since the gui on the receiving side, where I would think there are database updates going on, is that side that has an effect.

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Dec 24, 2014

Yes, but database has to be either limited by IO or by CPU in order to introduce lock contention... But lock contention should only be a problem once we are gitting somesort of limit which is not obvious.

It would be interesting to run a profiler to see whats eating up the time, but I am not sure how to debug this with the profiler, I guess you need to add logging around lock aquisition.

@BlackEdder
Copy link

@BlackEdder BlackEdder commented Jan 12, 2015

On the forum this issue comes up quite often (and I experience it myself on the raspberrypi). It seems to occur primarily in cases where people access the webgui remotely.

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Jan 12, 2015

I don't think it matters to be honest.
I guess someone just needs to run profiling to figure out what's causing it.

@AudriusButkevicius AudriusButkevicius added this to the v1.0 milestone Jan 12, 2015
@calmh calmh modified the milestones: v1.0, v1.0-maybe Jan 15, 2015
@facastagnini
Copy link
Contributor

@facastagnini facastagnini commented Jan 24, 2015

I can confirm the same behavior.

Scenario:
INFO: syncthing v0.10.21 (go1.4.1 darwin-amd64 default)
Devices: iMac [MacOS 10.10.1] <---> MacBook Pro [MacOS 10.10.1]
link: gigabit wired ethernet connection.

How can I help?

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Jan 24, 2015

-help explains how to run a cpu profile. Make sure its transfering stuff to capture the right stuff in the profile.

@facastagnini
Copy link
Contributor

@facastagnini facastagnini commented Jan 24, 2015

Here we go: http://www.filedropper.com/cpu-1653

Please let me know it if works.
Thanks

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Jan 24, 2015

what version was this produced with (os, bits, syncthing version)?

@risoul
Copy link

@risoul risoul commented Jan 27, 2015

Same issue here, no problem when I do not use web Gui ou GTK Gui. It would be fine to inform users about that, I was near to quit syncthing to btsync because of that.

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Jan 27, 2015

It only happens on some machines.
@facastagnini still hasn't answered my question in order for me to try and work out what's causing it.

@risoul
Copy link

@risoul risoul commented Jan 27, 2015

It happens on mine.
Majaro Linux 64 bits/ Kernel 3.16 / Thinkpad T410 Core i5 / Syncthing v0.10.21 from AUR
Let me know if I can do anything (log...)

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Jan 27, 2015

Well it's the same steps as I've explained above.

@canton7
Copy link
Member

@canton7 canton7 commented Mar 9, 2015

Maybe related: if you leave the GUI open, it's not uncommon to see requests start to pile up. That is, the GUI has done a set of polls, but these haven't all been responded to when the next set of polls is sent out. Once this starts to happen, it snowballs: I can end up with hundreds of HTTP requests which are all pending, and yet more are created every couple of seconds.

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Mar 9, 2015

We only perform requests once every 10 seconds, hence if we fail to respond within 10 seconds, that's already an issue.

I am keen to see which requests start snowballing.

@canton7
Copy link
Member

@canton7 canton7 commented Mar 9, 2015

I'll make a note of it (in a separate issue, and link it in) when I next see it.

@devurandom
Copy link

@devurandom devurandom commented Mar 9, 2015

Instead of performing HTTP requests every Ns, wouldn't it be better to use BOSH?

@Zillode
Copy link
Member

@Zillode Zillode commented Mar 9, 2015

@AudriusButkevicius it is related to #734

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Mar 9, 2015

BOSH?

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Mar 9, 2015

We are using long polling for the events interface, but for how much data I have transferred long polling does not make sense, as the number is constantly changing. This makes the problem worse, not better.

Bosh seems to be slightly different to long polling, but I am not sure I understand the benefits.

@risoul
Copy link

@risoul risoul commented Mar 9, 2015

No more issue for me (high cpu usage when gui open), I use 0.10.25 now and I can keep gui open.

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Apr 7, 2015

Should be fixed in 0.11.

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Apr 7, 2015

454e688 Ref commit.
Needs to be adapted by all downstream people for this to work too.

@EmilioMoreno
Copy link

@EmilioMoreno EmilioMoreno commented Oct 13, 2015

Hi,
I can confirm this is still happening with v0.11.26. When I open WebUI, bandwidth transfer slows to very low values after all the information is loaded in the WebUI (about 1 min or so). Then, if you have the WebUI open, speed is higher but it does not increases to where it should be, and once you close the webUI and after a while (about 2 min or so), it goes back to the original speed.

In my case, running without webUI open shows 1.5Mbps. When I open WebUI and while it is loading WebUI data it slows down to 100Kbps. Once the data is loaded and while the WebUI is active, speed is about 400Kbps. Closing the WebUI tab gets speed back to 1.5Mbps.

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Oct 13, 2015

What sort of device are you running on?

@EmilioMoreno
Copy link

@EmilioMoreno EmilioMoreno commented Oct 13, 2015

on one side i have a synology NAS1813+, (intelAtomD2700, 2 cores with HT, 4GB Ram).
on the other side, a MacbookPro (intel core 2 Duo, 2.8Ghz, 4 GB Ram).
When i open WebUI on the Synology, everything is slowing down as described.

Cpu is never at 100% and the NAS is responding perfectly.

I have tried renicing all the synching process to -10 in the nas side and it seems that the behaviour is more stable, but if i keep nice to 0, it happens again.

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Oct 13, 2015

I guess its just not getting enough CPU time when doing more then one thing. Also, I'd check how many threads syncthing spawns, as it should be number of cores usually.

@EmilioMoreno
Copy link

@EmilioMoreno EmilioMoreno commented Oct 13, 2015

The number of main processes is 2, which matches the number of cores in NAS side. However, taking a look at htop i can see the number of threads is higher, about 31.

In the MAcbookpro side, 1 process, which matches the number of cores too.

If i do the same test in the MacBook Pro side (open the WebUI), even if i have the process with nice -15, the network transfer slows heavily.

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Oct 13, 2015

We always have 2 processes when not launched via service manager. I guess you could provide a block profile and a cpu profile, but it's not obvious to me how to debug this. @calmh ideas?

Should we make the summarySvc pause configurable?

@calmh
Copy link
Member

@calmh calmh commented Oct 13, 2015

Whichever way we twist and turn, providing info for the GUI is a lot of extra work, that is avoided by not having the GUI up. The CPU isn't the only thing here, there's also lots of database reads added by the summary service, which can be expensive if the db is a slow NAS disk.

I wouldn't worry about the number of threads and processes and the niceness, and just accept the fact that not having the GUI open is a performance win.

In the long run, not having to do database scans to get at the information we need would be an ever larger win...

@EmilioMoreno
Copy link

@EmilioMoreno EmilioMoreno commented Oct 13, 2015

I agree that non-having the WebUI will provide the best performance increase :-), but at least people (at least myself) would need a way to see what is happening. I only need to see what is current status of each end, what file is being transferred and the speed, but having to enable the debug in different modules to see how it goes is not practical.

Anyway, decreasing the niceness of the process to -15 has decreased the impact to open WebUI on the NAS side, so at least i can make myself an idea.

@Nattuma
Copy link

@Nattuma Nattuma commented Oct 16, 2015

Okay I have the same problem... I am however wondering what a web Gui is in the first place. Does it simply relate to opening sync thing in a webbrowser? Or is there some function I need to manually shut off to make sure webgui is not running?

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Oct 16, 2015

No, just close the browser tab.

@chrisf4lc0n
Copy link

@chrisf4lc0n chrisf4lc0n commented Jan 2, 2016

It seems to significantly raise the CPU usage, but still staying low at around 7% with E3-1230 and just around 2% with the E5-2697v2 which it still not enough to slow the transfers down on the 6 ZFS RAIDZ 10 config... Once the WebUI is shut transfer tripples to what it should be for 70 mbps connection...

@AudriusButkevicius
Copy link
Member

@AudriusButkevicius AudriusButkevicius commented Jan 2, 2016

I think its potentially lock contention related to calculating the UI info.

@syncthing syncthing locked and limited conversation to collaborators Jun 16, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.