New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apt-get update/upgrade/install downloads packages at unacceptably slow speeds. #2477

Open
RalphORama opened this Issue Sep 8, 2017 · 50 comments

Comments

Projects
None yet
@RalphORama

RalphORama commented Sep 8, 2017

  • Your Windows build number: Microsoft Windows [Version 10.0.15063]

  • What you're doing and what's happening: sudo apt-get update && sudo apt-get upgrade -y downloads packages at less that 100 kB/s.
    image

  • What's wrong / what should be happening instead: apt should fetch packages much faster. My internet speed is 100 down / 10 up, on a bare-metal Linux installation or VirtualBox instance, apt is blazing fast. Packages are downloaded at upwards of 1024 kB/s.

  • Strace of the failing command, if applicable: n/a

@gravcat

This comment has been minimized.

gravcat commented Sep 8, 2017

If there is nothing meaningful in the environment and you don't mind completely destroying it, try the following in an elevated command prompt:

lxrun /uninstall /full /y
lxrun /install

See if the problem persists if you completely redeploy the environment.

@sunilmut

This comment has been minimized.

Member

sunilmut commented Sep 8, 2017

@RalphORama - Thanks for your post. We are aware that WSL networking perf is not at par with native Linux, but should not be unbearable. There are plans to improve the WSL n/w perf. I am not sure if uninstall/reinstalling will change much here for you.

@gravcat

This comment has been minimized.

gravcat commented Sep 8, 2017

network speeds wget

my network speeds are fine and maxing out my link speed on Microsoft Windows [Version 10.0.15063]

@therealkenc

This comment has been minimized.

Collaborator

therealkenc commented Sep 8, 2017

We are aware that WSL networking perf is not at par with native Linux

Yeah WSL might not be on par with Real Linux, but we're talking Phoronix-style quibbling not mass degradation. I get awful apt-get speeds like the OP sometimes too (measured kbs/sec on a mbps link), but best evidence seems to be it is on Canonical's end (or something in between). I get the same thing in my Ubunutu VM from time to time. I am not sure why this is, but I think (speculating) that sometimes you just get a redirect to an unlucky server.

@RalphORama
Best thing to do would be take Ubuntu out as a variable. Use speedtest from WSL CLI like @gravcat suggests and then visit their website on Windows, and see if there is a serious gap in performance.

$ sudo apt install speedtest-cli
$ speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Skyway West (69.50.175.134)...
Selecting best server based on latency...
Hosted by Shaw Communications (Vancouver, BC) [28.08 km]: 28.778 ms
Testing download speed........................................
Download: 34.39 Mbit/s
Testing upload speed..................................................
Upload: 5.22 Mbit/s

If you see an orders-of-magnitude difference between Windows and WSL, report back because that would be good to know; it is always possible there is something unique going on with your setup.

@Vaporeon

This comment has been minimized.

Vaporeon commented Sep 9, 2017

I found that Windows Defender absolutely murders performance in WSL, especially tools like apt-get.
Try temporally disabling it and see if that improves it or not.

@heldchen

This comment has been minimized.

heldchen commented Sep 9, 2017

I found that Windows Defender absolutely murders performance in WSL, especially tools like apt-get.
Try temporally disabling it and see if that improves it or not.

this. Windows Defender really messes with I/O actions in WSL, not just with apt-get... adding scan exemptions does not seem to help much.

@therealkenc

This comment has been minimized.

Collaborator

therealkenc commented Sep 9, 2017

this. Windows Defender really messes with I/O actions in WSL

It does. But keep in mind, in the OP this was the 'Get' phase, which is literally just a curl of a .deb. 46.6 kilobytes per second on a 100Mbit/s downlink. Install phase I understand, sure. Windows Defender gets upset when thousands of little untrusted files start appearing from nowhere (from Defender's perspective) though its filesystem filter driver. But if it were Defender you should see the same slowdown on a bog standard curl from a fast server, and you don't; at least not to this degree. Couple that with the fact that turning Defender off doesn't help. So, something else. I could be wrong, because Defender is a ginormous black box that does who knows what, on or off. But I'm not seeing the causality here.

@RalphORama

This comment has been minimized.

RalphORama commented Sep 9, 2017

I reinstalled WSL as @gravcat suggested, and installed speedtest-cli. After running speedtest-cli and visiting the speedtest website, it appears WSL only uses 5% of my available bandwidth, maximum (1.6 Mbit/s vs 36 Mbit/s). I believe I have Windows Defender disabled, and I don't think I have any other programs that interfere with I/O on my machine. I use Malwarebytes as my antivirus, which doesn't run in the background. I'll try creating a firewall rule for bash.exe and report back.

Update: Creating a firewall rule and switching to OpenDNS had no effect on apt's download speeds.

image
image

@vinibudd

This comment has been minimized.

vinibudd commented Sep 17, 2017

image

image

@benhillis

This comment has been minimized.

Member

benhillis commented Oct 5, 2017

@therealkenc - Thanks for the repro steps. I tracked this down to a directory rename failing due something that has a handle open to one of the directory's children. Unfortunately this is an NTFS limitation that I can't think of a workaround without NTFS supporting posix-style rename. I know they were looking at that at some point, but I'm not sure of the status.

@DarkIlluminatus

This comment has been minimized.

DarkIlluminatus commented Jan 27, 2018

I am also experiencing this issue, did the solution suggested by @gravcat work for anyone?
I would hate to have to download the entire distro again at these below 100 kB/s speeds, so am reticent about attempting it.

@leandrw

This comment has been minimized.

leandrw commented Jan 27, 2018

Just turn off Windows Defender Real-time protection.

Add WSL folder and apt process in exclusion list will not work.

You should to completely disable real-time protection, reboot and will see how fast it will be

@DarkIlluminatus

This comment has been minimized.

DarkIlluminatus commented Jan 28, 2018

@Garry050

This comment has been minimized.

Garry050 commented Feb 22, 2018

same here.
speedtest: both good speed (download: 70-90Mbps)
apt: always 30-300kb/s (ftp.jaist.ac.jp)

Using ESET Internet Security 11. no help disable anti-virus.
16299.214

@benhillis

This comment has been minimized.

Member

benhillis commented Feb 22, 2018

@DarthSpock - unfortunately I think the posix-style rename work the ntfs team did doesn't handle the case where a directory has children with open handles. Adding @SvenGroot to confirm.

@therealkenc

This comment has been minimized.

Collaborator

therealkenc commented Feb 22, 2018

@benhillis
Your "this is an NTFS limitation that I can't think of a workaround" comment was posted to the wrong thread at the time (you thank me for repro steps that don't exist here). That is #640 aka #1529, which is unrelated to "apt-get update/upgrade/install downloads packages at unacceptably slow speeds".

Adding SvenGroot to confirm.

Like this

@sewashinobi

This comment has been minimized.

sewashinobi commented Feb 28, 2018

I can confirm, this is slow as hell. For me the speed is 219 B/s, while my internet speed on windows is 24mbps

@mskd12

This comment has been minimized.

mskd12 commented Mar 3, 2018

I can confirm as well, the speed has dropped down significantly especially when downloading many things

@Wingmore

This comment has been minimized.

Wingmore commented Mar 7, 2018

Has anyone found a fix? getting speeds of 40 kB/s

@rondi74

This comment has been minimized.

rondi74 commented Mar 7, 2018

I'm having the same problem! >_<

@SvenGroot

This comment has been minimized.

Member

SvenGroot commented Mar 7, 2018

Ubuntu on WSL uses the Ubuntu's main archive mirror, which may not be the fastest option depending on your location. Could those who are having this issue try choosing a local mirror from https://launchpad.net/ubuntu/+archivemirrors (tip: most countries have a mirror like us.archive.ubuntu.com, so you can try that for your country before scouring the list), updating /etc/apt/sources.list to use that mirror instead of archive.ubuntu.com, and see if that improves your download speed?

Make sure you run sudo apt update after editing sources.list.

You can use sed to make the change. For example, to change from archive.ubuntu.com to us.archive.ubuntu.com:
sudo sed -i "s/archive.ubuntu.com/us.archive.ubuntu.com/" /etc/apt/sources.list

@DarthSpock

This comment has been minimized.

DarthSpock commented Mar 7, 2018

Running Synaptic on Kali goes rather fast. Faster than normal apt surprisingly. But even apt on Kali runs faster than Ubuntu. Which seems to point to agreement with what's said above.

@therealkenc

This comment has been minimized.

Collaborator

therealkenc commented Mar 8, 2018

You can confirm whether the variable is your Ubuntu mirror by testing against speedtest.net from within WSL with speedtest-cli.

sudo apt install python-pip
pip install speedtest-cli   # installs in ~/.local/bin which is in your default path
speedtest-cli
@brunoeustaquio

This comment has been minimized.

brunoeustaquio commented Mar 8, 2018

@SvenGroot tks man, it´s way better now

@doomsbuster

This comment has been minimized.

doomsbuster commented Mar 24, 2018

Turning off Windows Defender Realtime protection did the trick. All other options did not work.

@navneethc

This comment has been minimized.

navneethc commented Mar 30, 2018

I'm having the same problems as well. Installed WSL only a month or so ago, and any time I try to install/update using apt-get I first get abysmal speeds (100s of Bps to KBps on a 75 Mbps connection) and then everything comes to a screeching halt, at which point I kill it.

Windows 10.0.16299 (and Defender is turned off).

@bazald

This comment has been minimized.

bazald commented Apr 8, 2018

Just upgraded to build 17133.1 and this seems to be a new issue for me. Speed test results aren't bad. It seems like it's just taking forever to initialize each and every TCP connection. Pings are fine, as is actual bandwidth. Disabling Windows Defender Realtime protection has no effect.

@therealkenc

This comment has been minimized.

Collaborator

therealkenc commented Apr 8, 2018

seems like it's just taking forever to initialize each and every TCP connection

I think (but can't prove because I don't have ipv6 with my ISP) the above issue might be a DNS lookup thing. Which is different than Sven's (very good) "distant mirror" hypothesis. There an issue# ref for the ipv6 DNS lookup problem but too lazy to look it up atm. Your download speed with apt should be okay once the connection is made if that is the problem tho. One thing people can try to isolate is to find the .deb url of one of the "slow" gets that apt is fetching, and then just try getting it with wget to see how the speed compares. That is (to a first order) what apt is doing under the hood after all.

@bazald

This comment has been minimized.

bazald commented Apr 9, 2018

@therealkenc good guess! Disabling IPv6 resolves the issue. If it was a problem before the upgrade, it didn't seem as noticeable? I'm not sure why that would be, but it's interesting.

@therealkenc

This comment has been minimized.

Collaborator

therealkenc commented Apr 9, 2018

If it was a problem before the upgrade, it didn't seem as noticeable? I'm not sure why that would be, but it's interesting.

There were WSL improvements in some ipv6 scenarios since your last update. So what might be happening is your ipv6 use was just failing outright before and falling back to ipv4 immediately. Now it is still failing, but "trying harder" (and taking longer). Couldn't tell you for sure because it would take a "before and after" strace compare, and that is not at all practical. But at least you are not alone; which is good in the scheme, since it is a known/reported problem as opposed to an unknown problem.

@ohnoimdead

This comment has been minimized.

ohnoimdead commented Apr 23, 2018

image

i realize this isn't exactly an apples-to-apples comparison, however, this is two somewhat similar ubuntu images (xenial in wsl and artful in vmware player) running on the same windows host at the same time. apt-get update called at as close to the same time as i could reasonably do manually. wsl takes nearly three minutes to complete compared to 2.5 seconds in a vm. happy to provide more information on my environment if that helps.

@nutcasev15

This comment has been minimized.

nutcasev15 commented May 24, 2018

Same problem on 1803 stable branch. Build 17134.48

@bazald How did you disable ipv6? The sysctl method which one would use on normal ubuntu doesn't work here.

I can mostly confirm @therealkenc 's suspicion on the ipv6 connection timeout theory as this issue appeared in the RC for 1803. 16299 or Insider builds about halfway between 1709 & 1803 didn't have this issue.

I changed the server to my local Indian server too. Didn't change anything.
I tried modifying the command at execution:
sudo apt-get update -o Acquire::ForceIPv4=true

But it didn't change the situation. I can confirm it happens with all connections on my setup, not just ipv6.

Some interesting patterns in the bug:

  1. Always Hangs at [Waiting for Headers]
  2. Hangs at specific points in the update process. Usually 22%, 28%, 93%
  3. If you kill the command with Ctrl+C there is like a 50% chance it will go further than before. Killing at 22% & reexecuting the command might get you past 22% but will get you stuck at 93%.
  4. Once It hangs, It does not recover & it fails to fetch whatever it wanted.
@bazald

This comment has been minimized.

bazald commented May 24, 2018

@nutcasev15 I disabled IPv6 for the entire distro by uncommenting "precedence ::ffff:0:0/96 100" in /etc/gai.conf.

@nutcasev15

This comment has been minimized.

nutcasev15 commented May 24, 2018

No change; confirms that my issues are not ipv6 or ipv4 specific.
Thanks Bazald

@therealkenc

This comment has been minimized.

Collaborator

therealkenc commented May 24, 2018

Using a slow server, which is demonstrated well by @ohnoimdead's post (one server is archive.ubuntu.com and the other is us.archive.ubuntu.com) is one issue. A separate issue is some folks having problems with ipv6 name resolution taking a long time. [A third issue, unrelated to this thread, is the install phase (versus get) being slow due to Defender among other things.]

@nutcasev15

This comment has been minimized.

nutcasev15 commented May 25, 2018

@therealkenc

  1. I changed my server to the local Indian version which works fine with Budgie in VM. Its not the server in my case.

  2. My ISP does not have ipv6 support, as far as I know. So I have this problem on ipv4.

  3. I have the slow install issue as well, slow download speed for packages too. Plus the apt-get update one.

Some logs to help clear things up:
ipv4 variant is with sudo apt-get update -o Acquire::ForceIPv4=true
Default variant is without any modifications to any system configs & without forcing ipv4.
Both use the Indian server. Both hang at 28%.
apt_log_ipv4.txt
apt_log_default.txt

@therealkenc

This comment has been minimized.

Collaborator

therealkenc commented May 25, 2018

I changed my server to the local Indian version which works fine with Budgie in VM. Its not the server in my case.

How to proceed was explained above here and here. Eliminate apt as a variable and see how your connection fares with speedtest.net under WSL versus Windows. If it is any consolation, I think there is something else going on here other than Sven's hypothesis (which may in fact be an issue for some folks in distant lands). My ISP doesn't support ipv6 either and my apt get phase is unacceptably slow as well. If there was a concrete and substantiated explanation this issue# wouldn't still be in the open state.

"It's a process"

@nutcasev15

This comment has been minimized.

nutcasev15 commented May 25, 2018

& The Plot Unnecessarily Thickens:

http://www.speedtest.net/result/7339057179

Retrieving speedtest.net configuration... Testing from Excell Media Pvt (175.101.9.82)... Retrieving speedtest.net server list... Selecting best server based on ping... Hosted by Excell Media (Hyderabad) [0.84 km]: 9.672 ms Testing download speed................................................................................ Download: 18.86 Mbit/s Testing upload speed................................................................................................ Upload: 33.82 Mbit/s
Upload in WSL is faster than Windows. WSL always had better upload, from the early days, but didn't think the speed would be close to double.
Meanwhile download suffers by the same factor in WSL.
The hell is happening?

EDIT: Speculation:
I am an android rom/kernel dev. I had this weird pattern on my tester's device upload/download speeds too.
I solved it there by tweaking the TCP buffers for both ipv4 & ipv6 & enabling TCP msg_fastopen by default. Took them to 8M max for download & 65K max for upload. They then normalized to the same speeds basically.

Download went from 27Mb/s to 50Mb/s while upload went from 60Mb/s to ~55Mb/s

@ohnoimdead

This comment has been minimized.

ohnoimdead commented May 30, 2018

@therealkenc interesting, i hadn't even noticed that apt in wsl was using the generic archive.ubuntu.com. updating to local us drastically reduced the time, however, it's still off by several seconds compared to vm (about 8 seconds now). oddly, an "npm install" (yes yes i know) seemed to take quite a bit longer on wsl vs vm for the same project about a month ago but now seems to be almost on par? ¯_(ツ)_/¯

@nosferatu500

This comment has been minimized.

nosferatu500 commented Jun 14, 2018

If disable Windows diffender that issue is gone

@navneethc

This comment has been minimized.

navneethc commented Jun 15, 2018

What is this black magic?! For the first time in 6 months I was able to successfully run update and upgrade. I haven't changed anything on my own, but between the last time I checked (some weeks ago) and today, there were some Windows updates (installed yesterday). Speeds weren't that impressive (started at 4000 kBps and eventually varied between 350-450 kBps), but considering what it used to be, it's a HUGE improvement.

However, now that this happened, I notice that 18.04 is available at the store. :D

@alanhyue

This comment has been minimized.

alanhyue commented Jul 15, 2018

@SvenGroot your answer saved my ass! Changing to a local mirror feels like.. reborn. Thanks a lot!

@Neurrone

This comment has been minimized.

Neurrone commented Sep 19, 2018

I'm getting dialup-like speeds on a fibre connection, and none of the solutions here work for me.

  • Disabling windows defender realtime protection
  • Disabling IPV6
  • Changing to a different mirror
@Neurrone

This comment has been minimized.

Neurrone commented Sep 19, 2018

Retrieving speedtest.net server list...
Selecting best server based on latency...
Testing download speed........................................
Download: 0.32 Mbit/s
Testing upload speed..................................................
Upload: 65.27 Mbit/s

Seems like a critical bug in downloading.

@mikevaleriano

This comment has been minimized.

mikevaleriano commented Sep 23, 2018

Had this issue when the distro was 16.04, trying again today with 18.04, the issue remains.

The slow network renders the whole product unusable, honestly. =/

@hrai

This comment has been minimized.

hrai commented Oct 10, 2018

Disabling the antivirus did it for me. I had symantec running.

@TimothyCole

This comment has been minimized.

TimothyCole commented Oct 11, 2018

Any updates on this? I ran update/upgrade earlier today not realizing this was a thing now it's rendered my env completely useless.

I'm also not running an anti-virus (even Windows Defender)

@otonieru

This comment has been minimized.

otonieru commented Oct 24, 2018

i also facing this issue

WSL only able to use a max bandwith around 200KB/s, while my connection is around 5MB/s (50Mbps)

and its not only for apt get, it also happen to git clone from google or github (or any) repo

disabling windows defender and use only iPv4 doesn't help.

@M4cs

This comment has been minimized.

M4cs commented Nov 23, 2018

image

Im guessing this is just due to Kali servers being slow but my Windows Speedtest.net got these results:

Same Upload speed but more than half of the download speed is cut.

Could windows system DNS settings effect this? I am routing through Cloudflare DNS.

@yuri-vashchenko

This comment has been minimized.

yuri-vashchenko commented Dec 3, 2018

The same here. tried everything suggested here, but nothing helps.
It has nothing to do with apt (I can live without often updates), but I cannot use subversion sever in local network because it takes ages to 'svn update', while the same 'svn update' completes almost instantly on hyper-v virtual machine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment