This repository has been archived by the owner. It is now read-only.

Package restore on mono super slow and tons of requests end up being cancelled #1590

Closed
davidfowl opened this Issue Apr 4, 2015 · 22 comments

Comments

Projects
None yet
@davidfowl
Copy link
Member

davidfowl commented Apr 4, 2015

image

This was on mono 3.12.0.

We might be better off reducing the concurrency on mono and seeing if that helps. HttpClient has a default timeout and that's what is causing requests to be cancelled. It seems like there's an underlying queue limit being hit somewhere on mono and ServicePointManager isn't being respected.

/cc @bradwilson @ChengTian @migueldeicaza

@anfomin

This comment has been minimized.

Copy link

anfomin commented May 7, 2015

Package restore is still very slow on mono 4.0.1. Any plans when performance will be improved?

It takes more than 10 minutes to restore packages for normal MVC+EF project on OS X. However DNU on Windows restores it in 30-60 seconds.

@ryan1234

This comment has been minimized.

Copy link

ryan1234 commented Jun 8, 2015

This is killing me too. I have a build process that wants to restore packages on Mono and it takes 15 minutes to restore packages. =(

I think for the time being I'll archive them and unpack them manually from an earlier build.

Would love for creative solutions/hacks outside of tar'ing up dependencies.

@nosami

This comment has been minimized.

Copy link
Contributor

nosami commented Jul 9, 2015

@ryan1234 try using dnu from beta6 for restoring packages. It's single threaded and more reliable.

OmniSharp does this. https://github.com/OmniSharp/omnisharp-roslyn/blob/master/build.sh#L7-L14

@adamreeve

This comment has been minimized.

Copy link

adamreeve commented Jul 14, 2015

I'm still seeing this issue with Mono 4.0.1 and the latest beta6 (1.0.0-beta6-12226). It definitely looks like it's still trying to download packages in parallel.

@Aveline67

This comment has been minimized.

Copy link

Aveline67 commented Jul 21, 2015

You can copy paste packages from a windows computer to a linux, it works fine

@aggieben

This comment has been minimized.

Copy link

aggieben commented Jul 23, 2015

I tried looking into this somewhat and haven't really been able to reproduce any obvious issues like observed by the OP. On Gentoo hardware with mono 4.3 + dnx 1.0.0-beta6-12256, I'm seeing total restore times typically around 1 minute. On a Ubuntu VM with mono 4.0.3 + dnx-1.0.0-beta6-12256, I'm seeing much the same thing.

One problem I had on the Gentoo box was a lack of CA trust which caused restore to completely break altogether and never finish. I had to manually run Mono's sync-cert command to set it right; on Ubuntu this wasn't an issue because the package-installed mono takes care of this for you. Also, trimming our NuGet feed list helped.

@ganeshran

This comment has been minimized.

Copy link

ganeshran commented Jul 23, 2015

facing same issue on Arch Linux. Finally got all packages to install after running dnu restore at least 5 times.

Version : beta7-12264

@rkorsun

This comment has been minimized.

Copy link

rkorsun commented Jul 27, 2015

same here. Debian 7.8

@aggieben

This comment has been minimized.

Copy link

aggieben commented Jul 30, 2015

Are any of you running in a docker container?

@SamuelFisher

This comment has been minimized.

Copy link

SamuelFisher commented Jul 30, 2015

I'm having this problem using the microsoft/aspnet:1.0.0-beta5 Docker container

@ganeshran

This comment has been minimized.

Copy link

ganeshran commented Jul 30, 2015

I tried both on a natively installed Linux and a VM. Same issue on both.

@kplett

This comment has been minimized.

Copy link

kplett commented Jul 30, 2015

@ganeshran Are you using Mono 4.3? I've experienced similar issues until I switched to Mono JIT compiler version 4.0.1 (tarball Mon Apr 27 01:00:48 PDT 2015). (Currently I'm on beta6)

@ganeshran

This comment has been minimized.

Copy link

ganeshran commented Jul 30, 2015

@kplett i am using Mono 4.0.2.

@mpseidel

This comment has been minimized.

Copy link

mpseidel commented Jul 31, 2015

having the same issue when trying to run inside docker on digitalocean (german datacenter) following this tutorial http://blogs.msdn.com/b/webdev/archive/2015/01/14/running-asp-net-5-applications-in-linux-containers-with-docker.aspx using aspnet-Home\samples\latest\HelloWeb

building and running a node container with an express app only takes a few seconds in the same environment

@aggieben

This comment has been minimized.

Copy link

aggieben commented Aug 2, 2015

I feel inclined to think this is a problem with Mono's ThreadPool implementation. Setting MONO_THREADS_PER_CPU to a high number (say, 2000) allows previously problematic restores to succeed. Here are the ThreadPool defaults in my docker container: ThreadPool, Min/Max Threads: {worker: 2, io: 4}/{worker: 200, io: 8}. With MONO_THREADS_PER_CPU=2000: ThreadPool, Min/Max Threads: {worker: 200, io: 4}/{worker: 200, io: 8}.

It's worth investigating more to be sure, but it appears to me that this is a Mono problem, rather than a dnx problem.

@ChengTian

This comment has been minimized.

Copy link
Contributor

ChengTian commented Aug 7, 2015

👍 @aggieben 's workaround (setting MONO_THREADS_PER_CPU to 2000) does help me get rid of the timeout errors on our Linux CI machine.

@bradwilson

This comment has been minimized.

Copy link

bradwilson commented Aug 7, 2015

👍 It feels indistinguishable from restore on Windows now. Thanks!

@migueldeicaza

This comment has been minimized.

Copy link
Contributor

migueldeicaza commented Aug 10, 2015

While we have replaced Mono's thread pool with Microsoft in recent versions of Mono (4.2 and github/mono/mono master), it is unlikely that the issue is related to the threadpool implementation and even with that implementation, these issues will likely surface.

The sources are likely rooted here (while this information is old, chances are, some of the scenarios that made this bad are present in another form).

http://www.mono-project.com/archived/articlethreadpool_deadlocks/

@migueldeicaza

This comment has been minimized.

Copy link
Contributor

migueldeicaza commented Aug 11, 2015

Addendum: we could figure out whether the source is the threadpool, or the BeginInvoke support if you try Mono/master which now uses the CLR's thread pool implementation.

@sunsided

This comment has been minimized.

Copy link

sunsided commented Sep 7, 2015

Indeed, I added ENV MONO_THREADS_PER_CPU 2000 to my Dockerfile and now the image build is insanely fast. What kind of black magic is this? 😁

@muratg

This comment has been minimized.

Copy link
Member

muratg commented Sep 14, 2015

Thanks everyone! Closing this one.

@muratg muratg closed this Sep 14, 2015

@akoeplinger

This comment has been minimized.

Copy link
Contributor

akoeplinger commented Sep 14, 2015

For the record, the workaround of setting MONO_THREADS_PER_CPU was put directly in dnu with #2620, so users shouldn't need to set it themselves in the future.

darrencauthon added a commit to darrencauthon/csharp-sparkpost that referenced this issue May 19, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.