Package restore on mono super slow and tons of requests end up being cancelled

[davidfowl]

This was on mono 3.12.0.

We might be better off reducing the concurrency on mono and seeing if that helps. HttpClient has a default timeout and that's what is causing requests to be cancelled. It seems like there's an underlying queue limit being hit somewhere on mono and ServicePointManager isn't being respected.

/cc @bradwilson @ChengTian @migueldeicaza

[anfomin]

Package restore is still very slow on mono 4.0.1. Any plans when performance will be improved?

It takes more than 10 minutes to restore packages for normal MVC+EF project on OS X. However DNU on Windows restores it in 30-60 seconds.

[ryan1234]

This is killing me too. I have a build process that wants to restore packages on Mono and it takes 15 minutes to restore packages. =(

I think for the time being I'll archive them and unpack them manually from an earlier build.

Would love for creative solutions/hacks outside of tar'ing up dependencies.

[nosami]

@ryan1234 try using dnu from beta6 for restoring packages. It's single threaded and more reliable.

OmniSharp does this. https://github.com/OmniSharp/omnisharp-roslyn/blob/master/build.sh#L7-L14

[adamreeve]

I'm still seeing this issue with Mono 4.0.1 and the latest beta6 (1.0.0-beta6-12226). It definitely looks like it's still trying to download packages in parallel.

[Aveline67]

You can copy paste packages from a windows computer to a linux, it works fine

[aggieben]

I tried looking into this somewhat and haven't really been able to reproduce any obvious issues like observed by the OP. On Gentoo hardware with mono 4.3 + dnx 1.0.0-beta6-12256, I'm seeing total restore times typically around 1 minute. On a Ubuntu VM with mono 4.0.3 + dnx-1.0.0-beta6-12256, I'm seeing much the same thing.

One problem I had on the Gentoo box was a lack of CA trust which caused restore to completely break altogether and never finish. I had to manually run Mono's sync-cert command to set it right; on Ubuntu this wasn't an issue because the package-installed mono takes care of this for you. Also, trimming our NuGet feed list helped.

[ganeshran]

facing same issue on Arch Linux. Finally got all packages to install after running dnu restore at least 5 times.

Version : beta7-12264

[rkorsun]

same here. Debian 7.8

[aggieben]

Are any of you running in a docker container?

[SamuelFisher]

I'm having this problem using the microsoft/aspnet:1.0.0-beta5 Docker container

[ganeshran]

I tried both on a natively installed Linux and a VM. Same issue on both.

[kplett]

@ganeshran Are you using Mono 4.3? I've experienced similar issues until I switched to Mono JIT compiler version 4.0.1 (tarball Mon Apr 27 01:00:48 PDT 2015). (Currently I'm on beta6)

[ganeshran]

@kplett i am using Mono 4.0.2.

[mpseidel]

having the same issue when trying to run inside docker on digitalocean (german datacenter) following this tutorial http://blogs.msdn.com/b/webdev/archive/2015/01/14/running-asp-net-5-applications-in-linux-containers-with-docker.aspx using aspnet-Home\samples\latest\HelloWeb

building and running a node container with an express app only takes a few seconds in the same environment

[aggieben]

I feel inclined to think this is a problem with Mono's ThreadPool implementation. Setting MONO_THREADS_PER_CPU to a high number (say, 2000) allows previously problematic restores to succeed. Here are the ThreadPool defaults in my docker container: ThreadPool, Min/Max Threads: {worker: 2, io: 4}/{worker: 200, io: 8}. With MONO_THREADS_PER_CPU=2000: ThreadPool, Min/Max Threads: {worker: 200, io: 4}/{worker: 200, io: 8}.

It's worth investigating more to be sure, but it appears to me that this is a Mono problem, rather than a dnx problem.

[ChengTian]

👍 @aggieben 's workaround (setting MONO_THREADS_PER_CPU to 2000) does help me get rid of the timeout errors on our Linux CI machine.

[bradwilson]

👍 It feels indistinguishable from restore on Windows now. Thanks!

[migueldeicaza]

While we have replaced Mono's thread pool with Microsoft in recent versions of Mono (4.2 and github/mono/mono master), it is unlikely that the issue is related to the threadpool implementation and even with that implementation, these issues will likely surface.

The sources are likely rooted here (while this information is old, chances are, some of the scenarios that made this bad are present in another form).

http://www.mono-project.com/archived/articlethreadpool_deadlocks/

[migueldeicaza]

Addendum: we could figure out whether the source is the threadpool, or the BeginInvoke support if you try Mono/master which now uses the CLR's thread pool implementation.

[sunsided]

Indeed, I added ENV MONO_THREADS_PER_CPU 2000 to my Dockerfile and now the image build is insanely fast. What kind of black magic is this? 😁

[muratg]

Thanks everyone! Closing this one.

[akoeplinger]

For the record, the workaround of setting MONO_THREADS_PER_CPU was put directly in dnu with #2620, so users shouldn't need to set it themselves in the future.