HttpWebRequest BeginGetResponse timeout if main thread is blocked

[hardeep-fm]

Steps to Reproduce

Using HttpWebRequest BeginGetResponse always results in a timeout if we block the main thread waiting for a response. This behaviour only recently started happening -- it was working fine on older versions of Xamarin Android.

The demo app below is written using the example documentation from the Microsoft site (https://docs.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest.begingetresponse?view=netframework-4.7.2) which implies that blocking the main thread should be okay. This problem also doesn't happen when running on a Mac using Mono.

NetworkTest.zip

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.contoso.com");
request.ContentType = "text/html";
request.Method = "GET";

ManualResetEvent allDone = new ManualResetEvent(false);
int DefaultTimeout = 15 * 1000;

IAsyncResult result = (IAsyncResult)request.BeginGetResponse(new AsyncCallback(RespCallback), allDone);
ThreadPool.RegisterWaitForSingleObject(result.AsyncWaitHandle, new WaitOrTimerCallback(TimeoutCallback), request, DefaultTimeout, true);
allDone.WaitOne();

Expected Behavior

Network request should be called asynchronously in a background thread and response callback should be called from that or another non-main thread.

Actual Behavior

TimerCallback is triggered with "timeout=true" and request is aborted. I should mention that a network trace confirms that the http request is actually being made and returning successfully so it's only the response callback that doesn't get called as expected.

Version Information

=== Visual Studio Professional 2017 for Mac ===

Version 7.6.9 (build 22)
Installation UUID: 2b082c66-cf75-41c5-af8d-cbe2346a53a9
Runtime:
Mono 5.12.0.309 (2018-02/39d89a335c8) (64-bit)
GTK+ 2.24.23 (Raleigh theme)
Xamarin.Mac 4.4.1.178 (master / eeaeb7e6)

Package version: 512000309

=== NuGet ===

Version: 4.3.1.4445

=== .NET Core ===

Runtime: /usr/local/share/dotnet/dotnet
Runtime Version: 2.1.2
SDK: /usr/local/share/dotnet/sdk/2.1.302/Sdks
SDK Version: 2.1.302
MSBuild SDKs: /Library/Frameworks/Mono.framework/Versions/5.12.0/lib/mono/msbuild/15.0/bin/Sdks

=== Xamarin.Profiler ===

Version: 1.6.3
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Apple Developer Tools ===

Xcode 10.0 (14320.25)
Build 10A255

=== Xamarin.Android ===

Version: 9.0.0.20 (Visual Studio Professional)
Android SDK: /Users/hardeep.sidhu/Library/Developer/Xamarin/android-sdk-macosx
Supported Android versions:
5.0 (API level 21)
6.0 (API level 23)
7.1 (API level 25)
8.0 (API level 26)
8.1 (API level 27)

SDK Tools Version: 26.1.1
SDK Platform Tools Version: 28.0.0
SDK Build Tools Version: 27.0.3

Java SDK: /usr
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

=== Xamarin.Mac ===

Version: 5.0.0.0 (Visual Studio Professional)
Hash: b40230c0
Branch:
Build date: 2018-09-27 11:41:37-0400

=== Xamarin.iOS ===

Version: 12.0.0.15 (Visual Studio Professional)
Hash: 84552a46
Branch: xcode10
Build date: 2018-09-17 21:54:33-0400

=== Xamarin Inspector ===

Version: 1.4.3
Hash: db27525
Branch: 1.4-release
Build date: Mon, 09 Jul 2018 21:20:18 GMT
Client compatibility: 1

=== Build Information ===

Release ID: 706090022
Git revision: 0a0ba3c4593e9adb1c6ff6324e641036146af376
Build date: 2018-10-05 16:38:51+00
Build branch: release-7.6
Xamarin extensions: f7856b13f2c03a58e08381d3a5970bba18f5c7d7

=== Operating System ===

Mac OS X 10.13.6
Darwin 17.7.0 Darwin Kernel Version 17.7.0
Thu Jun 21 22:53:14 PDT 2018
root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64

[dellis1972]

Thank you for the detailed bug report. We will investigate and see if we can track down the issue

[gautambjain]

Is there any workaround for this? I am having the same issue.

[marek-safar]

@baulig could you please investigate

[baulig]

This looks very much like what is described in this blog article "Don't Block on Async Code" (which should be one of the first results if you google for "block on async").

The reason this code may have previously worked for you is merely an implementation detail - and with the Begin/End pattern getting more and more replaced by Task based async, you shouldn't really rely on being able to sync-block on a Begin/End "async" method.

What happens is that our BeginGetResponse() wraps GetResponseAsync(), which returns a Task. This Task continues on the current synchronization context - which on mobile is by default the UI Context, so the continuation would be ran on the main thread.

I am not entirely sure whether anything can be done about this on the BCL side of things; the Task-wrapping is actually copied from CoreFX; see src/Common/src/CoreLib/System/Threading/Tasks/TaskToApm.cs.

However, the more important question is why do you need this code to block on the main thread? Blocking on the UI Thread is really bad practice in any mobile app and should be avoided whenever possible.

As a workaround (though I haven't tested this), you should be able to wrap the thing in a Task and use TaskScheduler.Default (using on of those constructors that allow you to specify one).

@marek-safar You're more familiar with the Task code than me, would that work to avoid the deadlock and do you know which Task constructor to use?

[baulig]

Added Area:Documentation because we should really document this.

[hardeep-fm]

The example code provided comes directly from the Microsoft documentation at https://docs.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest.begingetresponse?view=netframework-4.7.2. If this is not a valid/correct way to do this then should that documentation be updated?