Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HttpWebRequest throws an uncatchable task exception when processing requests simultaneously with timeout specified #10488

Closed
david4neblio opened this issue Sep 6, 2018 · 17 comments

Comments

@david4neblio
Copy link

@david4neblio david4neblio commented Sep 6, 2018

Steps to Reproduce

Just open up a new project and copy and paste the following code to replicate the error.

using System;
using System.Net;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

namespace TestProgram
{
    class MainClass
    {
		static bool close_program = false;

        public static void Main(string[] args)
		{
			//Create a catch all Task exception handler
			//This will catch the eventual task exception
			TaskScheduler.UnobservedTaskException += (s, e) =>
                LastExceptionHandler(e.Exception, "TaskScheduler.UnobservedTaskException");

			Console.WriteLine("This program contains 2 types of exceptions");
			Console.WriteLine("One that is catchable, the other that is not. The latter will be displayed here.");
			Console.WriteLine("Now wait for the uncatchable exception (less than 60 seconds in most cases), do not press any key.");
			int counter = 0;
			while(true){
				if(Console.KeyAvailable){
					break;
				}
				Thread.Sleep(1000);
				counter++;

                //We will run 2 queries to the same URL in an alternate task
				Task.Run(() => QueryHttp());
				Task.Run(() => QueryHttp());

				Console.WriteLine(counter + " Second(s)");
				if(close_program == true){
					break;
				}
			}
        }

        public static string QueryHttp()
		{
			try
			{
				ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
                
				HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://httpstat.us/200?sleep=3000");
				request.Credentials = System.Net.CredentialCache.DefaultCredentials;
				request.Timeout = 1000;
                            
				using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
				{
					using (StreamReader respreader = new StreamReader(response.GetResponseStream()))
					{
						return respreader.ReadToEnd();
					}
				}
			}
			catch (Exception)
			{
				//This exception will normally be a timeout exception which is catchable
			}
			return "";
		}

		public static void LastExceptionHandler(Exception e, string source)
        {
			//This will catch the eventual task exception
			Console.WriteLine("Uncatchable thread exception: " + e);
			close_program = true;
        }
    }
}

Current Behavior

This code throws a task exception by method MyGetResponseAsync found within HttpWebRequest. The exception here is uncatchable due to the fact that the MyGetReponseAsync is a task created by BeginGetResponse and there is no current way (I'm aware of) to add an exception handler to the MyGetResponseAsync task. So when the exception is thrown, it bubbles to the global task exception handler if one is set.

Expected Behavior

That the exception created by MyGetResponseAsync should be catchable outside of global exception handlers. The exception itself is probably caused by multithreading issues with HttpWebRequest (as it only happens when multiple requests are sent around the same time) or the threadpool used by the created task.

On which platforms did you notice this

[ ] macOS
[X] Linux
[ ] Windows

Version Used:
This error happens also in the most recent version of Mono as well.

Mono JIT compiler version 5.12.0.226 (tarball Thu May  3 09:42:09 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	Interpreter:   yes
	LLVM:          supported, not enabled.
	GC:            sgen (concurrent by default)

Stacktrace

N/A - Run the code to see the stacktrace
@marek-safar

This comment has been minimized.

Copy link
Member

@marek-safar marek-safar commented Sep 6, 2018

@baulig another async rewrite side effect :-(. Please look into it

@baulig baulig self-assigned this Sep 11, 2018
@baulig

This comment has been minimized.

Copy link
Member

@baulig baulig commented Sep 12, 2018

I thought we fixed this in 8facc4a, but it looks like we didn't.

baulig added a commit to baulig/mono that referenced this issue Sep 13, 2018
baulig added a commit to baulig/mono that referenced this issue Sep 13, 2018
…ono#10488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes mono#10488.
baulig added a commit to baulig/mono that referenced this issue Sep 13, 2018
…ono#10488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes mono#10488.
baulig added a commit to baulig/mono that referenced this issue Sep 13, 2018
…ono#10488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes mono#10488.
@marek-safar marek-safar added this to the 2018-08 (5.18.xx) milestone Sep 20, 2018
monojenkins added a commit to monojenkins/mono that referenced this issue Sep 21, 2018
…no#10488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes mono#10488.
monojenkins added a commit to monojenkins/mono that referenced this issue Sep 21, 2018
…no#10488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes mono#10488.
marek-safar added a commit that referenced this issue Sep 21, 2018
…10488. (#10595)

* [System]: Make sure `HttpWebRequest` observes exceptions on timeout. #10488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes #10488.

* Add `TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.OnlyOnCanceled` as suggested by Marek.

* Use 'OnlyFaulted' only.
luhenry added a commit that referenced this issue Sep 27, 2018
… timeout. #10488. (#10732)

* [System]Make sure `HttpWebRequest` observes exceptions on timeout. #10488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes #10488.

* Add `TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.OnlyOnCanceled` as suggested by Marek.

* Use 'OnlyFaulted' only.
marek-safar added a commit that referenced this issue Oct 8, 2018
…0488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes #10488.
jonpryor added a commit to xamarin/xamarin-android that referenced this issue Oct 9, 2018
Bumps to mono/llvm:release_60@117a508c
Bumps to xamarin/xamarin-android-api-compatibility:master@7ccb4802

	$ git diff --shortstat e1af6ea..ab3c897d       # mono
        1443 files changed, 66049 insertions(+), 45745 deletions(-)
	$ git diff --shortstat bdb3a116..117a508c      # llvm
	 26794 files changed, 4110589 insertions(+), 754376 deletions(-)
	$ git diff --shortstat c550d1bd..7ccb4802      # xamarin-android-api-compatibility
	 2 files changed, 16260 insertions(+), 12347 deletions(-)

Incomplete summary of easily `grep`able fixes:

Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=11199
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=19436
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=23668
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=26983
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=33728
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=46917
fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=60065
Fixes: mono/mono#6173
Fixes: mono/mono#6466
Fixes: mono/mono#6647
Fixes: mono/mono#6834
Fixes: mono/mono#7058
Fixes: mono/mono#7137
Fixes: mono/mono#7260
Fixes: mono/mono#7305
Fixes: mono/mono#7402
Fixes: mono/mono#7525
Fixes: mono/mono#7610
Fixes: mono/mono#7649
Fixes: mono/mono#7655
Fixes: mono/mono#7683
Fixes: mono/mono#7685
Fixes: mono/mono#7716
Fixes: mono/mono#7731
Fixes: mono/mono#7785
Fixes: mono/mono#7828
Fixes: mono/mono#7944
Fixes: mono/mono#7947
Fixes: mono/mono#8036
Fixes: mono/mono#8074
Fixes: mono/mono#8089
Fixes: mono/mono#8112
Fixes: mono/mono#8122
Fixes: mono/mono#8143
Fixes: mono/mono#8149
Fixes: mono/mono#8152
Fixes: mono/mono#8175
Fixes: mono/mono#8177
Fixes: mono/mono#8250
Fixes: mono/mono#8267
Fixes: mono/mono#8273
Fixes: mono/mono#8282
Fixes: mono/mono#8310
Fixes: mono/mono#8311
Fixes: mono/mono#8329
Fixes: mono/mono#8340
Fixes: mono/mono#8372
Fixes: mono/mono#8407
Fixes: mono/mono#8409
Fixes: mono/mono#8422
Fixes: mono/mono#8430
Fixes: mono/mono#8439
fixes: mono/mono#8447
Fixes: mono/mono#8469
Fixes: mono/mono#8504
Fixes: mono/mono#8575
Fixes: mono/mono#8597
Fixes: mono/mono#8623
Fixes: mono/mono#8627
Fixes: mono/mono#8698
Fixes: mono/mono#8701
Fixes: mono/mono#8712
Fixes: mono/mono#8721
Fixes: mono/mono#8726
Fixes: mono/mono#8759
Fixes: mono/mono#8787
Fixes: mono/mono#8820
Fixes: mono/mono#8848
Fixes: mono/mono#8866
Fixes: mono/mono#8897
Fixes: mono/mono#8915
Fixes: mono/mono#8970
Fixes: mono/mono#8979
Fixes: mono/mono#9023
Fixes: mono/mono#9031
Fixes: mono/mono#9033
Fixes: mono/mono#9179
Fixes: mono/mono#9234
Fixes: mono/mono#9262
Fixes: mono/mono#9277
Fixes: mono/mono#9318
Fixes: mono/mono#9542
Fixes: mono/mono#9753
Fixes: mono/mono#9839
Fixes: mono/mono#9869
Fixes: mono/mono#9870
Fixes: mono/mono#9943
Fixes: mono/mono#9996
Fixes: mono/mono#10000
Fixes: mono/mono#10303
Fixes: mono/mono#10447
Fixes: mono/mono#10483
Fixes: mono/mono#10488
Fixes: xamarin/maccore#628
Fixes: xamarin/maccore#673
Fixes: #1561 (comment)
Fixes: #1845
Fixes: xamarin/xamarin-macios#4347
Fixes: xamarin/xamarin-macios#4617
Fixes: xamarin/xamarin-macios#4618
jonpryor added a commit to xamarin/xamarin-android that referenced this issue Dec 6, 2018
Bumps to mono/api-snapshot@b99fc87.
Bumps to mono/bockbuild@5af573e.
Bumps to mono/boringssl@41221b4.
Bumps to mono/corefx@23d0b58.
Bumps to mono/corert@af496fc.
Bumps to mono/linker@7af03ce.
Bumps to mono/NUnitLite@00e259a.
Bumps to mono/reference-assemblies@9325826.
Bumps to mono/roslyn-binaries@249709f.
Bumps to mono/xunit-binaries@bb58347.

	$ git diff --shortstat b63e5378..23f2024a      # mono 
	 1630 files changed, 50926 insertions(+), 92212 deletions(-)

Fixes: mono/mono#6352
Fixes: mono/mono#6947
Fixes: mono/mono#6992
Fixes: mono/mono#7615
Fixes: mono/mono#8340
Fixes: mono/mono#8407
Fixes: mono/mono#8575
Fixes: mono/mono#8627
Fixes: mono/mono#8707
Fixes: mono/mono#8766
Fixes: mono/mono#8848
Fixes: mono/mono#8866
Fixes: mono/mono#8935
Fixes: mono/mono#9010
Fixes: mono/mono#9023
Fixes: mono/mono#9031
Fixes: mono/mono#9033
Fixes: mono/mono#9106
Fixes: mono/mono#9109
Fixes: mono/mono#9155
Fixes: mono/mono#9179
Fixes: mono/mono#9232
Fixes: mono/mono#9234
Fixes: mono/mono#9262
Fixes: mono/mono#9277
Fixes: mono/mono#9292
Fixes: mono/mono#9318
Fixes: mono/mono#9318
Fixes: mono/mono#9332
Fixes: mono/mono#9407
Fixes: mono/mono#9421
Fixes: mono/mono#9505
Fixes: mono/mono#9542
Fixes: mono/mono#9581
Fixes: mono/mono#9623
Fixes: mono/mono#9684
Fixes: mono/mono#9750
Fixes: mono/mono#9753
Fixes: mono/mono#9772
Fixes: mono/mono#9839
Fixes: mono/mono#9869
Fixes: mono/mono#9921
Fixes: mono/mono#9943
Fixes: mono/mono#9947
Fixes: mono/mono#9973
Fixes: mono/mono#9996
Fixes: mono/mono#10000
Fixes: mono/mono#10031
Fixes: mono/mono#10035
Fixes: mono/mono#10227
Fixes: mono/mono#10243
Fixes: mono/mono#10303
Fixes: mono/mono#10448
Fixes: mono/mono#10483
Fixes: mono/mono#10488
Fixes: mono/mono#10863
Fixes: mono/mono#11123
Fixes: mono/mono#11138
Fixes? mono/mono#11146
Fixes: mono/mono#11202
Fixes: mono/mono#11378
Fixes: mono/mono#11479
Fixes: mono/mono#11613
Fixes: #1951
Fixes: xamarin/xamarin-macios#4347
Fixes: xamarin/xamarin-macios#4617
Fixes: xamarin/xamarin-macios#4984
@bocciab

This comment has been minimized.

Copy link

@bocciab bocciab commented Jan 17, 2019

I just tried installing Mono 5.18 along with Xamarin.Mac 5.2.1.13 and I still see the uncachable exception in the sample repro project. Is there something else needed to get the fix working?

@baulig

This comment has been minimized.

Copy link
Member

@baulig baulig commented Jan 21, 2019

It is certainly possible that your version of Xamarin.Mac has not picked up that fix yet.

@bocciab

This comment has been minimized.

Copy link

@bocciab bocciab commented Jan 21, 2019

This is a critical issue. I'm currently at a decision point to keep this crashing my app or downgrade users back to that 10.14 invisible-UI issue from prior mono releases.

Is the fix going to come to mono 5.12 or at least 5.16 (current VS-Stable)? The other posts on this make it sound like we're waiting for VS 2019 to go to full public release before we'll get this.

@baulig

This comment has been minimized.

Copy link
Member

@baulig baulig commented Jan 24, 2019

This should've been fixed in #10732.

@bocciab

This comment has been minimized.

Copy link

@bocciab bocciab commented Jan 24, 2019

I don't know what release version that is or should be in (we really could use public release versioning when talking about things - I can't follow your branching/release milestone naming). What I know is that if I look in the code in the latest stable 5.18 framework install I have (5.18.0.245), the code in the commit is not there. It wasn't in 5.16 either, but I've been installing all different versions trying to test things, so I lost my 5.16 install and VSMac is erroring trying to check updates to re-install VS Stable. I've also tried installing several versions of Mono and trying the sample repro posted in this issue and it's still throwing the same uncatchable task exception.

@NickAVolpe

This comment has been minimized.

Copy link

@NickAVolpe NickAVolpe commented Jan 24, 2019

If it helps anyone, when I replace HttpWebRequest with HttpClient the error is not reproducible. Which is frustrating on my end because my code is written with HttpClient and I am still seeing this exception in our production environment. Suggestions welcome :)

`

var client = new HttpClient();
client.Timeout = new System.TimeSpan(0, 0, 1000);

var request = new HttpRequestMessage(HttpMethod.Get, uri);

resp = await client.SendAsync(request);

var foo = await resp.Content.ReadAsStringAsync();
return foo;

`

Visual Studio Community 2017 for Mac
Version 7.7.3 (build 43)

Mono Framework MDK
Runtime:
Mono 5.16.0.221 (2018-06/b63e5378e38) (64-bit)
Package version: 516000221

Xamarin.Mac
Version: 5.2.1.13 (Visual Studio Community)

Xamarin.Android
Version: 9.1.5.1 (Visual Studio Community)

@baulig

This comment has been minimized.

Copy link
Member

@baulig baulig commented Jan 24, 2019

Are you seeing this in Xamarin.Mac or Desktop Mono?

The fix is definitely in 5.18 (5.18.0.245 is this commit). I just downloaded the package and the test case from the top of this issue runs fine with it.

The latest release of Xamarin.Mac seems to be 5.2.1.11 (which is this commit). The Mono in it is commit 000780, which is from the 2018-04 branch. Your 5.2.1.13 is also using the same Mono commit.

We did not back port the fix into 2018-04.

@NickAVolpe

This comment has been minimized.

Copy link

@NickAVolpe NickAVolpe commented Jan 24, 2019

Forgot to mention, I am running this in a Xamarin.Android project which I do not believe has mono 5.18 yet.

@bocciab

This comment has been minimized.

Copy link

@bocciab bocciab commented Jan 25, 2019

I was running the latest stable Xamarin.Mac. Updating to the latest stable version of it, and even trying to update to 5.18, I still see the crash occurring.

I downgraded to XM 4.4 (a known prior build that wasn't seeing this crash) and the sample app had no uncatchable unobserved task exceptions.

As noted before, I'm not able to get updates for VS Mac, so I can't easily re-update to the latest stable. So I'll just ask if you can clarify if there's a Visual Studio (Mac / Xamarin.Mac) Stable version that has the fix in it? If not, will it be pack ported to whatever is out in the Stable channel or do we have to wait for VS 2019 or something?

@roceh

This comment has been minimized.

Copy link

@roceh roceh commented Jan 25, 2019

Is there anyway to escalate a backport? This regression has been in Xamarin.Android since October.

@xleon

This comment has been minimized.

Copy link

@xleon xleon commented Jan 28, 2019

For those using Xamarin.Android, this pre-release is using mono 5.18

Not sure about iOS though

monojenkins added a commit to monojenkins/mono that referenced this issue Jan 28, 2019
…no#10488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes mono#10488.
marek-safar added a commit that referenced this issue Jan 29, 2019
…0488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes #10488.
@gautambjain

This comment has been minimized.

Copy link

@gautambjain gautambjain commented Jan 31, 2019

Badly need this fix. How to get it in VS 2017 without upgrading to VS 2019 Preview 2?

jonpryor added a commit to xamarin/xamarin-android that referenced this issue Jan 31, 2019
@gautambjain

This comment has been minimized.

Copy link

@gautambjain gautambjain commented Feb 1, 2019

Badly need this fix. How to get it in VS 2017 without upgrading to VS 2019 Preview 2?

It works now after I downgraded Visual Studio 2017 to version 15.7.6 from this link: https://docs.microsoft.com/en-us/visualstudio/productinfo/installing-an-earlier-release-of-vs2017

@MagicAndre1981

This comment has been minimized.

Copy link

@MagicAndre1981 MagicAndre1981 commented Feb 4, 2019

look if this latest VS 2017 15.9 compatible version has the fix:

https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-builds-d15-9/

The commit shows it has landed end January 2019 for 15.9 branch, so it should be included.

UnityAlex added a commit to Unity-Technologies/mono that referenced this issue Aug 29, 2019
…ono#10488. (mono#10595)

* [System]: Make sure `HttpWebRequest` observes exceptions on timeout. mono#10488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes mono#10488.

* Add `TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.OnlyOnCanceled` as suggested by Marek.

* Use 'OnlyFaulted' only.
UnityAlex added a commit to Unity-Technologies/mono that referenced this issue Sep 3, 2019
…ono#10488. (mono#10595)

* [System]: Make sure `HttpWebRequest` observes exceptions on timeout. mono#10488.

`HttpWebRequest.RunWithTimeoutWorker()` needs to observe the worker task's exception
on timeout to prevent it from being thrown via the `TaskScheduler.UnobservedTaskException`
event.  Fixes mono#10488.

* Add `TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.OnlyOnCanceled` as suggested by Marek.

* Use 'OnlyFaulted' only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.