Segmentation fault on dotnet [name-of-app].dll on Debian

[Petermarcu]

@ChristianHaase commented on Sun Feb 11 2018

Getting segmentation fault when running the application. Only on HTTP client request sending to external API. The app start fine and I can navigate fine, but whenever I try to login (calling external API), I'm getting the segfault. Same goes for creating a new user, when calling the external API from my MVC backend.

Running the app locally, everything works as expected, and I can login fine, as well as creating a new user.

I have a try/catch around the enitre method, but I'm never hitting the catch. The segfault happens instead of the catch.

Technical details

dotnet --info

.NET Command Line Tools (1.0.4)

Product Information
  Version: 1.0.4
  Commit SHA-1 hash: afefd

Runtime Environment
  OS Name: debian
  OS Version: 9
  OS Platform: Linux
  RID: debian.8-x64
  Base Path: /opt/dotnet/sdk/1..4

Is there any fancy way for me te debug, or get some kind of log about what's happenening?

---

@Petermarcu commented on Tue Feb 20 2018

@leecow is Debian 9 supported by 1.0?

---

@leecow commented on Wed Feb 21 2018

No, Debian 9 is 2.0. https://github.com/dotnet/core/blob/master/os-lifecycle-policy.md.

---

@ChristianHaase commented on Wed Feb 21 2018

I suppose that would explain some issues, but the application runs fine on the server. It's only remote HTTP request that causes the segfault?

---

@wfurt commented on Wed Feb 21 2018

What version of curl and openssl do you have? This may be same as #23965 and #23796
You can try container with supported OS. That should get you around linkage problems.

---

@ChristianHaase commented on Wed Feb 21 2018

@wfurt

OpenSSL version:
OpenSSL 1.10f 25 May 2017

curl version:
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2l zlib/1.2.8 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) libssh2/1.7.0 nghttp2/1.18.1 librtmp/2.3

I don't know what issues you are referring to (#24891 and #23796)?

---

@wfurt commented on Wed Feb 21 2018

oh, the link did not work ;(
https://github.com/dotnet/corefx/issues/24552 and https://github.com/dotnet/corefx/issues/24891

---

@ChristianHaase commented on Wed Feb 21 2018

I'm trying to convert my project to .NET core 2.0, and updating the installed version on my server. I will update my progress here once done. Thanks for the inputs.

---

@ChristianHaase commented on Sat Feb 24 2018

I updated the project to 2.0, but it's still a problem. I'll take a look at the links you provided @wfurt

---

@ChristianHaase commented on Sat Feb 24 2018

I looked at the installed version of libcurl, and this is what I could find:

Anything I can do here?

---

@Petermarcu commented on Sun Mar 11 2018

I'm going to move this to the corefx repo where it will be tracked better.

[karelz]

Were you able to create minimal repro? (ideally even without MVC)
What exactly do you mean by "calling external API"? Are you doing any PInvokes?

[karelz]

cc @tmds

[detilium]

I haven't created a repro no. But it should be easy enough.

Basically it's just an HttpClient request, like this:

var httpClient = new HttpClient();
var response = await httpClient.GetAsync("some-uri");

I've trying just calling an HttpGet to https.//google.com. locally on my machine it works fine, but on my remote AWS test server, this throws the segfault.

[danmoseley]

Can you run the app under gdb and get a stack trace?

[detilium]

That's where my knowledge of linux fails me. GDB is installed on the server, but I'm not sure how to run the application with GDB to get the stacktrace. I managed to create a coredump, but again, no knowledge about how to actually read it.

If you have time to write a short step-by-step guide, I'd be happy to get it done. My apologies :/

[karelz]

It might be easiest to create a standalone repro we can try locally.
Quick search on internet shows bt as the right gdb command for stacktraces.

[detilium]

I created a repro, but I want to test it myself first. I've installed a local machine with debian 9, if it fails there, I'll send you the repro. If not, I'll test on the AWS debian server, and send you the repro if it fails. I'll get back to you once I've tested myself.

[detilium]

test.zip
Here's an attachment with a simple repro. To make the request, simple press the "Make call" link on the front page, to the left. This is a very simple repro, but basically that's what's causing the segfault (a simple HttpClient request). I haven't tested this myself yet, but I might as well dump this here.

[karelz]

@ChristianHaase can you try to reproduce it outside of ASP.NET? Or is the ASP.NET somehow important for it to repro? Console application with minimal dependencies would be best.
What do you mean by "I haven't tested this myself yet"? ... It would be best to wait for a repro you can confirm you can reliably reproduce to avoid wasting time by chasing things which may or may not reproduce.

[detilium]

No of course not. I apologize. I'm waiting for my server technician to boot up our test environment so I can test the project. I'll get back to you once I've tested it.

I'll test a simple console application as well, and attach a repro if it fails too.

[karelz]

No repro yet. Closing. If you have a repro, or more details leading to root-cause, feel free to reopen. Thanks!

[detilium]

I'm so sorry, we've had problems with our test server. We just got it rebooted yesterday, and I'm working on a repro. I'll make sure to upload the repro, once I've tested it. My apologies.

[detilium]

I'm sorry it took so long. I've attached a simple repro that should causes the segmentation fault on my test server. I've attached the project as a .rar to this comment.

NETCORE_Debian9_Repro.zip

[tmds]

@ChristianHaase please try running this on your AWS machine with gdb.

$ gdb dotnet
(gdb) r bin/Release/netcoreapp2.0/NETCORE_Debian9_Repro.dll

gdb will break when you get a segfault.
Then use the bt command to get a stacktrace.

[detilium]

Not really giving me anything readable

[tmds]

@ChristianHaase maybe this thread gets you further: dotnet/core#973 (comment)

[karelz]

@ChristianHaase try to get symbols. Please post text instead of pictures (they are not searchable)

Just to confirm, your repro seems to contain just this:

class Program
{
    static void Main()
    {
        using (var client = new System.Net.Http.HttpClient())
        {
            var response = client.GetAsync("https://google.dk");
            var content = response.Result.Content.ReadAsStringAsync();
        }

        System.Console.WriteLine("Hello World!");
    }
}

Is that what fails for you? Did you try it on more than 1 machine/VM? Does it fail for you also on latest .NET Core 2.1 RC?
@tmds @wfurt can you please check if it segfaults for you as well?

[detilium]

Yes this is the only code the project contains. Every remote http request caused the segfault. What exactly do you mean by "try to get symbols"?

I did try more than one machine. My test environment as well as my production environment. I haven't tried with 2.1 yet.

I'll test @tmds scenario later today

[tmds]

@karelz the code runs fine on my system. The referenced issue has a similar stacktrace. That issue is solved by uninstalling ssl 1.0.

[detilium]

I can confirm that removing the openssl version 1.0.0 does solve the problem, as 1.0.2 already was installed. Many thanks for the support, and I apologize it took so long.

[shanselman]

sudo apt-get remove libssl1.0.0 works for me

Was this fixed in 2.1 by dotnet/corefx#27208 by @janvorli?

[wfurt]

dotnet/corefx#27208

Should we close it now?

[detilium]

You may close this

[karelz]

Thanks for confirmation @ChristianHaase, closing.