Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Assembly.Location is sometimes corrupted on .NET 8 #93127

Closed
loop-evgeny opened this issue Oct 6, 2023 · 11 comments · Fixed by #93238
Closed

Assembly.Location is sometimes corrupted on .NET 8 #93127

loop-evgeny opened this issue Oct 6, 2023 · 11 comments · Fixed by #93238
Assignees
@AaronRobinsonMSFT
Labels
area-AssemblyLoader-coreclr blocking-release
Milestone
8.0.0

Comments

@loop-evgeny
Copy link

loop-evgeny commented Oct 6, 2023 •
edited by AaronRobinsonMSFT
Loading

Description

The Assembly.Location property sometimes (5%-50% of the time) returns a corrupted string (most commonly "/", but sometimes other values that appear to be a truncated version of the real path with one extra character appended). This only seems to happen for the System.Net.Sockets.dll assembly and only with COMPlus_PerfMapEnabled=1 and LANG=sv_SE.UTF-8. It did not happen on .NET 6.0, 3.1 or earlier versions with the same code. It does not always happen on .NET 8.

Sometimes the real application also crashes with a segfault around the same point in the code somewhere in libcoreclr.so, but I have not been able to get a proper stack trace or repro the segfault. I cannot be sure if it's part of the same problem, but happy to share a core dump privately.

Reproduction Steps

Extract locationbug.zip, edit go.sh to set DOTNET_ROOT to your .NET 8 SDK directory, make it executable and run it.

The main method code:

	private static void Main(string[] args)
	{
		const string hostname = "www.example.com"; // Host needs to resolve, but does not need to be a MongoDB server

		new MongoClient(new MongoClientSettings	{ Servers = new[] {new MongoServerAddress(hostname)} });
		new HttpClient().GetAsync("http://" + hostname);

		var socketAssembly = typeof(Socket).Assembly;
		Console.Error.WriteLine($"Socket assembly location: {socketAssembly.Location}");

		if (!socketAssembly.Location.EndsWith(".dll", StringComparison.Ordinal))
			throw new ApplicationException($"Unexpected location \"{socketAssembly.Location}\" for assembly FullName={socketAssembly.FullName}");
	}

The Mongo and HttpClient code looks bizarre, but it does not happen without those! (The real application does not use HttpClient, but actually connects to a MongoDB server using MongoClient.)

Expected behavior

Socket assembly location: .../dotnet/sdk-8.0-rc1/shared/Microsoft.NETCore.App/8.0.0-rc.1.23419.4/System.Net.Sockets.dll

This is the output most of the time.

Actual behavior

Sometimes:

Socket assembly location: /
Unhandled exception. System.ApplicationException: Unexpected location "/" for assembly FullName=System.Net.Sockets, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

Sometimes this happens on the first attempt, sometimes it takes 50 or so.

Regression?

Yes, worked on .NET 6.0, 3.1 and earlier. (Not tested on 7.0.)

Known Workarounds

Don't set COMPlus_PerfMapEnabled=1 or LANG=sv_SE.UTF-8.
It also seems to be very sensitive to the order of statements.

Configuration

.NET SDK 8.0.100-rc.1.23463.5
Linux x64
Reproduced on Linux Mint 20 VM (kernel 5.4.0-163-generic) and multiple Ubuntu 22.04.3 servers (kernels 5.15.0-86-generic and 6.2.0-1013-aws).

Other information

No response
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Oct 6, 2023
@ghost
Copy link

ghost commented Oct 6, 2023

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

The Assembly.Location property sometimes (5%-50% of the time) returns a corrupted string (most commonly "/", but sometimes other values that appear to be a truncated version of the real path with one extra character appended). This only seems to happen for the System.Net.Sockets.dll assembly and only with COMPlus_PerfMapEnabled=1 and LANG=sv_SE.UTF-8. It did not happen on .NET 6.0, 3.1 or earlier versions with the same code. It does not always happen on .NET 8, hence the bash loop in go.sh.

Sometimes the real application also crashes with a segfault around the same point in the code somewhere in libcoreclr.so, but I have not been able to get a proper stack trace or repro the segfault. I cannot be sure if it's part of the same problem, but happy to share a core dump privately.

Reproduction Steps

Extract locationbug.zip, edit go.sh to set DOTNET_ROOT to your .NET 8 SDK directory, make it executable and run it.

The main method code:

	private static void Main(string[] args)
	{
		const string hostname = "www.example.com"; // Host needs to resolve, but does not need to be a MongoDB server

		new MongoClient(new MongoClientSettings	{ Servers = new[] {new MongoServerAddress(hostname)} });
		new HttpClient().GetAsync("http://" + hostname);

		var socketAssembly = typeof(Socket).Assembly;
		Console.Error.WriteLine($"Socket assembly location: {socketAssembly.Location}");

		if (!socketAssembly.Location.EndsWith(".dll", StringComparison.Ordinal))
			throw new ApplicationException($"Unexpected location \"{socketAssembly.Location}\" for assembly FullName={socketAssembly.FullName}");
	}

Expected behavior

Socket assembly location: .../dotnet/sdk-8.0-rc1/shared/Microsoft.NETCore.App/8.0.0-rc.1.23419.4/System.Net.Sockets.dll

This is the output most of the time.

Actual behavior

Sometimes:

Socket assembly location: /
Unhandled exception. System.ApplicationException: Unexpected location "/" for assembly FullName=System.Net.Sockets, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

Sometimes this happens on the first attempt, sometimes it takes 50 or so.

Regression?

Yes, worked on .NET 6.0, 3.1 and earlier. (Not tested on 7.0.)

Known Workarounds

Don't set COMPlus_PerfMapEnabled=1 or LANG=sv_SE.UTF-8.
It also seems to be very sensitive to the order of statements.

Configuration

.NET SDK 8.0.100-rc.1.23463.5
Linux x64
Reproduced on Linux Mint 20 VM (kernel 5.4.0-163-generic) and multiple Ubuntu 22.04.3 servers (kernels 5.15.0-86-generic and 6.2.0-1013-aws).

Other information

No response

Author: loop-evgeny
Assignees: -
Labels:

area-System.Net.Sockets
Milestone: -

@danmoseley
Copy link
Member

@jkotas I recall a report of this recently and you mentioned it was likely a UTF8 confusion. I can't find it though..?

@vcsjones
Copy link
Member

vcsjones commented Oct 6, 2023

This sounds like #90657. Based on the commit info and the back port, I think this is fixed for RC2.

@ghost
Copy link

ghost commented Oct 6, 2023

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

The Assembly.Location property sometimes (5%-50% of the time) returns a corrupted string (most commonly "/", but sometimes other values that appear to be a truncated version of the real path with one extra character appended). This only seems to happen for the System.Net.Sockets.dll assembly and only with COMPlus_PerfMapEnabled=1 and LANG=sv_SE.UTF-8. It did not happen on .NET 6.0, 3.1 or earlier versions with the same code. It does not always happen on .NET 8.

Sometimes the real application also crashes with a segfault around the same point in the code somewhere in libcoreclr.so, but I have not been able to get a proper stack trace or repro the segfault. I cannot be sure if it's part of the same problem, but happy to share a core dump privately.

Reproduction Steps

Extract locationbug.zip, edit go.sh to set DOTNET_ROOT to your .NET 8 SDK directory, make it executable and run it.

The main method code:

	private static void Main(string[] args)
	{
		const string hostname = "www.example.com"; // Host needs to resolve, but does not need to be a MongoDB server

		new MongoClient(new MongoClientSettings	{ Servers = new[] {new MongoServerAddress(hostname)} });
		new HttpClient().GetAsync("http://" + hostname);

		var socketAssembly = typeof(Socket).Assembly;
		Console.Error.WriteLine($"Socket assembly location: {socketAssembly.Location}");

		if (!socketAssembly.Location.EndsWith(".dll", StringComparison.Ordinal))
			throw new ApplicationException($"Unexpected location \"{socketAssembly.Location}\" for assembly FullName={socketAssembly.FullName}");
	}

The Mongo and HttpClient code looks bizarre, but it does not happen without those! (The real application does not use HttpClient, but actually connects to a MongoDB server using MongoClient.)

Expected behavior

Socket assembly location: .../dotnet/sdk-8.0-rc1/shared/Microsoft.NETCore.App/8.0.0-rc.1.23419.4/System.Net.Sockets.dll

This is the output most of the time.

Actual behavior

Sometimes:

Socket assembly location: /
Unhandled exception. System.ApplicationException: Unexpected location "/" for assembly FullName=System.Net.Sockets, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

Sometimes this happens on the first attempt, sometimes it takes 50 or so.

Regression?

Yes, worked on .NET 6.0, 3.1 and earlier. (Not tested on 7.0.)

Known Workarounds

Don't set COMPlus_PerfMapEnabled=1 or LANG=sv_SE.UTF-8.
It also seems to be very sensitive to the order of statements.

Configuration

.NET SDK 8.0.100-rc.1.23463.5
Linux x64
Reproduced on Linux Mint 20 VM (kernel 5.4.0-163-generic) and multiple Ubuntu 22.04.3 servers (kernels 5.15.0-86-generic and 6.2.0-1013-aws).

Other information

No response

Author: loop-evgeny
Assignees: -
Labels:

area-AssemblyLoader-coreclr, area-VM-coreclr, untriaged
Milestone: -

@jkotas
Copy link
Member

jkotas commented Oct 6, 2023

Thank you for reporting this issue! Yes, this is duplicate of #90657 that is fixed in RC2.

cc @AaronRobinsonMSFT

@jkotas jkotas closed this as completed Oct 6, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Oct 6, 2023
@loop-evgeny
Copy link
Author

OK, thanks for the quick response @jkotas !

Do you expect this to have caused the segfaults, too, or is that a separate issue?

When is RC2 expected to be released so we can test this?

Also, do you have any idea why it only happened under such specific circumstances - especially only for that one assembly and with COMPlus_PerfMapEnabled=1?

@jkotas
Copy link
Member

jkotas commented Oct 6, 2023

Do you expect this to have caused the segfaults, too, or is that a separate issue?

Yes, this issue can cause seg faults.

When is RC2 expected to be released so we can test this?

RC2 should be released in less than 2 weeks. If you would like to try something sooner, you can install daily build of the runtime (https://github.com/dotnet/runtime/blob/main/docs/project/dogfooding.md#daily-builds-table).

do you have any idea why it only happened under such specific circumstances

It is a race condition. Tracing makes it more likely to hit the race condition.

@loop-evgeny
Copy link
Author

@jkotas Thanks, I've downloaded daily builds of the SDK and the runtime and can still repro, though it seems to take longer now:

$ ./go.sh 
MSBuild version 17.8.1+585e09762 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
/home/em/dotnet/sdk-8.0-daily/sdk/8.0.100-rtm.23506.1/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(311,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/media/sf_work/Bugs/System.Net.Sockets-Location/locationbug/locationbug.csproj]
  locationbug -> /media/sf_work/Bugs/System.Net.Sockets-Location/locationbug/bin/Debug/net8.0/locationbug.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.49
==== TEST RUN 1 ====
Socket assembly location: /home/em/dotnet/sdk-8.0-daily/shared/Microsoft.NETCore.App/8.0.0-rtm.23506.12/System.Net.Sockets.dll
...
==== TEST RUN 65 ====
Socket assembly location: /
Unhandled exception. System.ApplicationException: Unexpected location "/" for assembly FullName=System.Net.Sockets, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
   at LocationBug.Program.Main(String[] args) in /media/sf_work/Bugs/System.Net.Sockets-Location/locationbug/Program.cs:line 21
./go.sh: line 12:  7387 Aborted                 (core dumped) ./bin/Debug/net8.0/locationbug

(took 108 attempts on the next go)

$ dotnet --info
.NET SDK:
 Version:           8.0.100-rtm.23506.1
 Commit:            0ffc9fdc93
 Workload version:  8.0.100-manifests.eeb18dd1

Runtime Environment:
 OS Name:     linuxmint
 OS Version:  20
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /home/em/dotnet/sdk-8.0-daily/sdk/8.0.100-rtm.23506.1/

.NET workloads installed:
 Workload version: 8.0.100-manifests.eeb18dd1
There are no installed workloads to display.

Host:
  Version:      8.0.0-rtm.23506.12
  Architecture: x64
  Commit:       b17a34c818

.NET SDKs installed:
  8.0.100-rtm.23506.1 [/home/em/dotnet/sdk-8.0-daily/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.0-rtm.23502.22 [/home/em/dotnet/sdk-8.0-daily/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.0-rtm.23506.12 [/home/em/dotnet/sdk-8.0-daily/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

@jkotas jkotas reopened this Oct 9, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Oct 9, 2023
@jkotas jkotas added this to the 8.0.0 milestone Oct 9, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Oct 9, 2023
@jkotas
Copy link
Member

jkotas commented Oct 9, 2023

@AaronRobinsonMSFT Looks like your fix missed this place for some reason

runtime/src/coreclr/vm/perfinfo.cpp

Line 52 in 6524d17

value.Printf("%s%c%s%c%p", path.GetUTF8(), sDelimiter, guid, sDelimiter, baseAddr);

@jkotas jkotas changed the title Assembly.Location for System.Net.Sockets.dll is sometimes corrupted on .NET 8 RC1 Assembly.Location is sometimes corrupted on .NET 8 RC1 Oct 9, 2023
@jkotas jkotas changed the title Assembly.Location is sometimes corrupted on .NET 8 RC1 Assembly.Location is sometimes corrupted on .NET 8 Oct 9, 2023
@loop-evgeny
Copy link
Author

loop-evgeny commented Oct 9, 2023 •
edited
Loading

runtime/src/coreclr/vm/perfinfo.cpp

Line 52 in 6524d17

value.Printf("%s%c%s%c%p", path.GetUTF8(), sDelimiter, guid, sDelimiter, baseAddr); 

// Logs image loads into the process' perfinfo-%d.map file
void PerfInfo::LogImage(PEAssembly* pPEAssembly, CHAR* guid)

Ha, so that's why it only happens with COMPlus_PerfMapEnabled=1!

@AaronRobinsonMSFT AaronRobinsonMSFT self-assigned this Oct 9, 2023
@AaronRobinsonMSFT AaronRobinsonMSFT mentioned this issue Oct 9, 2023
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Oct 9, 2023
@AaronRobinsonMSFT AaronRobinsonMSFT mentioned this issue Oct 9, 2023
Merged
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Oct 9, 2023
@loop-evgeny
Copy link
Author

Confirmed fixed in 8.0.0-rtm.23513.17 - I cannot repro now even after 1000 iterations.

@ghost ghost locked as resolved and limited conversation to collaborators Nov 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@AaronRobinsonMSFT AaronRobinsonMSFT

Labels
area-AssemblyLoader-coreclr blocking-release
Projects
None yet
Milestone
8.0.0
Development

Successfully merging a pull request may close this issue.

Make a copy of assembly path. AaronRobinsonMSFT/runtime
5 participants
@vcsjones @danmoseley @jkotas @AaronRobinsonMSFT @loop-evgeny