Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mono issuing SIGSEGV running C# Semantic Tests #10309

Closed
jaredpar opened this issue Aug 27, 2018 · 22 comments
Assignees

Comments

@jaredpar
Copy link
Contributor

@jaredpar jaredpar commented Aug 27, 2018

Steps to Reproduce

This was found attempting to get the C# semantic tests running on Mono

  1. Clone https://github.com/jaredpar/roslyn
  2. Check out the branch repro/mono-semantic-crash2
  3. Make sure mono is on your path
  4. Run ./build.sh --restore --build
  5. Run ./build.sh --test --mono

Actual Behavior

This will run the Semantic unit tests only and eventually Mono will crash with a SIGSEGV in the following function:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <0x00064>
  at (wrapper alloc) object.AllocVector (intptr,intptr) <0x00163>

Full stack and dump info at the end of the report.

Note: xunit runs tests in a random-ish order. Hence it's possible you will hit a different crash than this one. I pretty consistently saw this locally though.

Expected Behavior

The xunit process runs to completion likely with a number of failing tests.

On which platforms did you notice this

This was discovered on Ubuntu 18.04. I'm using the latest stable mono as described by the Download page

https://www.mono-project.com/download/stable/#download-lin

jaredpar@jaredpar-ubuntu1:~/code/roslyn/src/Compilers/CSharp/Test/Semantic$ mono --version
Mono JIT compiler version 5.14.0.177 (tarball Mon Aug  6 09:07:45 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	Interpreter:   yes
	LLVM:          yes(3.6.0svn-mono-/)
	GC:            sgen (concurrent by default)

Stacktrace and full details


Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <0x00064>
  at (wrapper alloc) object.AllocVector (intptr,intptr) <0x00163>
  at System.IO.MemoryStream.set_Capacity (int) [0x00057] in <2943701620b54f86b436d3ffad010412>:0
  at System.IO.MemoryStream.EnsureCapacity (int) [0x00061] in <2943701620b54f86b436d3ffad010412>:0
  at System.IO.MemoryStream.Write (byte[],int,int) [0x000b3] in <2943701620b54f86b436d3ffad010412>:0
  at System.IO.BinaryWriter.Write (string) [0x0007b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.__BinaryWriter.WriteString (string) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.BinaryObjectString.Write (System.Runtime.Serialization.Formatters.Binary.__BinaryWriter) [0x00013] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.__BinaryWriter.WriteObjectString (int,string) [0x00026] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.__BinaryWriter.WriteMemberString (System.Runtime.Serialization.Formatters.Binary.NameInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,string) [0x0000d] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteString (System.Runtime.Serialization.Formatters.Binary.NameInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,object) [0x00034] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteKnownValueClass (System.Runtime.Serialization.Formatters.Binary.NameInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,object) [0x0000d] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteMembers (System.Runtime.Serialization.Formatters.Binary.NameInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,object,System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo) [0x00128] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteMemberSetup (System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,string,System.Type,object,System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo) [0x00054] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write (System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,string[],System.Type[],object[],System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo[]) [0x000c1] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write (System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo) [0x00197] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize (object,System.Runtime.Remoting.Messaging.Header[],System.Runtime.Serialization.Formatters.Binary.__BinaryWriter,bool) [0x001d3] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream,object,System.Runtime.Remoting.Messaging.Header[],bool) [0x0006e] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream,object,System.Runtime.Remoting.Messaging.Header[]) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream,object) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Runtime.Remoting.RemotingServices.SerializeCallData (object) [0x00053] in <2943701620b54f86b436d3ffad010412>:0
  at (wrapper xdomain-invoke) Xunit.MessageSinkAdapter.OnMessageWithTypes (Xunit.Abstractions.IMessageSinkMessage,System.Collections.Generic.HashSet`1<string>) [0x00049] in <76951d54e5564f89923967e66e39e57a>:0
  at Xunit.OptimizedRemoteMessageSink.OnMessage (Xunit.Abstractions.IMessageSinkMessage) [0x0000e] in <76951d54e5564f89923967e66e39e57a>:0
  at Xunit.Sdk.MessageBus.DispatchMessages () [0x00003] in <72a4ae576f6946c394b2b07638b5aa79>:0
  at Xunit.Sdk.MessageBus.ReporterWorker () [0x0000e] in <72a4ae576f6946c394b2b07638b5aa79>:0
  at Xunit.Sdk.XunitWorkerThread/<>c.<.ctor>b__1_0 (object) [0x00000] in <72a4ae576f6946c394b2b07638b5aa79>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00025] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ThreadHelper.ThreadStart (object) [0x0000f] in <2943701620b54f86b436d3ffad010412>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x0004f] in <2943701620b54f86b436d3ffad010412>:0
/proc/self/maps:
40610000-40630000 rwxp 00000000 00:00 0 
407e1000-40941000 rwxp 00000000 00:00 0 
40947000-40a6a000 rwxp 00000000 00:00 0 
40ca0000-40d50000 rwxp 00000000 00:00 0 
40d93000-40e53000 rwxp 00000000 00:00 0 
40e57000-40fb7000 rwxp 00000000 00:00 0 
41131000-41161000 rwxp 00000000 00:00 0 
41163000-41173000 rwxp 00000000 00:00 0 
41227000-412b7000 rwxp 00000000 00:00 0 
412f7000-41307000 rwxp 00000000 00:00 0 
41317000-4139a000 rwxp 00000000 00:00 0 
4166f000-4167f000 rwxp 00000000 00:00 0 
41683000-41716000 rwxp 00000000 00:00 0 
41723000-417d3000 rwxp 00000000 00:00 0 
41a42000-41c92000 rwxp 00000000 00:00 0 
41c98000-424a8000 rwxp 00000000 00:00 0 
5624e45bb000-5624e49e6000 r-xp 00000000 08:02 3678536                    /usr/bin/mono-sgen
5624e4be5000-5624e4bed000 r--p 0042a000 08:02 3678536                    /usr/bin/mono-sgen
5624e4bed000-5624e4bf1000 rw-p 00432000 08:02 3678536                    /usr/bin/mono-sgen
5624e4bf1000-5624e4c07000 rw-p 00000000 00:00 0 
5624e4c6b000-5624e92da000 rw-p 00000000 00:00 0                          [heap]
7fd4c84f8000-7fd4c8578000 rw-p 00000000 00:00 0 
7fd4c857c000-7fd4c85fc000 rw-p 00000000 00:00 0 
7fd4c8600000-7fd4c8700000 rw-p 00000000 00:00 0 
<<<SNIP>>>
7fd539723000-7fd539964000 r--p 00000000 08:02 2375324                    /home/jaredpar/code/roslyn/Binaries/Debug/UnitTests/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests/net46/Microsoft.CodeAnalysis.dll
7fd539964000-7fd539a3a000 r--p 00000000 08:02 2375327                    /home/jaredpar/code/roslyn/Binaries/Debug/UnitTests/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests/net46/Roslyn.Test.Utilities.dll
7fd539a3a000-7fd539a5d000 r--p 00000000 08:02 2375323                    /home/jaredpar/code/roslyn/Binaries/Debug/UnitTests/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests/net46/Microsoft.CodeAnalysis.CSharp.Test.Utilities.dll
7fd539a5d000-7fd53abc6000 r--p 00000000 08:02 2375350                    /home/jaredpar/code/roslyn/Binaries/Debug/UnitTests/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests/net46/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests.dll
7fd53abc6000-7fd53abd4000 r--p 00000000 08:02 2375319                    /home/jaredpar/code/roslyn/Binaries/Debug/UnitTests/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests/net46/xunit.core.dll
7fd53abd4000-7fd53ac55000 rw-p 00000000 00:00 0 
7fd53ac55000-7fd53ac56000 rw-p 00000000 00:00 0 
7fd53ac56000-7fd53ae97000 r--p 00000000 08:02 2375324                    /home/jaredpar/code/roslyn/Binaries/Debug/UnitTests/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests/net46/Microsoft.CodeAnalysis.dll
7fd53ae97000-7fd53c000000 r--p 00000000 08:02 2375350                    /home/jaredpar/code/roslyn/Binaries/Debug/UnitTests/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests/net46/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests.dll
7fd53c000000-7fd53c026000 rw-p 00000000 00:00 0 
7fd53c026000-7fd540000000 ---p 00000000 00:00 0 
7fd540000000-7fd540005000 rw-p 00000000 00:00 0 
7fd540005000-7fd540040000 r--p 00000000 08:02 2375320                    /home/jaredpar/code/roslyn/Binaries/Debug/UnitTests/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests/net46/xunit.execution.desktop.dll
7fd540040000-7fd5401c9000 rw-p 00000000 00:00 0 
7fd5401c9000-7fd54069e000 r--p 00000000 08:02 2375322                    /home/jaredpar/code/roslyn/Binaries/Debug/UnitTests/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests/net46/Microsoft.CodeAnalysis.CSharp.dll
7fd54069e000-7fd540774000 r--p 00000000 08:02 2375327                    /home/jaredpar/code/roslyn/Binaries/Debug/UnitTests/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests/net46/Roslyn.Test.Utilities.dll
7fd540774000-7fd54087a000 rw-p 00000000 00:00 0 
7fd54087a000-7fd54087b000 r--p 00000000 08:02 4330368                    /usr/lib/mono/4.5/Facades/System.Linq.dll
7fd54087b000-7fd540a04000 rw-p 00000000 00:00 0 
7fd540a04000-7fd540a0d000 rw-p 00000000 00:00 0 
7fd540a0d000-7fd540a0f000 r--p 00000000 08:02 4330396                    /usr/lib/mono/4.5/Facades/System.Resources.ResourceManager.dll
7fd540a0f000-7fd540a58000 r--p 00000000 08:02 3152204                    /tmp/ffe70b7f-27f2-421c-a615-930771773aa4/ffe70b7f-27f2-421c-a615-930771773aa4/assembly/shadow/4cf9e2d8/640fc5fe_28f62726_00000001/xunit.runner.utility.net452.dll
7fd540a58000-7fd540ad8000 rw-p 00000000 00:00 0 
7fd540ad8000-7fd540adc000 r--p 00000000 08:02 4330409                    /usr/lib/mono/4.5/Facades/System.Runtime.dll
7fd540adc000-7fd540dc0000 r--p 00000000 08:02 4325574                    /usr/lib/mono/gac/System.Web/4.0.0.0__b03f5f7f11d50a3a/System.Web.dll
7fd540dc0000-7fd5410c2000 r--p 00000000 08:02 4198077                    /usr/lib/mono/gac/System.Xml/4.0.0.0__b77a5c561934e089/System.Xml.dll
7fd5410c2000-7fd5411bb000 r--p 00000000 08:02 4325412                    /usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll
7fd5411bb000-7fd54147e000 r--p 00000000 08:02 4198057                    /usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll
7fd54147e000-7fd541a06000 r-xp 00000000 08:02 4326175                    /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7fd541a06000-7fd541c05000 ---p 00588000 08:02 4326175                    /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7fd541c05000-7fd541c06000 r--p 00587000 08:02 4326175                    /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7fd541c06000-7fd541c07000 rw-p 00588000 08:02 4326175                    /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7fd541c07000-7fd541c2d000 rw-p 00000000 00:00 0 
7fd541c2d000-7fd541fff000 r--p 00000000 08:02 4198073                    /usr/lib/mono/4.5/mscorlib.dll
7fd541fff000-7fd542fff000 rw-p 00000000 00:00 0 
7fd542fff000-7fd543000000 ---p 00000000 00:00 0 
7fd543000000-7fd543c02000 rw-p 00000000 00:00 0 
7fd543c02000-7fd543c04000 r--p 00000000 08:02 4330443                    /usr/lib/mono/4.5/Facades/System.Threading.Tasks.dll
7fd543c04000-7fd543c0c000 rw-p 00000000 00:00 0 
7fd543c0c000-7fd543c10000 r--p 00000000 08:02 4330409                    /usr/lib/mono/4.5/Facades/System.Runtime.dll
7fd543c10000-7fd543c33000 r--p 00000000 08:02 2375323                    /home/jaredpar/code/roslyn/Binaries/Debug/UnitTests/Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests/net46/Microsoft.CodeAnalysis.CSharp.Test.Utilities.dll
7fd543c33000-7fd543c43000 rwxp 00000000 00:00 0 
7fd543c43000-7fd543c44000 rw-p 00000000 00:00 0 
7fd543c44000-7fd543c90000 r--p 00000000 08:02 4198431                    /usr/lib/mono/gac/Mono.Security/4.0.0.0__0738eb9f132ed756/Mono.Security.dll
7fd543c90000-7fd543cae000 r--p 00000000 08:02 4198051                    /usr/lib/mono/gac/System.Configuration/4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
7fd543cae000-7fd543caf000 ---p 00000000 00:00 0 
7fd543caf000-7fd543cb0000 rw-p 00000000 00:00 0 
7fd543cb0000-7fd543cb8000 ---p 00000000 00:00 0 
7fd543cb8000-7fd543eaf000 rw-p 00000000 00:00 0 
7fd543eaf000-7fd54487e000 r--p 00000000 08:02 3676869                    /usr/lib/locale/locale-archive
7fd54487e000-7fd544a65000 r-xp 00000000 08:02 4199299                    /lib/x86_64-linux-gnu/libc-2.27.so
7fd544a65000-7fd544c65000 ---p 001e7000 08:02 4199299                    /lib/x86_64-linux-gnu/libc-2.27.so
7fd544c65000-7fd544c69000 r--p 001e7000 08:02 4199299                    /lib/x86_64-linux-gnu/libc-2.27.so
7fd544c69000-7fd544c6b000 rw-p 001eb000 08:02 4199299                    /lib/x86_64-linux-gnu/libc-2.27.so
7fd544c6b000-7fd544c6f000 rw-p 00000000 00:00 0 
7fd544c6f000-7fd544c86000 r-xp 00000000 08:02 4199336                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd544c86000-7fd544e85000 ---p 00017000 08:02 4199336                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd544e85000-7fd544e86000 r--p 00016000 08:02 4199336                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd544e86000-7fd544e87000 rw-p 00017000 08:02 4199336                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd544e87000-7fd544ea1000 r-xp 00000000 08:02 4199432                    /lib/x86_64-linux-gnu/libpthread-2.27.so
7fd544ea1000-7fd5450a0000 ---p 0001a000 08:02 4199432                    /lib/x86_64-linux-gnu/libpthread-2.27.so
7fd5450a0000-7fd5450a1000 r--p 00019000 08:02 4199432                    /lib/x86_64-linux-gnu/libpthread-2.27.so
7fd5450a1000-7fd5450a2000 rw-p 0001a000 08:02 4199432                    /lib/x86_64-linux-gnu/libpthread-2.27.so
7fd5450a2000-7fd5450a6000 rw-p 00000000 00:00 0 
7fd5450a6000-7fd5450a9000 r-xp 00000000 08:02 4199322                    /lib/x86_64-linux-gnu/libdl-2.27.so
7fd5450a9000-7fd5452a8000 ---p 00003000 08:02 4199322                    /lib/x86_64-linux-gnu/libdl-2.27.so
7fd5452a8000-7fd5452a9000 r--p 00002000 08:02 4199322                    /lib/x86_64-linux-gnu/libdl-2.27.so
7fd5452a9000-7fd5452aa000 rw-p 00003000 08:02 4199322                    /lib/x86_64-linux-gnu/libdl-2.27.so
7fd5452aa000-7fd5452b1000 r-xp 00000000 08:02 4199440                    /lib/x86_64-linux-gnu/librt-2.27.so
7fd5452b1000-7fd5454b0000 ---p 00007000 08:02 4199440                    /lib/x86_64-linux-gnu/librt-2.27.so
7fd5454b0000-7fd5454b1000 r--p 00006000 08:02 4199440                    /lib/x86_64-linux-gnu/librt-2.27.so
7fd5454b1000-7fd5454b2000 rw-p 00007000 08:02 4199440                    /lib/x86_64-linux-gnu/librt-2.27.so
7fd5454b2000-7fd54564f000 r-xp 00000000 08:02 4199362                    /lib/x86_64-linux-gnu/libm-2.27.so
7fd54564f000-7fd54584e000 ---p 0019d000 08:02 4199362                    /lib/x86_64-linux-gnu/libm-2.27.so
7fd54584e000-7fd54584f000 r--p 0019c000 08:02 4199362                    /lib/x86_64-linux-gnu/libm-2.27.so
7fd54584f000-7fd545850000 rw-p 0019d000 08:02 4199362                    /lib/x86_64-linux-gnu/libm-2.27.so
7fd545850000-7fd545877000 r-xp 00000000 08:02 4199271                    /lib/x86_64-linux-gnu/ld-2.27.so
7fd545877000-7fd54587e000 rw-p 00000000 00:00 0 
7fd54587e000-7fd54587f000 rw-p 00000000 00:00 0 
7fd54587f000-7fd54588b000 r--p 00000000 08:02 4861455                    /home/jaredpar/.nuget/packages/xunit.runner.console/2.3.1/tools/net452/xunit.runner.reporters.net452.dll
7fd54588b000-7fd5458a9000 r--p 00000000 08:02 4325755                    /usr/lib/mono/gac/System.Xml.Linq/4.0.0.0__b77a5c561934e089/System.Xml.Linq.dll
7fd5458a9000-7fd5458af000 r--p 00000000 08:02 4861450                    /home/jaredpar/.nuget/packages/xunit.runner.console/2.3.1/tools/net452/xunit.abstractions.dll
7fd5458af000-7fd5458f8000 r--p 00000000 08:02 4861456                    /home/jaredpar/.nuget/packages/xunit.runner.console/2.3.1/tools/net452/xunit.runner.utility.net452.dll
7fd5458f8000-7fd54590d000 r--p 00000000 08:02 4861451                    /home/jaredpar/.nuget/packages/xunit.runner.console/2.3.1/tools/net452/xunit.console.exe
7fd54590d000-7fd5459fb000 rw-p 00000000 00:00 0 
7fd5459fb000-7fd545a5a000 ---p 00000000 00:00 0 
7fd545a5a000-7fd545a62000 rw-p 00000000 00:00 0 
7fd545a62000-7fd545a76000 rw-p 00000000 00:00 0 
7fd545a76000-7fd545a77000 rw-s 00000000 00:18 6                          /dev/shm/mono.63619
7fd545a77000-7fd545a78000 r--p 00027000 08:02 4199271                    /lib/x86_64-linux-gnu/ld-2.27.so
7fd545a78000-7fd545a79000 rw-p 00028000 08:02 4199271                    /lib/x86_64-linux-gnu/ld-2.27.so
7fd545a79000-7fd545a7a000 rw-p 00000000 00:00 0 
7ffd4bbc4000-7ffd4bbcc000 ---p 00000000 00:00 0 
7ffd4c3a2000-7ffd4c3c3000 rw-p 00000000 00:00 0                          [stack]
7ffd4c3dc000-7ffd4c3df000 r--p 00000000 00:00 0                          [vvar]
7ffd4c3df000-7ffd4c3e1000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Native stacktrace:
@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Aug 27, 2018

@jaredpar this is a crash in the runtime, but unfortunately Native stacktrace: is empty (at the end of the log). Could you make sure to install gdb and mono-runtime-dbg and run again?

@jaredpar

This comment has been minimized.

Copy link
Contributor Author

@jaredpar jaredpar commented Aug 27, 2018

@lewurm I added those packages, re-ran the repro steps and the process just hangs at the Native stacktrace: line. I checked to see if it was busy generating data but the mono process appears to be idle. Let it run for a bit but nothing happened.

@jaredpar

This comment has been minimized.

Copy link
Contributor Author

@jaredpar jaredpar commented Aug 27, 2018

@lewurm also if you all have a guide on how to best report errors like this please feel free to point me to it. I looked around and didn't see one. There are likely to be a few more of these coming (have at least one I'm holding back until I make sure I have the right info on this issue).

@jaykrell

This comment has been minimized.

Copy link
Collaborator

@jaykrell jaykrell commented Aug 27, 2018

Is there a verbose switch, i.e. to report a command to run under a debugger? I'll look.

@jaykrell

This comment has been minimized.

Copy link
Collaborator

@jaykrell jaykrell commented Aug 28, 2018

Does this break the terminal afterward for anyone else?

Anyway. Our running of gdb is broken?
I had to fill in the right hand column by running mono again under gdb and plugging in the offsets.
Or maybe gdb failed to find symbols??

        mono(+0x1f494b) [0x55f9a067694b]       dump_native_stacktrace
	mono(+0x1f4bc0) [0x55f9a0676bc0]       mono_dump_native_crash_info
	mono(+0x11f8c2) [0x55f9a05a18c2]       mono_handle_native_crash
	mono(+0x34c77) [0x55f9a04b6c77]        mono_sigsegv_signal_handler
	/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fe48c2e6890] __restore_rt
	mono(+0x4667de) [0x55f9a08e87de]       major_block_is_evacuating
	mono(+0x49542b) [0x55f9a091742b]       major_scan_ptr_field_concurrent_with_evacuation 
	mono(+0x452d0e) [0x55f9a08d4d0e]       precisely_scan_objects_from
	mono(+0x453aed) [0x55f9a08d5aed]       scan_from_registered_roots
	mono(+0x453f0c) [0x55f9a08d5f0c]       job_scan_from_registered_roots
	mono(+0x4cdf4c) [0x55f9a094ff4c]       sgen_workers_enqueue_job
	mono(+0x454998) [0x55f9a08d6998]       enqueue_scan_from_roots_jobs
	mono(+0x455c4a) [0x55f9a08d7c4a]       major_copy_or_mark_from_roots
	mono(+0x456152) [0x55f9a08d8152]       major_start_collection
	mono(+0x4568ec) [0x55f9a08d88ec]       major_start_concurrent_collection
	mono(+0x456ece) [0x55f9a08d8ece]       sgen_perform_collection_inner
	mono(+0x4571df) [0x55f9a08d91df]       sgen_perform_collection
	mono(+0x456c32) [0x55f9a08d8c32]       sgen_ensure_free_space
	mono(+0x461bc6) [0x55f9a08e3bc6]       sgen_los_alloc_large_inner
	mono(+0x43de01) [0x55f9a08bfe01]       sgen_alloc_obj_nolock
	mono(+0x41f609) [0x55f9a08a1609]       mono_gc_alloc_vector
	[0x4032391e]

p/x 0x555555554000+0x4667de
0x5555559ba7de

Dump of assembler code for function major_block_is_evacuating:
   0x00005555559ba7bd <+0>:	push   %rbp
   0x00005555559ba7be <+1>:	mov    %rsp,%rbp
   0x00005555559ba7c1 <+4>:	mov    %rdi,-0x8(%rbp)
   0x00005555559ba7c5 <+8>:	mov    0x42b634(%rip),%rdx        # 0x555555de5e00 <evacuate_block_obj_sizes>
   0x00005555559ba7cc <+15>:	mov    -0x8(%rbp),%rax
   0x00005555559ba7d0 <+19>:	movzwl 0x2(%rax),%eax
   0x00005555559ba7d4 <+23>:	movzwl %ax,%eax
   0x00005555559ba7d7 <+26>:	shl    $0x2,%rax
   0x00005555559ba7db <+30>:	add    %rdx,%rax
   0x00005555559ba7de <+33>:	mov    (%rax),%eax                    << here
   0x00005555559ba7e0 <+35>:	test   %eax,%eax
   0x00005555559ba7e2 <+37>:	je     0x5555559ba809 <major_block_is_evacuating+76>
   0x00005555559ba7e4 <+39>:	mov    -0x8(%rbp),%rax
   0x00005555559ba7e8 <+43>:	movzbl 0xa(%rax),%eax
   0x00005555559ba7ec <+47>:	and    $0x4,%eax
   0x00005555559ba7ef <+50>:	test   %al,%al
   0x00005555559ba7f1 <+52>:	jne    0x5555559ba809 <major_block_is_evacuating+76>
   0x00005555559ba7f3 <+54>:	mov    -0x8(%rbp),%rax
   0x00005555559ba7f7 <+58>:	movzbl 0xa(%rax),%eax
   0x00005555559ba7fb <+62>:	and    $0x8,%eax
   0x00005555559ba7fe <+65>:	test   %al,%al
   0x00005555559ba800 <+67>:	jne    0x5555559ba809 <major_block_is_evacuating+76>
   0x00005555559ba802 <+69>:	mov    $0x1,%eax
   0x00005555559ba807 <+74>:	jmp    0x5555559ba80e <major_block_is_evacuating+81>
   0x00005555559ba809 <+76>:	mov    $0x0,%eax
   0x00005555559ba80e <+81>:	pop    %rbp
   0x00005555559ba80f <+82>:	retq   
@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Aug 28, 2018

it's either a GC issue or memory corruption. /cc @BrzVlad

@jaredpar jaredpar referenced this issue Aug 28, 2018
5 of 5 tasks complete
@BrzVlad BrzVlad self-assigned this Aug 29, 2018
BrzVlad added a commit to BrzVlad/mono that referenced this issue Aug 29, 2018
This could lead to crashes when having strings with size at the LOS boundary.

mono#10309
@marek-safar marek-safar added this to the 2018-06 (5.16.xx) milestone Aug 29, 2018
monojenkins added a commit that referenced this issue Aug 30, 2018
[sgen] Fix string size inconsistency between alloc/scan

This could lead to crashes when having strings with size at the LOS boundary.

#10309



<!--
Thank you for your Pull Request!

If you are new to contributing to Mono, please try to do your best at conforming to our coding guidelines http://www.mono-project.com/community/contributing/coding-guidelines/ but don't worry if you get something wrong. One of the project members will help you to get things landed.

Does your pull request fix any of the existing issues? Please use the following format: Fixes #issue-number
-->
monojenkins added a commit to monojenkins/mono that referenced this issue Aug 31, 2018
This could lead to crashes when having strings with size at the LOS boundary.

mono#10309
monojenkins added a commit to monojenkins/mono that referenced this issue Aug 31, 2018
This could lead to crashes when having strings with size at the LOS boundary.

mono#10309
@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Aug 31, 2018

@BrzVlad does #10355 fix this issue, or are there multiple problems?

@BrzVlad

This comment has been minimized.

Copy link
Member

@BrzVlad BrzVlad commented Aug 31, 2018

I ran it a couple of times without any crashes on top of mono master.

@jaredpar If you encounter any additional crashes it might be worthwhile to also try nightly mono.

@BrzVlad BrzVlad closed this Aug 31, 2018
BrzVlad added a commit to monojenkins/mono that referenced this issue Aug 31, 2018
This could lead to crashes when having strings with size at the LOS boundary.

mono#10309
@jaredpar

This comment has been minimized.

Copy link
Contributor Author

@jaredpar jaredpar commented Aug 31, 2018

@BrzVlad what is the general stability of the nightly feed? Currently we use the stable feed to drive our Mono uni test CI. If the nightly feed is fairly stable we may consider switching to that. I'm sure we'll hit a few other small issues as we move our tests over.

@jaredpar

This comment has been minimized.

Copy link
Contributor Author

@jaredpar jaredpar commented Aug 31, 2018

@BrzVlad when will the updated package be available? When moving to the nightly feed and updating the build I get is from August 14th.

jaredpar@jaredpar-ubuntu1:~/code/roslyn$ mono --version
Mono JIT compiler version 5.19.0.72 (tarball Tue Aug 14 10:31:03 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	Interpreter:   yes
	LLVM:          yes(4)
	Suspend:       hybrid
	GC:            sgen (concurrent by default)

That, obviously, doesn't have your fix. I tried it out anyways and I'm getting a different SIGSEGV (this time in object.AllocSmall). Wanted to wait until I had your patch though before I filed a new bug.

luhenry added a commit that referenced this issue Sep 1, 2018
This could lead to crashes when having strings with size at the LOS boundary.

#10309
luhenry added a commit that referenced this issue Sep 1, 2018
This could lead to crashes when having strings with size at the LOS boundary.

#10309
@BrzVlad

This comment has been minimized.

Copy link
Member

@BrzVlad BrzVlad commented Sep 3, 2018

@jaredpar I'd say the nightly builds aren't too bad, you can try to see if it works for you. I think we used to update the packages on a weekly basis at some point. @directhex ?

@directhex

This comment has been minimized.

Copy link
Contributor

@directhex directhex commented Sep 4, 2018

Nightly is dispatched nightly, but fairly brittle - it needs to successfully build on 29 architecture-distribution combinations, and some pieces (like exported symbol checking) require manual fixing in the event of changes. i'll take a peek now.

@jaredpar

This comment has been minimized.

Copy link
Contributor Author

@jaredpar jaredpar commented Sep 4, 2018

@directhex where can I go to see the status of the nightly builds? I'm eager to try this fix out and see if I can make more progress porting the C# tests over.

@marek-safar

This comment has been minimized.

Copy link
Member

@marek-safar marek-safar commented Sep 4, 2018

We have backported that down to Mono 5.16 which we are previewing and the update with the fix should come out soon-ish

/cc @directhex

@jaredpar

This comment has been minimized.

Copy link
Contributor Author

@jaredpar jaredpar commented Sep 4, 2018

@marek-safar where does 5.16 come from? The feeds I've been looking at:

  • stable: 5.14
  • nightly: 5.19

Is there a third feed to look at?

@directhex

This comment has been minimized.

Copy link
Contributor

@directhex directhex commented Sep 5, 2018

A new nightly, 5.19.0.290, made its way through the Debian packaging pipeline 8 hours ago.

5.16 builds are in the preview repo. 5.16.0.138 is the latest published Debian/Ubuntu version, from September 2nd.

@marek-safar

This comment has been minimized.

Copy link
Member

@marek-safar marek-safar commented Sep 6, 2018

EgorBo added a commit to EgorBo/mono that referenced this issue Sep 10, 2018
This could lead to crashes when having strings with size at the LOS boundary.

mono#10309
@jaredpar

This comment has been minimized.

Copy link
Contributor Author

@jaredpar jaredpar commented Sep 24, 2018

I'm still seeing this issue in 5.19.0.72. By running the C# semantic tests in a loop 100 times it will eventually crash with the same stack trace:

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <0x0007d>
  at (wrapper alloc) object.AllocVector (intptr,intptr) <0x0018b>
  at System.IO.MemoryStream.set_Capacity (int) [0x00057] in <5ae58f886911473389a1a046017bd126>:0
  at System.IO.MemoryStream.EnsureCapacity (int) [0x00061] in <5ae58f886911473389a1a046017bd126>:0
  at System.IO.MemoryStream.Write (byte[],int,int) [0x000b3] in <5ae58f886911473389a1a046017bd126>:0
  at System.IO.BinaryWriter.Write (string) [0x0007b] in <5ae58f886911473389a1a046017bd126>:0
  at System.Runtime.Serialization.Formatters.Binary.__BinaryWriter.WriteString (string) [0x00000] in <5ae58f886911473389a1a046017bd126>:0
  at System.Runtime.Serialization.Formatters.Binary.BinaryObjectString.Write (System.Runtime.Serialization.Formatters.Binary.__BinaryWriter) [0x00013] in <5ae58f886911473389a1a046017bd126>:0
  at System.Runtime.Serialization.Formatters.Binary.__BinaryWriter.WriteObjectString (int,string) [0x00026] in <5ae58f886911473389a1a046017bd126>:0
  at System.Runtime.Serialization.Formatters.Binary.__BinaryWriter.WriteMemberString (System.Runtime.Serialization.Formatters.Binary.NameInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,string) [0x0000d] in <5ae58f886911473389a1a046017bd126>:0
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteString (System.Runtime.Serialization.Formatters.Binary.NameInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,object) [0x00034] in <5ae58f886911473389a1a046017bd126>:0
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteKnownValueClass (System.Runtime.Serialization.Formatters.Binary.NameInfo,System.Runtime.Serialization.Formatters.Binary.NameInfo,object) [0x0000d] in <5ae58f886911473389a1a046017bd126>:0

I've pushed a new branch to my fork, repro/mono-crash-4, where the C# semantic tests pass under Mono. Running it in a loop will cause this crash to hit.

@marek-safar

This comment has been minimized.

Copy link
Member

@marek-safar marek-safar commented Sep 25, 2018

@BrzVlad could you confirm that? I think this is similar to what we are seeing on our CI when Roslyn crashes

@BrzVlad

This comment has been minimized.

Copy link
Member

@BrzVlad BrzVlad commented Sep 25, 2018

@jaredpar I am running these tests for several hours now, run 75, with latest nightly mono (5.19.0.290) and there are no crashes yet. Could you also add a native stacktrace. Maybe run with MONO_DEBUG=suspend-on-sigsegv and attach after it crashes.

@jaredpar

This comment has been minimized.

Copy link
Contributor Author

@jaredpar jaredpar commented Sep 25, 2018

@BrzVlad unfortunately I updated my system this morning so I'm on 5.19.0.290 as well hence I can't grab the trace for 5.19.0.72. Let me re-run my scenario script though and see if I see the crash going away as well on this version.

@jaredpar

This comment has been minimized.

Copy link
Contributor Author

@jaredpar jaredpar commented Sep 25, 2018

@BrzVlad been running the 290 build for 30 minutes now and I'm not seeing this crash anymore. Nor am I seeing the other one I hadn't reported yet. I'm going to let the 100 iterations run to conclusion but I think this is solved now. Previously I couldn't get past 3 runs without hitting it.

Good news though: I should be able to turn on the C# semantic tests for Mono now in Roslyn CI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.