Outer Loop OpenSUSE 42.1 Debug: Test "System.Data.SqlClient.Tests.DiagnosticTest.ExecuteReaderTest" failed with "Xunit.Sdk.EqualException"

[KristinXie1]

Failed Test: System.Data.SqlClient.Tests.DiagnosticTest.ExecuteReaderTest

Detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_opensuse42.1_debug/lastFailedBuild/consoleText

Message:

System.Data.SqlClient.Tests.DiagnosticTest.ExecuteReaderTest [FAIL]
        Assert.Equal() Failure
        Expected: b6bade04-7eda-4678-a918-24502a6d7149
        Actual:   0893cbea-49ac-4282-9385-0a5060b6ee79

Stack Trace:

/mnt/resource/j/workspace/dotnet_corefx/master/outerloop_opensuse42.1_debug/src/System.Data.SqlClient/tests/FunctionalTests/DiagnosticTest.cs(458,0): at System.Data.SqlClient.Tests.DiagnosticTest.<>c__DisplayClass27_0.<CollectStatisticsDiagnostics>b__0(KeyValuePair`2 kvp)
           /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_opensuse42.1_debug/src/System.Data.SqlClient/tests/FunctionalTests/FakeDiagnosticListenerObserver.cs(35,0): at System.Data.SqlClient.Tests.FakeDiagnosticListenerObserver.FakeDiagnosticSourceWriteObserver.OnNext(KeyValuePair`2 value)
           /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_opensuse42.1_debug/src/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/DiagnosticListener.cs(192,0): at System.Diagnostics.DiagnosticListener.Write(String name, Object value)
           /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_opensuse42.1_debug/src/System.Data.SqlClient/src/System/Data/SqlClient/SqlClientDiagnosticListenerExtensions.cs(66,0): at System.Data.SqlClient.SqlClientDiagnosticListenerExtensions.WriteCommandAfter(DiagnosticListener this, Guid operationId, SqlCommand sqlCommand, String operation)
           /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_opensuse42.1_debug/src/System.Data.SqlClient/src/System/Data/SqlClient/SqlCommand.cs(1370,0): at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
           /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_opensuse42.1_debug/src/System.Data.SqlClient/src/System/Data/SqlClient/SqlCommand.cs(1331,0): at System.Data.SqlClient.SqlCommand.ExecuteReader()
           /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_opensuse42.1_debug/src/System.Data.SqlClient/tests/FunctionalTests/DiagnosticTest.cs(119,0): at System.Data.SqlClient.Tests.DiagnosticTest.<ExecuteReaderTest>b__10_0()
           /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_opensuse42.1_debug/src/System.Data.SqlClient/tests/FunctionalTests/DiagnosticTest.cs(585,0): at System.Data.SqlClient.Tests.DiagnosticTest.CollectStatisticsDiagnostics(Action sqlOperation)
           /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_opensuse42.1_debug/src/System.Data.SqlClient/tests/FunctionalTests/DiagnosticTest.cs(110,0): at System.Data.SqlClient.Tests.DiagnosticTest.ExecuteReaderTest()

[danmoseley]

@saurabh500

[saurabh500]

@tstringer Can you or someone from your team look at this issue?

[danmoseley]

@tstringer ?

[trstringer]

@JiayiLi1 was it just in openSUSE that you ran into this issue? I don't see why this would be a distro-specific issue. Do you have any other surrounding information on this? Did your tests run successfully on Ubuntu?

[tarekgh]

https://ci.dot.net/job/dotnet_corefx/job/master/job/portablelinux_debug_prtest/471/

[KristinXie1]

@tstringer it is just in openSUSE, If I ran into this issue on other OS system, I will update the comment here.

[trstringer]

@JiayiLi1 what are the .net core build steps for openSUSE 42.x? I don't see anything documented. Do you have a quick list of zypper packages required on the host?

[trstringer]

@JiayiLi1 can you please link me to the test failure instead of just the console output?

If this is related to the issue that @tarekgh pointed out, then there is an underlying problem outside of the diagnostics:

System.Data.SqlClient.SqlException : A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 2 - Connection was terminated)

That's a transport layer issue.

cc @saurabh500

[KristinXie1]

@tstringer The link is https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_opensuse42.1_debug/77/

It is failed on build#77, but this page could not be found now.

[trstringer]

@JiayiLi1 Can you please reproduce the issue through CI so that we can have the full picture of the failure?

[KristinXie1]

@danmosemsft Can you do a favor? Since I couldn't navigate to the previous builds, the page could not be found, https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_opensuse42.1_debug/ there only shows recent 20 builds, but this issue is repro on build#77.

[danmoseley]

@JiayiLi1 after a while (?2 weeks) the log are deleted. If the log is gone, there is nothing that can be done. If there is a newer build with a failure, please link that. If there is not, please close this until there is one.

[KristinXie1]

@danmosemsft Thanks Dan, I will check this issue on next few builds.

[stephentoub]

Occured here on macOS:
https://ci.dot.net/job/dotnet_corefx/job/master/job/osx_release_prtest/3525/

[stephentoub]

Failed again here:
https://ci.dot.net/job/dotnet_corefx/job/master/job/portablelinux_release/296/

[KristinXie1]

This issue is repro on InnerLoop_PortableLinux_release (build#296).

Detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/portablelinux_release/296/testReport/System.Data.SqlClient.Tests/DiagnosticTest/ExecuteReaderTest/

Message:

Assert.Equal() Failure
        Expected: 402b697d-f354-43b5-8d15-7048ae97092e
        Actual:   f2f6ba90-48ca-4fb2-9572-7d12b0387246

Stack Trace:

/mnt/resource/j/workspace/dotnet_corefx/master/portablelinux_release/src/System.Data.SqlClient/tests/FunctionalTests/DiagnosticTest.cs(460,0): at System.Data.SqlClient.Tests.DiagnosticTest.<>c__DisplayClass27_0.<CollectStatisticsDiagnostics>b__0(KeyValuePair`2 kvp)
           /mnt/resource/j/workspace/dotnet_corefx/master/portablelinux_release/src/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/DiagnosticListener.cs(209,0): at System.Diagnostics.DiagnosticListener.Write(String name, Object value)
           /mnt/resource/j/workspace/dotnet_corefx/master/portablelinux_release/src/System.Data.SqlClient/src/System/Data/SqlClient/SqlClientDiagnosticListenerExtensions.cs(78,0): at System.Data.SqlClient.SqlClientDiagnosticListenerExtensions.WriteCommandAfter(DiagnosticListener this, Guid operationId, SqlCommand sqlCommand, String operation)
           /mnt/resource/j/workspace/dotnet_corefx/master/portablelinux_release/src/System.Data.SqlClient/src/System/Data/SqlClient/SqlCommand.cs(1373,0): at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
           /mnt/resource/j/workspace/dotnet_corefx/master/portablelinux_release/src/System.Data.SqlClient/src/System/Data/SqlClient/SqlCommand.cs(1331,0): at System.Data.SqlClient.SqlCommand.ExecuteReader()

[KristinXie1]

This issue failed again on: https://ci.dot.net/job/dotnet_corefx/job/master/job/centos7.1_debug/1104/testReport/

[saurabh500]

@tstringer all except one failure was the guid mismatch from the diagnostics test.
I have seen the tcp exception and looking into it. Can you diagnose the guid mismatch failure?

[saurabh500]

Since there are multiple listeners beings installed for diagnostics and the tests are executing in parallel, each listener can listen to all commands being executed.
Since the tests are executing in parallel, one of the listeners is getting a notification about the beginning of one of the commands and then the notification for the completion of another command. As a result the Guid retrieved before command execution doesn't match the Guid of the command post execution.

This can be solved by executing the tests in serial instead of parallel.
I have a pull request open dotnet/corefx#15798 where I am trying to make the different DiagnosticsTests not interfere with each other. I will try to complete asap and it should take care of this issue of Guid mismatch as well.

[JonHanna]

https://ci.dot.net/job/dotnet_corefx/job/master/job/osx_release_prtest/3540/ is OS X but seems perhaps a variant of the same

[stephentoub]

Failed again here:
https://ci.dot.net/job/dotnet_corefx/job/master/job/portablelinux_debug_prtest/916/

[danmoseley]

https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_osx_release/lastCompletedBuild/testReport/System.Data.SqlClient.Tests/DiagnosticTest/ExecuteReaderTest/

[stephentoub]

This is causing a bunch of failures. @saurabh500, I know you have a larger set of changes in the works, but I'm going to submit a fix for this particular issue to unblock things.