Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Security.Cryptography.OpenSsl.Tests failing on CI runs #2176

Closed
jaredpar opened this issue Jan 25, 2020 · 32 comments · Fixed by #32255
Closed

System.Security.Cryptography.OpenSsl.Tests failing on CI runs #2176

jaredpar opened this issue Jan 25, 2020 · 32 comments · Fixed by #32255
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' untriaged New issue has not been triaged by the area owner

Comments

@jaredpar
Copy link
Member

Tests are crashing on several OSX runs

===========================================================================================================
/private/tmp/helix/working/BB380A7A/w/CADB0A98/e /private/tmp/helix/working/BB380A7A/w/CADB0A98/e
  Discovering: System.Security.Cryptography.OpenSsl.Tests (method display = ClassAndMethod, method display options = None)
No usable version of libssl was found
./RunTests.sh: line 161: 65451 Abort trap: 6           "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -notrait category=nonnetcoreapptests -notrait category=nonosxtests $RSP_FILE
/private/tmp/helix/working/BB380A7A/w/CADB0A98/e
----- end Thu Jan 23 05:44:06 PST 2020 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

Builds

Test Runs

  • netcoreapp5.0-OSX-Release-x64-CoreCLR_release-OSX.1013.Amd64.Open
  • netcoreapp5.0-OSX-Release-x64-CoreCLR_release-OSX.1014.Amd64.Open
  • netcoreapp5.0-OSX-Release-x64-Mono_release-OSX.1014.Amd64.Open
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added area-System.Security untriaged New issue has not been triaged by the area owner labels Jan 25, 2020
@safern
Copy link
Member

safern commented Jan 25, 2020

This was supposed to be fixed by @ilyas1974 already. OpenSSL was not installed correctly. He installed it now in all the OSX machines. @ilyas1974 could you please follow up in this?

@ilyas1974
Copy link

Yes, I will follow up on what is going on with these systems.

@safern
Copy link
Member

safern commented Jan 31, 2020

@ilyas1974 I haven't seen this failure anymore, did you follow up? I just want to confirm we can close this issue.

@ilyas1974
Copy link

Apologize for not updating this thread early. DDFUN is currently working on going though all the systems to verify this functionality. Their work item for this task can be for at http://ddweb/TechEase/TicketEdit.aspx?id=1636188. I believe a majority of the systems have OpenSSL configured properly - there are a few that do not (what will be resolved with the associated work item.

@safern
Copy link
Member

safern commented Jan 31, 2020

Apologize for not updating this thread early.

No problem. Would you mind getting back whenever this is finished so that we can close this issue?

@jkotas
Copy link
Member

jkotas commented Feb 3, 2020

Hit by #31641

@safern
Copy link
Member

safern commented Feb 3, 2020

@ilyas1974
Copy link

Reaching out to DDFUN as they are performing the work to get the current status on this.

@ilyas1974
Copy link

We have reached out to the tech and are currently waiting for a reply. I should have an update for you on the current status by tomorrow morning.

/cc @JpratherMS

@jkotas jkotas added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Feb 4, 2020
@ilyas1974
Copy link

@JpratherMS, is there an update on the status of the OpenSSL work? Last I heard there were only about 6 systems that were not configured correctly. If we can get that list, I can remove them from rotation until DDFUN is able to fix the OpenSSL issues.

@JpratherMS
Copy link

The affected machines have been fixed, please let me know if this issue persists.

@safern
Copy link
Member

safern commented Feb 4, 2020

Thanks @JpratherMS and @ilyas1974 -- I'm going to close this issue and if someone hits it again we can re-open.

@safern safern closed this as completed Feb 4, 2020
@safern
Copy link
Member

safern commented Feb 4, 2020

@JpratherMS just happened again on: dci-mac-build-103.local machine. I'll re-open this until it is clear if there are some machines with this issue still.

https://helix.dot.net/api/2019-06-17/jobs/fd9733fb-53cf-41b8-a9ce-f450b21ed229/workitems/System.Security.Cryptography.OpenSsl.Tests/console

@safern safern reopened this Feb 4, 2020
@JpratherMS
Copy link

Thanks for reaching out, we found an issue with our process in discovering the ones that are not configured correctly. We are working on resolving that now, I will update again when that is completed.

@jaredpar
Copy link
Member Author

jaredpar commented Feb 5, 2020

Data from the last 100 builds (CI and PR)

Builds

Build Pull Request Test Failure Count
#507937 #1787 10
#507961 #31734 1
#508096 Rolling 1
#508266 #2087 1
#508472 #2275 1
#508547 #31765 1
#508623 #31807 1
#508780 #31765 1
#508787 #27540 1
#508835 #5523 1
#508862 #2140 1
#508923 #31819 1
#509133 #31826 1
#509169 #31802 1
#509342 #1612 20
#509347 #2259 1
#509362 Rolling 1
#509423 #1787 5
#509654 #31838 5
#509747 Rolling 1
#510085 #31854 1
#510294 #31814 1

Configurations

  • netcoreapp5.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open
  • netcoreapp5.0-OSX-Debug-x64-CoreCLR_release-OSX.1013.Amd64.Open
  • netcoreapp5.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open
  • netcoreapp5.0-OSX-Debug-x64-Mono_release-OSX.1013.Amd64.Open
  • netcoreapp5.0-OSX-Debug-x64-Mono_release-OSX.1014.Amd64.Open
  • netcoreapp5.0-OSX-Release-x64-Mono_release-OSX.1014.Amd64.Open

Helix Logs

Build Pull Request Console Core Test Results
#507937 #1787 console.338b1956.log
#507937 #1787 console.b76cc7b6.log
#507937 #1787 console.55abdc57.log
#507937 #1787 console.46da5f7c.log
#507937 #1787 console.df261f95.log
#507937 #1787 console.8558d07e.log
#507937 #1787 console.b5f5c68c.log
#507937 #1787 console.aba60104.log
#507937 #1787 console.86a05f30.log
#507937 #1787 console.13060cb7.log
#507961 #31734 console.075b1564.log core.31566
#508096 Rolling console.6d912e31.log core.37103
#508266 #2087 console.79f77369.log core.42773
#508472 #2275 console.ceed47a0.log core.44769
#508547 #31765 console.8a957c50.log core.47155
#508623 #31807 console.f30e2088.log core.48721
#508780 #31765 console.0b771628.log core.4026
#508787 #27540 console.8d8e1010.log core.4686
#508835 #5523 console.9a2025a2.log core.5188
#508862 #2140 console.dc340c3f.log core.5999
#508923 #31819 console.923c5d72.log core.7127
#509133 #31826 console.185f15e8.log core.9506
#509169 #31802 console.3ed533eb.log core.11807
#509342 #1612 console.e7f10be1.log core.1000.22
#509342 #1612 console.0683c3ae.log core.1000.57
#509342 #1612 console.6f815067.log core.1000.5878
#509342 #1612 console.9dee786b.log core.1000.6159
#509342 #1612 console.399a5c79.log core.1000.7512
#509342 #1612 console.17059d0d.log core.1000.11287
#509342 #1612 console.140aba91.log core.1000.7884
#509342 #1612 console.905f82c4.log
#509342 #1612 console.781e76fc.log
#509342 #1612 console.2ded3daa.log
#509342 #1612 console.329188b9.log
#509342 #1612 console.50d9e385.log
#509342 #1612 console.fd7064ef.log
#509342 #1612 console.265efa80.log
#509342 #1612 console.660f0d2f.log
#509342 #1612 console.cc8940d2.log core.50894
#509342 #1612 console.7fa942a1.log core.17123
#509342 #1612 console.0d7eed7c.log
#509342 #1612 console.4fcd040b.log
#509342 #1612 console.0b7b0aef.log
#509347 #2259 console.886e5091.log core.14348
#509362 Rolling console.0584d0a8.log core.12129
#509423 #1787 console.1eb1dc5f.log
#509423 #1787 console.dc372806.log
#509423 #1787 console.b32d0cb4.log
#509423 #1787 console.11f19c52.log
#509423 #1787 console.53b4997f.log
#509654 #31838 console.82475420.log
#509654 #31838 console.5f2b49ac.log
#509654 #31838 console.0edaed0e.log
#509654 #31838 console.6469a128.log
#509654 #31838 console.c328dcce.log
#509747 Rolling console.749984ce.log core.20578
#510085 #31854 console.e56c99e9.log core.26802
#510294 #31814 console.b5a100a6.log core.27087

@ahsonkhan
Copy link
Member

I am seeing other System.Security.Cryptography.* tests failing on netcoreapp5.0-OSX-Debug-x64-Mono_release-OSX.1014.Amd64.Open with (what looks like) the same error:
https://helix.dot.net/api/2019-06-17/jobs/6da71858-a172-4863-a07e-c2be8c8dd875/workitems/System.Security.Cryptography.Algorithms.Tests/console

===========================================================================================================
/private/tmp/helix/working/A6850955/w/BD6409D4/e /private/tmp/helix/working/A6850955/w/BD6409D4/e
  Discovering: System.Security.Cryptography.Algorithms.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.Algorithms.Tests (found 1234 of 1239 test cases)
  Starting:    System.Security.Cryptography.Algorithms.Tests (parallel test collections = on, max threads = 4)
No usable version of libssl was found

…
…
…

=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at System.Security.Cryptography.AesCcm:Encrypt <0x00112>
	  at System.Security.Cryptography.AesCcm:Encrypt <0x00462>
	  at System.Security.Cryptography.Algorithms.Tests.AesCcmTests:AesCcmNistTestsTamperCiphertext <0x00312>
	  at <Module>:runtime_invoke_void_object <0x000b0>
	  at <unknown> <0xffffffff>
	  at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x000b8>
	  at System.Reflection.RuntimeMethodInfo:Invoke <0x0016a>
	  at System.Reflection.MethodBase:Invoke <0x00058>
	  at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x0005a>
	  at <<InvokeTestMethodAsync>b__1>d:MoveNext <0x0040a>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0010a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x00042>
	  at <>c__DisplayClass48_1:<InvokeTestMethodAsync>b__1 <0x000de>
	  at <AggregateAsync>d__4:MoveNext <0x000f5>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0010a>
	  at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x000e2>
	  at <>c__DisplayClass48_1:<InvokeTestMethodAsync>b__0 <0x0018a>
	  at <RunAsync>d__9:MoveNext <0x000a0>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00102>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000da>
	  at <InvokeTestMethodAsync>d__48:MoveNext <0x003ba>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00112>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
	  at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x0010e>
	  at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x0013a>
	  at <<RunAsync>b__47_0>d:MoveNext <0x00779>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0011a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
	  at Xunit.Sdk.TestInvoker`1:<RunAsync>b__47_0 <0x000e6>
	  at <RunAsync>d__10`1:MoveNext <0x000cb>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00102>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000fa>
	  at Xunit.Sdk.TestInvoker`1:RunAsync <0x0012a>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x0010a>
	  at <InvokeTestAsync>d__4:MoveNext <0x00210>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00112>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x000ea>
	  at <>c__DisplayClass43_0:<RunAsync>b__0 <0x00043>
	  at <RunAsync>d__10`1:MoveNext <0x000ab>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0010a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x0015e>
	  at <RunAsync>d__43:MoveNext <0x005b2>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0011a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
	  at Xunit.Sdk.TestRunner`1:RunAsync <0x0010e>
	  at <RunTestAsync>d__11:MoveNext <0x00242>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0012a>
	  at Xunit.Sdk.XunitTheoryTestCaseRunner:RunTestAsync <0x000c2>
	  at <RunAsync>d__19:MoveNext <0x003ca>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00112>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
	  at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x0010e>
	  at Xunit.Sdk.XunitTheoryTestCase:RunAsync <0x000ea>
	  at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x000bd>
	  at <RunTestCasesAsync>d__32:MoveNext <0x001f5>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0011a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
	  at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x0010e>
	  at <RunAsync>d__31:MoveNext <0x001f8>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0010a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
	  at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x00106>
	  at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x0012a>
	  at <RunTestMethodsAsync>d__38:MoveNext <0x00b00>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0012a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
	  at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x0011e>
	  at <RunAsync>d__37:MoveNext <0x00405>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00112>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
	  at Xunit.Sdk.TestClassRunner`1:RunAsync <0x0010e>
	  at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x00132>
	  at <RunTestClassesAsync>d__28:MoveNext <0x004ef>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0011a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
	  at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x0010e>
	  at <RunAsync>d__27:MoveNext <0x00402>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00112>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
	  at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x0010e>
	  at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x0010a>
	  at <>c__DisplayClass14_2:<RunTestCollectionsAsync>b__2 <0x00067>
	  at System.Threading.Tasks.Task`1:InnerInvoke <0x00097>
	  at <>c:<.cctor>b__274_0 <0x00036>
	  at System.Threading.ExecutionContext:RunInternal <0x00118>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00312>
	  at System.Threading.Tasks.Task:ExecuteEntry <0x000f2>
	  at <>c:<.cctor>b__8_0 <0x0006a>
	  at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x0006e>
	  at <>c__DisplayClass11_0:<WorkerThreadProc>b__0 <0x0005a>
	  at System.Threading.ExecutionContext:RunInternal <0x00118>
	  at System.Threading.ExecutionContext:Run <0x00072>
	  at System.Object:lambda_method2 <0x0008a>
	  at Xunit.Sdk.ExecutionContextHelper:Run <0x00081>
	  at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x00262>
	  at <>c:<QueueUserWorkItem>b__5_0 <0x00076>
	  at System.Threading.Tasks.Task:InnerInvoke <0x000b5>
	  at <>c:<.cctor>b__274_0 <0x00036>
	  at System.Threading.ExecutionContext:RunInternal <0x00118>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00312>
	  at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x000ba>
	  at <>c:<.cctor>b__10_0 <0x0006a>
	  at System.Threading.Thread:StartCallback <0x0012c>
	  at System.Object:runtime_invoke_void__this__ <0x000b0>
=================================================================
./RunTests.sh: line 161: 32067 Abort trap: 6           (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.Algorithms.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.Algorithms.Tests.deps.json xunit.console.dll System.Security.Cryptography.Algorithms.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -notrait category=nonnetcoreapptests -notrait category=nonosxtests $RSP_FILE
/private/tmp/helix/working/A6850955/w/BD6409D4/e
----- end Tue Feb 4 18:00:28 PST 2020 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

From #31772 Libraries Test Run release mono OSX x64 Debug leg in https://dev.azure.com/dnceng/public/_build/results?buildId=507900&view=logs&j=c6f8dc49-92a1-5760-c098-ba97b8142bfb&t=22b0078b-0469-5ba6-8725-2121fdbae049

https://dev.azure.com/dnceng/public/_build/results?buildId=507900&view=ms.vss-test-web.build-test-results-tab&runId=16121704&paneView=debug&resultId=172169

@safern
Copy link
Member

safern commented Feb 5, 2020

Yes it is the same error, mono just crashes in a different way and prints the whole call stack, but it is caused by the same: No usable version of libssl was found.

Thanks @ahsonkhan

@jaredpar
Copy link
Member Author

jaredpar commented Feb 6, 2020

Looks like right now this is failing 14% of all runtime jobs: CI and PRs. Particularly though it's for OSX10.13 and OSX10.14. Think at this point we need to pull those configurations until we can get the underlying issue fixed.

@safern
Copy link
Member

safern commented Feb 6, 2020

Think at this point we need to pull those configurations until we can get the underlying issue fixed.

I was already working on that, talked with @JpratherMS offline and he hopes the work to update the machines will be done by EOD. He will update if so, if it is not ready I'll put up my PR to disable the affected tests on OSX.

@JpratherMS
Copy link

Apologies for the delay in getting this resolved, we had some miscommunication on our side and we are going through the rest of the machines to fix them. ETA is tomorrow morning.

@CoffeeFlux
Copy link
Contributor

If this isn't fixed by EOD, I'd really appreciate disabling the tests. I'm hitting this constantly, even sometimes on multiple re-runs. I hit it just a little while ago, so at least as of this afternoon it is still not fixed.

@safern
Copy link
Member

safern commented Feb 6, 2020

Thanks @CoffeeFlux... Do you have the build when you hit it?

@JpratherMS I guess this got delayed a little bit ?

@CoffeeFlux
Copy link
Contributor

CoffeeFlux commented Feb 6, 2020

I've been seeing it frequently on #31814 - I think this time it took 3 runs to get it green? Looks like Koplinger hit it a few hours ago as well.

@jaredpar
Copy link
Member Author

jaredpar commented Feb 6, 2020

I updated the failed build list for all the failures since last night. Looks like it's hit a number of new PRs.

@jaredpar
Copy link
Member Author

jaredpar commented Feb 6, 2020

@JpratherMS where is the core-eng issue we're using to track the OSX configuration update?

@safern
Copy link
Member

safern commented Feb 6, 2020

@JpratherMS
Copy link

We have people actively working on fixing these. We are close but there are quite a few machines to manually check and resolve.

@safern
Copy link
Member

safern commented Feb 6, 2020

@JpratherMS do we have an ETA? To see if it is worth merging my PR to disable.

@JpratherMS
Copy link

I believe we are very close, just rerunning the automation to verify now.

@safern
Copy link
Member

safern commented Feb 13, 2020

@JpratherMS is this done? I would like to re-enable the tests.

@JpratherMS
Copy link

Apologies, yes this has been completed. Please enable and test and let me know if you see further issues

@safern
Copy link
Member

safern commented Feb 13, 2020

Thanks @JpratherMS will put up a PR to do so.

@ghost ghost locked as resolved and limited conversation to collaborators Dec 11, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' untriaged New issue has not been triaged by the area owner
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants