[Android] System.Security.Cryptography.X509Certificates ExportCertificatePems_MultiCert Failure

[mdh1418]

<test name="System.Security.Cryptography.X509Certificates.Tests.CollectionTests.ExportCertificatePems_MultiCert" type="System.Security.Cryptography.X509Certificates.Tests.CollectionTests" method="ExportCertificatePems_MultiCert" time="0.0181399" result="Fail">
    <failure exception-type="Xunit.Sdk.EqualException">
        <message><![CDATA[Assert.Equal() Failure\nExpected: X509Certificate2[] [[Subject]\n  CN=tiny\n\n[Issuer]\n  CN=tiny\n\n[Serial Number]\n  00F52B54E599C198CE\n\n[Not Before]\n  9/15/2021 10:02:43 PM\n\n[Not After]\n  9/15/2021 10:02:43 PM\n\n[Thumbprint]\n  CB437BDB1C0DC99C864EBEDB7E1B569C0A0C4CF4\n, [Subject]\n  CN=tiny\n\n[Issuer]\n  CN=tiny\n\n[Serial Number]\n  008382F7436468D2F7\n\n[Not Before]\n  9/15/2021 10:05:10 PM\n\n[Not After]\n  9/15/2021 10:05:10 PM\n\n[Thumbprint]\n  0CC63492B8A2EE3B7D3085DF575A8B67B69261D7\n]\nActual:   X509Certificate2[] [[Subject]\n  CN=tiny\n\n[Issuer]\n  CN=tiny\n\n[Serial Number]\n  008382F7436468D2F7\n\n[Not Before]\n  9/15/2021 10:05:10 PM\n\n[Not After]\n  9/15/2021 10:05:10 PM\n\n[Thumbprint]\n  0CC63492B8A2EE3B7D3085DF575A8B67B69261D7\n, [Subject]\n  CN=tiny\n\n[Issuer]\n  CN=tiny\n\n[Serial Number]\n  00F52B54E599C198CE\n\n[Not Before]\n  9/15/2021 10:02:43 PM\n\n[Not After]\n  9/15/2021 10:02:43 PM\n\n[Thumbprint]\n  CB437BDB1C0DC99C864EBEDB7E1B569C0A0C4CF4\n]]]></message>
        <stack-trace><![CDATA[   at System.Security.Cryptography.X509Certificates.Tests.CollectionTests.<AssertPkcs7PemExport>g__AssertPem|79_0(X509Certificate2Collection expected, ReadOnlySpan`1 pemActual) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CollectionTests.cs:line 1747
                     at System.Security.Cryptography.X509Certificates.Tests.CollectionTests.AssertPkcs7PemExport(X509Certificate2Collection collection) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CollectionTests.cs:line 1752
                     at System.Security.Cryptography.X509Certificates.Tests.CollectionTests.ExportCertificatePems_MultiCert() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CollectionTests.cs:line 1610
                     at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)]]></stack-trace>
    </failure>
</test>

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1393183&view=logs&j=b6e5d97f-d4f8-506b-9e47-e3e32385754a&t=94ce2805-f3e2-5f9a-1d92-0da5613f733d
Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-59762-merge-4055e4601c8e44dda7/System.Security.Cryptography.X509Certificates.Tests/1/console.99823d73.log?sv=2019-07-07&se=2021-10-19T18%3A38%3A09Z&sr=c&sp=rl&sig=LEPDic2W76gebkX3PdhWIyGcR4xrOqI8dDvJ26iXiN8%3D

cc/ @vcsjones @bartonjs

[ghost]

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq, @GrabYourPitchforks
See info in area-owners.md if you want to be subscribed.

Issue Details

---

<test name="System.Security.Cryptography.X509Certificates.Tests.CollectionTests.ExportCertificatePems_MultiCert" type="System.Security.Cryptography.X509Certificates.Tests.CollectionTests" method="ExportCertificatePems_MultiCert" time="0.0181399" result="Fail">
    <failure exception-type="Xunit.Sdk.EqualException">
        <message><![CDATA[Assert.Equal() Failure\nExpected: X509Certificate2[] [[Subject]\n  CN=tiny\n\n[Issuer]\n  CN=tiny\n\n[Serial Number]\n  00F52B54E599C198CE\n\n[Not Before]\n  9/15/2021 10:02:43 PM\n\n[Not After]\n  9/15/2021 10:02:43 PM\n\n[Thumbprint]\n  CB437BDB1C0DC99C864EBEDB7E1B569C0A0C4CF4\n, [Subject]\n  CN=tiny\n\n[Issuer]\n  CN=tiny\n\n[Serial Number]\n  008382F7436468D2F7\n\n[Not Before]\n  9/15/2021 10:05:10 PM\n\n[Not After]\n  9/15/2021 10:05:10 PM\n\n[Thumbprint]\n  0CC63492B8A2EE3B7D3085DF575A8B67B69261D7\n]\nActual:   X509Certificate2[] [[Subject]\n  CN=tiny\n\n[Issuer]\n  CN=tiny\n\n[Serial Number]\n  008382F7436468D2F7\n\n[Not Before]\n  9/15/2021 10:05:10 PM\n\n[Not After]\n  9/15/2021 10:05:10 PM\n\n[Thumbprint]\n  0CC63492B8A2EE3B7D3085DF575A8B67B69261D7\n, [Subject]\n  CN=tiny\n\n[Issuer]\n  CN=tiny\n\n[Serial Number]\n  00F52B54E599C198CE\n\n[Not Before]\n  9/15/2021 10:02:43 PM\n\n[Not After]\n  9/15/2021 10:02:43 PM\n\n[Thumbprint]\n  CB437BDB1C0DC99C864EBEDB7E1B569C0A0C4CF4\n]]]></message>
        <stack-trace><![CDATA[   at System.Security.Cryptography.X509Certificates.Tests.CollectionTests.<AssertPkcs7PemExport>g__AssertPem|79_0(X509Certificate2Collection expected, ReadOnlySpan`1 pemActual) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CollectionTests.cs:line 1747
                     at System.Security.Cryptography.X509Certificates.Tests.CollectionTests.AssertPkcs7PemExport(X509Certificate2Collection collection) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CollectionTests.cs:line 1752
                     at System.Security.Cryptography.X509Certificates.Tests.CollectionTests.ExportCertificatePems_MultiCert() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CollectionTests.cs:line 1610
                     at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)]]></stack-trace>
    </failure>
</test>

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1393183&view=logs&j=b6e5d97f-d4f8-506b-9e47-e3e32385754a&t=94ce2805-f3e2-5f9a-1d92-0da5613f733d
Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-59762-merge-4055e4601c8e44dda7/System.Security.Cryptography.X509Certificates.Tests/1/console.99823d73.log?sv=2019-07-07&se=2021-10-19T18%3A38%3A09Z&sr=c&sp=rl&sig=LEPDic2W76gebkX3PdhWIyGcR4xrOqI8dDvJ26iXiN8%3D

@vcsjones

Author:	mdh1418
Assignees:	-
Labels:	area-System.Security, untriaged
Milestone:	-

[vcsjones]

Hm, okay. Do Android tests not run as part of CI? Why was this green and got merged?

[mdh1418]

@vcsjones They are on CI, but they're non-blocking and show up in runtime-staging because of flakey reasons such as infrastructure failures, hitting a bad machine on the osx hosted pools, etc. As long as the build/lane completes its marked green on the github UI, but on close inspection by clicking the runtime-staging lane https://github.com/dotnet/runtime/pull/59674/checks?check_run_id=3740236485 the error is shown but again non-blocking.