GSS failures in System.Net.Http.Functional.Tests on Ubuntu 22.04

[omajid]

Description

Running runtime tests on Ubuntu 22.04 (which adds OpenSSL 3.0 resulting in a number of changes under the hood), leads to a bunch of tests failing: https://dev.azure.com/dnceng/public/_build/results?buildId=1690650&view=ms.vss-test-web.build-test-results-tab&runId=46193442&resultId=189361&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab

Some examples:

System.AggregateException : One or more errors occurred. (GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).) (Unexpected EOF trying to read request header)
---- System.ComponentModel.Win32Exception : GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).
---- System.IO.IOException : Unexpected EOF trying to read request header

    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(authScheme: "NTLM") [FAIL]
      System.AggregateException : One or more errors occurred. (GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).) (Unexpected EOF trying to read request header)
      ---- System.ComponentModel.Win32Exception : GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).
      ---- System.IO.IOException : Unexpected EOF trying to read request header
      Stack Trace:
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(88,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(101,0): at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(669,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(String authScheme)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace #1 (System.ComponentModel.Win32Exception) -----
        /_/src/libraries/Common/src/System/Net/Security/NegotiateStreamPal.Unix.cs(537,0): at System.Net.Security.NegotiateStreamPal.AcquireCredentialsHandle(String package, Boolean isServer, NetworkCredential credential)
        /_/src/libraries/Common/src/System/Net/NTAuthentication.Common.cs(128,0): at System.Net.NTAuthentication.Initialize(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding)
        /_/src/libraries/Common/src/System/Net/NTAuthentication.Common.cs(98,0): at System.Net.NTAuthentication..ctor(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs(169,0): at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(1033,0): at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.cs(240,0): at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(532,0): at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(676,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass34_0.<<Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)
        ----- Inner Stack Trace #2 (System.IO.IOException) -----
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(715,0): at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestHeaderBytesAsync()
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(777,0): at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestDataAsync(Boolean readBody)
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(1021,0): at System.Net.Test.Common.LoopbackServer.Connection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(1081,0): at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(686,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass34_0.<<Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader>b__1>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)

Reproduction Steps

From helix:

/root/helix/work/correlation/dotnet exec --runtimeconfig System.Net.Http.Functional.Tests.runtimeconfig.json --depsfile System.Net.Http.Functional.Tests.deps.json xunit.console.dll System.Net.Http.Functional.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 

Expected behavior

All tests pass

Actual behavior

Tests fail with GSS exceptions.

Regression?

Yes, the tests pass on older versions of Ubuntu currently running in CI

Known Workarounds

No response

Configuration

• dotnet/runtime main branch, commit 5a0564b01442f8ea9247e27c4fab85ee0d457265
• Ubuntu 22.04

Other information

No response

[msftbot]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Running runtime tests on Ubuntu 22.04 (which adds OpenSSL 3.0 resulting in a number of changes under the hood), leads to a bunch of tests failing: https://dev.azure.com/dnceng/public/_build/results?buildId=1690650&view=ms.vss-test-web.build-test-results-tab&runId=46193442&resultId=189361&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab

Some examples:

System.AggregateException : One or more errors occurred. (GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).) (Unexpected EOF trying to read request header)
---- System.ComponentModel.Win32Exception : GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).
---- System.IO.IOException : Unexpected EOF trying to read request header

    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(authScheme: "NTLM") [FAIL]
      System.AggregateException : One or more errors occurred. (GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).) (Unexpected EOF trying to read request header)
      ---- System.ComponentModel.Win32Exception : GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).
      ---- System.IO.IOException : Unexpected EOF trying to read request header
      Stack Trace:
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(88,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(101,0): at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(669,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(String authScheme)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace #1 (System.ComponentModel.Win32Exception) -----
        /_/src/libraries/Common/src/System/Net/Security/NegotiateStreamPal.Unix.cs(537,0): at System.Net.Security.NegotiateStreamPal.AcquireCredentialsHandle(String package, Boolean isServer, NetworkCredential credential)
        /_/src/libraries/Common/src/System/Net/NTAuthentication.Common.cs(128,0): at System.Net.NTAuthentication.Initialize(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding)
        /_/src/libraries/Common/src/System/Net/NTAuthentication.Common.cs(98,0): at System.Net.NTAuthentication..ctor(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs(169,0): at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(1033,0): at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.cs(240,0): at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(532,0): at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(676,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass34_0.<<Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)
        ----- Inner Stack Trace #2 (System.IO.IOException) -----
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(715,0): at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestHeaderBytesAsync()
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(777,0): at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestDataAsync(Boolean readBody)
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(1021,0): at System.Net.Test.Common.LoopbackServer.Connection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(1081,0): at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(686,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass34_0.<<Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader>b__1>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)

Reproduction Steps

From helix:

/root/helix/work/correlation/dotnet exec --runtimeconfig System.Net.Http.Functional.Tests.runtimeconfig.json --depsfile System.Net.Http.Functional.Tests.deps.json xunit.console.dll System.Net.Http.Functional.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 

Expected behavior

All tests pass

Actual behavior

Tests fail with GSS exceptions.

Regression?

Yes, the tests pass on older versions of Ubuntu currently running in CI

Known Workarounds

No response

Configuration

• dotnet/runtime main branch, commit 5a0564b01442f8ea9247e27c4fab85ee0d457265
• Ubuntu 22.04

Other information

No response

Author:	omajid
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[karelz]

Triage: Likely related to new image Ubuntu 22.04 (we do not have a queue yet). Seems to fail reliably. We should investigate.

[wfurt]

do you know @omajid if the image has the gss-ntlm package? Generally, I would think the OpenSSL is independent from Kerberos and GSS.

[omajid]

It does have gss: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/ubuntu/22.04/helix/amd64/Dockerfile

[wfurt]

Actually, you may be right about OpenSSL. It seems like md4 is no longer available from crypto

helixbot@9d96aeaca4ba:/$ openssl md4
Error setting digest
4037123E367F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:../crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (MD4 : 88), Properties ()
4037123E367F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:../crypto/evp/digest.c:237:

and

https://github.com/gssapi/gss-ntlmssp/blob/734e522c14a9821d7c03f2ce1691706d3d8131ad/src/crypto.c#L149-L153
While the gss-ntlmssp builds it is probably completely useless. (something to report ditto vendors..?)

For now, I think we detect presence of the page and we would skip tests as needed. Short term fix may be removing the ntlm package from docker image. That of course leaves NTLM auth broken.

The options would be to report/fix the package so it works with OpenSSL 3.x (e.g. add private fall-back implementation of md4) or switch to managed implementation #66879

cc: @filipnavara @bartonjs

[filipnavara]

Let's report upstream and see. Long term I am keen on making the Managed NTLM an option either through an app context switch, or as a fallback if gss-ntlmssp is not installed.

[omajid]

Should we flag this as part of Ubuntu 22.04 support? Looking at dotnet/core#7038 it seems like everything is 100% functional?

[wfurt]

I'm not sure. This looks like distribution bug to me @omajid as the package they provide does not work.

[vcsjones]

It seems like md4 is no longer available from crypto

You need to load the legacy provider for that to work in OpenSSL 3. You can either do that in openssl.cnf, or from the command line, this should work:

echo hi | openssl md4 -provider legacy

However for the runtime, we explicitly load the "legacy" provider, so MD4 should be available.

runtime/src/native/libs/System.Security.Cryptography.Native/openssl.c

Lines 1175 to 1180 in fe0f600

	void CryptoNative_RegisterLegacyAlgorithms()
	{
	#ifdef NEED_OPENSSL_3_0
	if (API_EXISTS(OSSL_PROVIDER_try_load))
	{
	OSSL_PROVIDER_try_load(NULL, "legacy", 1);

[vcsjones]

One way to tell is to change openssl.cnf to have the following provider_sect:

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

[filipnavara]

However for the runtime, we explicitly load the "legacy" provider, so MD4 should be available.

...but only after you use some crypto that initializes the OpenSSL native shim, right?

[vcsjones]

Ah, you're right. I thought we always loaded the legacy provider, but we only do it when you use an algorithm that is in the legacy provider:

runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RC2Implementation.OpenSsl.cs

Line 23 in 899bf97

Interop.Crypto.EnsureLegacyAlgorithmsRegistered();

Changing the openssl.cnf to load the legacy provider however would work, assuming the problem is the lack of the legacy provider's availability.

[wfurt]

This was fixed in gss-ntlm package. It is up to Ubuntu to pick up the fix. Big thanks to @simo5 who did the fix.

[karelz]

@wfurt why did you send it back to triage?
If it is external (and addressed there already), I would recommend to close it with details which version of library is needed to make it work.

[wfurt]

We can (should?) improve platform detection and skip the tests as needed instead of failing. This will bite us once #67345 is merged.
We may also choose to solve it via manage NTLM.

[karelz]

Triage: Platform detection needs to be improved to handle the case as well.

Note: This will be addressed once we have managed NTLM implementation - but there is no guarantee when it will happen.

[yaakov-h]

This isn't just a test-related issue. I just upgraded a system from Ubuntu 20.04 to Ubuntu 22.04 and a .NET 6 application could no longer use NTLM auth until I applied the workaround mentioned in #67353 (comment).

Should this be documented somewhere as a compatibility issue for developers / end-users?

[wfurt]

We could but it is really difficult to trace and keep in sync all Linux distributions and versions. We could add note that the functionality depends on underlying OS capabilities.
But as I mentioned above it is really up to Ubuntu to pick up the package fixes. Perhaps you can open issue with them.

[wooncherk]

One way to tell is to change openssl.cnf to have the following provider_sect:

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

This helped. Thanks! :)