NTLM interop problems with MacOS

[davidsh]

.NET Core NTLM support is broken on MacOS. It has probably been broken for a while. But due to lack of test coverage we didn't notice. I did my testing on MacOS X 10.15.2 (Catalina).

The root cause is complex and due to possible interop issues with the NTLM support in the Heimdal library that ships with the Mac. Prior to MacOS 10.7, the Mac used MIT Kerberos open-source libraries. In 10.7, it switched to Heimdal open-source libraries. You can see the source code of the various open-source libraries used in the Mac here.

We have had some reports by customers of problems with Mac .NET Core clients using NTLM. At first, we thought that some NTLM support package was missing on the Mac. On Linux, we require the installation of the 'gss-ntlmssp' package which by default is not present on Linux distros nor on our .NET Core docker images. But the Mac already has NTLM support installed in the GSS-API mechanisms.

We have some simple NTLM tests in our repo:

runtime/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Authentication.cs

Lines 588 to 593 in 57df202

	[ConditionalTheory(nameof(IsNtlmInstalled))]
	[InlineData("NTLM")]
	[InlineData("Negotiate")]
	public async Task Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(string authScheme)
	{
	await LoopbackServerFactory.CreateClientAndServerAsync(

And these were running and passing on the Mac. However, the tests only validate that an initial NTLM base64 token is generated and sent to the server. Those tests don't do end-to-end validation of a username/password.

We have other tests for HttpClient that do end-to-end validation:

runtime/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Authentication.cs

Lines 568 to 570 in 57df202

	[ConditionalTheory(nameof(IsNtlmInstalled), nameof(IsWindowsServerAvailable))]
	[MemberData(nameof(ServerUsesWindowsAuthentication_MemberData))]
	public async Task Credentials_ServerUsesWindowsAuthentication_Success(string server)

But those are not turned on right now. They require environment variables to be enabled to point to a remote NTLM server. After turning on those tests on my local Mac, I discovered that NTLM is broken for Mac .NET Core.

More NTLM (and Kerberos) tests will be enabled for CI as part of follow-up PRs to #463

If I use Safari on the Mac, it works fine against the NTLM test server. But the Safari app on the Mac uses its own NTLM library and not the GSS-API Heimdal library that .NET Core uses.

Looking at several WireShark traces of working and non-working scenarios shows differences in various NTLM protocol fields. Although the Mac is sending NTLM packets to the server, the end result is that the server (Windows in this case) denies the authentication. We'll need to work with the Windows protocol team to help enable advanced diagnostics to see why the Windows server denied the authentication.

I will continue looking into the problem by trying out different NTLM servers to test against (such as Linux Apache) to get more information on working versus non-working scenarios. My current hypothesis is that there is a problem with Heimdal's NTLM v2 support. The NTLM protocol has many differences between NTLMv1 versus NTLMv2.

[scalablecory]

triage: please upvote this if you're seeing this issue.

[scholtz]

Hi @davidsh , I am trying to do windows authentication in linux according to this: https://docs.microsoft.com/cs-cz/aspnet/core/security/authentication/windowsauth?view=aspnetcore-3.1&tabs=visual-studio#kestrel

What I have observed, .net core 3.1 connects to System.Net.NTAuthentication which calls System.Net.Security.NegotiateStreamPal which calls unix Interop.NetSecurityNative.InitSecContext which invokes somehow the gss method in linux

without gss-ntlmssp library it throws error "An unsupported mechanism was requested." because there is no mechanism defined in /etc/gss/mech.d/ folder.

when i install gss-ntlmssp library, it throws error: "No credentials were supplied, or the credentials were unavailable or inaccessible"

i have tried to compile my own gss-ntlmssp, and found out that it does not even trigger any gss_ methods.
when i comment the line in mech.conf (gssntlmssp_v1 1.3.6.1.4.1.311.2.2.10 @libdir@/gssntlmssp/gssntlmssp.so) it shows unsupported mech, when i uncomment it show no credentials supplied without triggering the library.

i tested this on centos7 vm, ubuntu windows subsytem (18.04) and debian docker image (customized mcr.microsoft.com/dotnet/core/sdk:3.1-buster) - all behave the same

please see my full config in here: https://stackoverflow.com/questions/60296237/how-can-i-enable-debug-log-with-microsoft-aspnetcore-authentication-negotiate-pa

also note, that klist shows my spn, and i am able to connect to mssql using integrated security

what am i missing?? does this work for anybody?

[davidsh]

@scholtz The problem you are describing is not related to this particular issue. This issue is only about NTLM problems (client-side) on MacOS. So, it would be best for you to open a new issue to discuss your authentication issues on Linux.

That said, the problem you are describing is a server-side problem with asp.net core. Kestrel server does NOT support NTLM protocol. There is no way to tell it to send back 'Www-Authenticate: NTLM'. It only supports 'Www-Authenticate: Negotiate' protocol. 'Negotiate' protocol is usually Kerberos but can fall back to using NTLM packets (over SPNEGO protocol) in certain cases. NTLM fallback with Negotiate is something that isn't really supported either in Kestrel although I was able to make it work better in .NET 5. See: dotnet/corefx#42522

So, are you really trying to get NTLM working? You mention 'klist shows my spn and mssql works with integrated security'. So, that probably means you want to use Kerberos anyways. The 'gss-ntlmssp' component is not going to help you here since it is only for NTLM and not Kerberos.

You might want to open an issue in aspnetcore repo to get help on getting Kestrel working with Kerberos.

I also recommend you set the following environment variable on Linux to get GSS-API traces, i.e

export KRB5_TRACE=/dev/stdout

@Tratcher

[scholtz]

Hi @davidsh Great work with the dotnet/corefx#42522 .. Is there any chance how i can test it in my environment? How can I compile .net5 source code? Would it take too much effort for you to port it to .net core 3 :) ?

I believe I am missing some very simple configuration thing to make it work. Could you plase take a look at here, and suggest me how to proceed?

The kerberos kinit thing and mssql authentication does not necessary needs to be setup the same as negotiate auth in kestrel. I believe that Kestrel does the auth negotiate, which goes to gssapi, which goes to gssntlmssp lib and somehow it should authenticate to AD. The thing is that the first negotiate request goes well, and i get the response to the user, so it fails at the challange verification somehow because the winbind library does not verify the user token. So the real question is what do you have setup on your linux machine to make it work.

According to your advise i have started the issue in aspnetcore: dotnet/aspnetcore#19397 . I do not want to spam this thread, so if you will be so kind, please reply there. Thanks

[karelz]

@scholtz we do not backport fixes unless there is strong demand for them. You can try .NET 5 daily builds (or public previews coming soon) - see installers and binaries

[scholtz]

Hi @karelz , thanks for that.. I am trying to extend docker image mcr.microsoft.com/dotnet/core/sdk:3.1-buster with .net5, with no success yet.

I was able to install runtime using this:

RUN wget https://dotnetcli.blob.core.windows.net/dotnet/Runtime/master/dotnet-runtime-latest-x64.deb
RUN wget https://dotnetcli.blob.core.windows.net/dotnet/Runtime/master/dotnet-runtime-deps-latest-x64.deb
RUN wget https://dotnetcli.blob.core.windows.net/dotnet/Runtime/master/dotnet-hostfxr-latest-x64.deb
RUN wget https://dotnetcli.blob.core.windows.net/dotnet/Runtime/master/dotnet-host-latest-x64.deb

RUN dpkg -i dotnet-runtime-latest-x64.deb dotnet-runtime-deps-latest-x64.deb dotnet-hostfxr-latest-x64.deb dotnet-host-latest-x64.deb

but i am not able to find dependencies for the sdk:
for example it complains that runtime is in lower version than required runtime for sdk:

Step 39/42 : RUN wget https://dotnetcli.blob.core.windows.net/dotnet/Sdk/master/dotnet-sdk-latest-x64.deb
 ---> Using cache
 ---> 3921bf94717f
Step 40/42 : RUN dpkg -i  dotnet-sdk-latest-x64.deb
 ---> Running in 844e40507c26
Selecting previously unselected package dotnet-sdk-5.0.
(Reading database ... 29940 files and directories currently installed.)
Preparing to unpack dotnet-sdk-latest-x64.deb ...
Unpacking dotnet-sdk-5.0 (5.0.100-preview.2.20151.5-1) ...
dpkg: dependency problems prevent configuration of dotnet-sdk-5.0:
 dotnet-sdk-5.0 depends on dotnet-runtime-5.0 (>= 5.0.0~preview.2.20129.8); however:
  Version of dotnet-runtime-5.0 on system is 5.0.0~alpha.1.19564.1-1.
 dotnet-sdk-5.0 depends on netstandard-targeting-pack-2.1 (>= 2.1.0); however:
  Package netstandard-targeting-pack-2.1 is not installed.
 dotnet-sdk-5.0 depends on aspnetcore-runtime-5.0 (>= 5.0.0~preview.2.20128.11); however:
  Package aspnetcore-runtime-5.0 is not installed.
 dotnet-sdk-5.0 depends on dotnet-targeting-pack-5.0 (>= 5.0.0~preview.2.20129.8); however:
  Package dotnet-targeting-pack-5.0 is not installed.
 dotnet-sdk-5.0 depends on aspnetcore-targeting-pack-5.0 (>= 5.0.0~preview.2.20128.11); however:
  Package aspnetcore-targeting-pack-5.0 is not installed.
 dotnet-sdk-5.0 depends on dotnet-apphost-pack-5.0 (>= 5.0.0~preview.2.20129.8); however:
  Package dotnet-apphost-pack-5.0 is not installed.

dpkg: error processing package dotnet-sdk-5.0 (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 dotnet-sdk-5.0

Any ideas where i can find

• dotnet-runtime > 5.0.0~preview.2.20129.8 ,
• netstandard-targeting-pack-2.1 (>= 2.1.0);
• aspnetcore-runtime-5.0 (>= 5.0.0~preview.2.20128.11);
• dotnet-targeting-pack-5.0 (>= 5.0.0~preview.2.20129.8);
• aspnetcore-runtime-5.0 (>= 5.0.0~preview.2.20128.11);
• dotnet-targeting-pack-5.0 (>= 5.0.0~preview.2.20129.8);
• aspnetcore-targeting-pack-5.0 (>= 5.0.0~preview.2.20128.11);

? or is there any .net5 sdk docker image available?

in here https://hub.docker.com/_/microsoft-dotnet-core-sdk/ i dont see any..

thanks

[karelz]

@scholtz I would expect to see .NET 5 Docker image. At minimum with Preview builds. @richlander @MichaelSimons do you know more about it? (or someone who does)

[MichaelSimons]

dotnet/core Docker images will be published for all preview releases of .NET 5 (e.g. preview 1). There are no .NET 5 images in the dotnet/core repositories yet because preview 1 hasn't been released. Don't worry though we have you covered 😸, you can always get the last-known-good (LKG) builds for the next release of .NET Core by going to the dotnet/core-nightly repositories.

@scholtz, take a look at the mcr.microsoft.com/dotnet/core-nightly/sdk:5.0-buster image and see if it contains what you are looking for

[scholtz]

@MichaelSimons Yes, exactly. Thanks for that :) I was able to to create project from tamplate, added the negotiate library, build and run the project however at the end it did not solved my issue :(

[jstlns]

It would be nice if significant defects like this would be back ported to the LTS version of the SDK.

[dustinsoftware]

For others running into this issue, I ran into an identical problem during dotnet restore for both macOS and a Docker container build.

GSSAPI operation failed with error - An invalid name was supplied (Configuration file does not specify default realm).

apt-get update && apt-get install -y gss-ntlmssp resolved the issue within the Docker container prior to build, but I could not find an equivalent for macOS. I'm on Catalina 10.15.7, and the 15.0.100-rc.1.20452.101 sdk.

[InsanelyOne]

Is this ever going to get fixed? It's been busted for over a year!!!

[wfurt]

yes, it is broken for long time @InsanelyOne. But the core team is small and there are many issues to address.

I have plan for 6.0 and we can hopefully close this gap. It would be great if interested uses can participate in early verification and provide feedback.

[filipnavara]

@wfurt What's the current status of your attempts? No rush, just surveying what are our options.

I verified that we have problem connecting to HTTP endpoints of MS Exchange servers that have only NTLM and Negotiate authentication enabled (macOS 11.3 on the client).

The managed NTLM implementation does work against those servers and all the other ones we have tested so far.

One thing that I noticed was that with the managed implementation it was necessary to ensure that the HTTP content of the first negotiation response had to be read and discarded. If that was not done the authentication failed despite looking perfectly correct in Fiddler. It got me wondering whether the same thing could also be happening when using the native Karberos (albeit unlikely). Nevermind, there's code to drain the response.

We also have our own managed NTLM implementation that we use for non-HTTP protocols and we also tested it successfully against the problematic servers. It implements few more of the NTLM options not present in yours but probably nothing too important.

[wfurt]

The progress is not so good @filipnavara. I'm stuck on other priorities and there was no progress in last few month.
There are essentially three bigger tasks.

1. I want to add MIC to the implementation to improve security
2. figure out rc4 pal or use managed implementation
3. integrate with NTAuthHelper

If anybody can help with either one it would increase changes to get it to 6.

[filipnavara]

I am willing to invest some time into it. I have a local branch with MD4 implementation (backed by OpenSSL, CommonCrypto and Java API). RC4 is readily available on macOS through CommonCrypto and easy to add. On OpenSSL it's generally available but deprecated from version 3.0, not sure about Android. I can test that. We do have a managed implementation as part of our NTLM code which I can open source under MIT license. However, due to patent situation and long standing rule of not shipping crypto algorithm implementations in .NET I didn't consider that an option.

[wfurt]

Let's start with the macOS and get it functional. Using CommonCrypto or load from system ressl seems fine.
We will have chance to do improvements later but the feature freeze is rapidly approaching and it will be difficult to get it afterwards.

[filipnavara]

WIP branch with MD4 and RC4 implementation: https://github.com/filipnavara/runtime/tree/md4

[wfurt]

Thanks @filipnavara. The md4 is interesting, Im not convinced (yet?) we need rc4. Is there anything else I should look at besides the crypto support?

[filipnavara]

RC4 would be needed for session key encryption but it may not be necessary for minimal implementation. You were right that MD4 is missing on Android so that's something that would have to be solved eventually. Only the crypto support is in the branch linked above.

I decided to dig a bit deeper to figure out why the macOS native NTLM provider is actually failing. I've some test infrastructure running but I didn't find a way to get much logging information from Windows yet. Notably the values sent by macOS seem quite reasonable in the context of NTLMv2 with all the advanced protections (target information, channel binding, MIC). Neither your implementation, our internal implementation or the one in MailKit supports all these features but the MailKit one has at least few of the missing bits. One thing I could not figure out was why the macOS implementation sends 24 bytes of zeros in the LM challenge and the specification was not much of a help but curiously MailKit had an answer.

So far, the only bug in Apple implementation I found was fixed upstream (heimdal/heimdal@622c4de) but remains broken in Apple code. The NTLM version number was incorrectly encoded.

[JeroenBer]

@wfurt @filipnavara if it's of any use, I converted the C reference implementation of MD4 to C#. You can find it here and works on all .NET platforms: https://github.com/JeroenBer/Ntlm/blob/main/NtlmHttp/MD4.cs

[wfurt]

Yes I know @JeroenBer. However, pushing new crypto code to runtime is different level of scrutiny. It would be easier if we can avoid it and depend on the platform.

[JeroenBer]

@wfurt 👍👍

[filipnavara]

@wfurt Do you by any chance have an insight how to get more info about NTLM failures on the Windows server side? I do get an event in the Event Viewer that is quite short on the information on why it failed. Event tracing didn't seem to produce anything useful either. I probably don't have an access to checked versions of Windows builds but I'd be fine installing it if there was a chance of getting more information.

Otherwise I'll probably try to implement flag by flag what macOS is sending and try to see when it breaks. That may take a while though since it means implementing all the target information logic and MIC.

[wfurt]

I can ask around. Is there specific scenarios you are trying to solve?

[filipnavara]

I am just generally peeking to see why the macOS NTLM implementation (ie. heavily modified Heimdal code) doesn't work. So far the only bug in the implementation I found is the incorrect OS version encoding which is documented as debug-only and to be ignored by the implementations. The event on server lists the authentication info properly but fails with pretty generic error 0x80090308 (The token supplied to the function is invalid) and empty LmPackageName where I would expect NTLM v2 to be listed.

While I still think that it's worth pursuing the managed NTLM implementation for multiple reasons I also believe it would be useful to understand why the built-in implementation actually fails to authenticate.

I have Wireshark and Fiddler logs with the dissected authentication packets but there's nothing obviously wrong.

[filipnavara]

Progress report:

I found a couple of minor issues in https://github.com/wfurt/Ntlm:

1. NegotiateVersion flag has incorrect value
2. Time in NtChallengeResponse is incorrectly set to DateTime.Now.Ticks. Firstly, it's supposed to be UTC. Secondly, it should be converted using ToFileTimeUtc() and not as ticks.

I've implemented various different features:

• Parsing target information (and taking the time from it to use in NtChallengeResponse calculation).
• Updating target information with Target Name and Channel Bindings attributes
• Key exchange (RC4)
• Negotiating version
• Sending zeros in LM response when doing NTLMv2 with target information

Now the exchange looked quite similar to what macOS was sending. The last difference was the missing MIC and Flags attribute. Once I implemented the basic calculation for that it immediately started failing in the same way as the native macOS implementation. It's very likely I have bug somewhere in the key exchange or the MIC calculation itself but it will take me a while to compare it to some working implementation. Gonna take a break from it at least for the weekend to get a clear mind.

If it turned out to be only some incompatibility with the way MIC is calculated it would likely be possible to make a very targeted fix. It's possible to use the GSSAPI to generate all the messages. Then we can ask for the session key (GSS_NTLM_GET_SESSION_KEY_X) and rewrite the last message and its MIC to our liking. At this point this is purely a speculation though.

[filipnavara]

So, funny story, there's a one line fix:

--- a/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs
+++ b/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs
@@ -119,7 +119,7 @@ private static async Task<HttpResponseMessage> SendWithNtAuthAsync(HttpRequestMe
                         }
 
                         ChannelBinding? channelBinding = connection.TransportContext?.GetChannelBinding(ChannelBindingKind.Endpoint);
-                        NTAuthentication authContext = new NTAuthentication(isServer: false, challenge.SchemeName, challenge.Credential, spn, ContextFlagsPal.Connection, channelBinding);
+                        NTAuthentication authContext = new NTAuthentication(isServer: false, challenge.SchemeName, challenge.Credential, spn, ContextFlagsPal.Connection | ContextFlagsPal.InitIntegrity, channelBinding);
                         string? challengeData = challenge.ChallengeData;
                         try
                         {

This causes the GSS_C_INTEG_FLAG flag to be set on the GSSAPI context. In turn this causes setting the NegotiateAlwaysSign flag on the packets which is necessary to get MIC to work. The macOS implementation otherwise violates this condition from the NTLM specification:

A session key MUST always exist to generate the MIC (section 3.1.5.1.2) in the authenticate message. NTLMSSP_NEGOTIATE_ALWAYS_SIGN MUST be set in the NEGOTIATE_MESSAGE to the server and the CHALLENGE_MESSAGE to the client.

It generates the MIC and key exchange fields but not all the flags necessary for the key exchange.

[wfurt]

could you push the changes/fixes you have so far back to the managed implementation @filipnavara (in case we will use it later)

[filipnavara]

@wfurt I'll clean them up and submit a PR to your repository. It may just take me a few days to do it while I deal with some higher priority issues at work.

[Michriko]

I'm glad to see the problem could be fixed for .NET 6! Thanks a lot @filipnavara
Are there any plans to include the fix in .NET core 3.1?

[wfurt]

not at the moment @Michriko. It would need to meet servicing bar and get approval from servicing committee.

[karelz]

The bar for servicing would be:

1. Larger number of customers are significantly impacted without reasonable workaround.
2. Customer workloads impacting larger number of their customers are impacted without reasonable workaround.

I didn't see signs of either on this issue yet ...

[filipnavara]

I was seriously considering to submit a backport to Mono (which is stuck on a post-.NET Core 2.1 snapshot of CoreFX) because it's blocking us on Xamarin workloads (macOS at the moment, possibly iOS and Android depending on our release timing). JFYI as a data point. At the moment we have very crude fallback to MonoWebRequestHandler which has managed NTLM implementation that falls short on our security audits.

(We specifically don't need a backport to .NET Core 3.1 or .NET 5.)