In non-sspi builds, NTLM cannot identify against saslauthd (cyrus-sasl)

[monnerat]

Current curl NTLM implementation is based on https://curl.se/rfc/ntlm.html: this document seems to be an outstanding work of reverse engineering and/or information compilation done at a time when NTLM was a proprietary protocol and no specification were published.

Since then, M$ released the specification: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-nlmp/

Comparing both documents, there seems to be some misunderstanding in the first one, one of them explaining why curl is not able to authenticate against saslauthd using NTLM:

Trying to be short:

• there are 3 ways to provide authentication information, NTLMv1, NTLMv1 with extended security and NTLMv2.
• curl understand negotiated flags NTLMFLAG_NEGOTIATE_NTLM2_KEY as a request to use NTLMv2 and NTLMFLAG_NEGOTIATE_NTLM_KEY to use NTLMv1 with extended security.
• according to the MS-NLMP document (that names these flags NTLMSSP_NEGOTIATE_EXTENDED_SESSIONSECURITY and NTLMSSP_NEGOTIATE_NTLM respectively), this is a misuse: there is no possible negotiation of NTLMv2 in the protocol (client should have an external way of selecting it or not). If NTLMv2 is not used, NTLMFLAG_NEGOTIATE_NTLM2_KEY tells to use NTLMv1 extended security, while the meaning of NTLMFLAG_NEGOTIATE_NTLM_KEY is not clear (probably can be cleared to downgrade the authentication to LM-only).
• curl negotiates the full capabilities (according to its protocol understanding and conditional compilation statements), that is: NTLMFLAG_NEGOTIATE_NTLM_KEY and NTLMFLAG_NEGOTIATE_NTLM2_KEY. But saslauthd does not support extended security and thus properly clears flag NTLMFLAG_NEGOTIATE_NTLM2_KEY in challenge message (type2). As NTLMFLAG_NEGOTIATE_NTLM_KEY is still set, curl uses NTLMv1 with extended security for authentication, which is wrong according to MS-NLMP.

Note that, although NTLMv2 is not negotiated, a server receiving an authentication (type3) message can determine it from the NTLM authentication data length. A client however, has no way to know if a server supports NTLMv2 or not.

Although MS-NLMP does not specifiy any meaning of NTLMFLAG_NEGOTIATE_NTLM2_KEY when using NTLMv2, it sets it in negotiation of all NTLMv2 examples.

This does not affect builds with ntlm-sspi.

[monnerat]

Some other client libraries behavior:

• cyrus-sasl checks a configuration parameter to use NTLMv2 or NTLMv1 without extended security.
• libgsasl uses libntlm that always uses NTLMv1 without extended security.

Possible fixes in curl
There is a tradeoff in choice: one end is the maximum compatibility with old servers at the price of a possible rejection by a server requesting a more secure level of authentication, the other is the maximum security level.

1. Maximum security level: Always use NTLMv2 if compiled in.
2. Securely mitigated: if NTLMFLAG_NEGOTIATE_NTLM2_KEY is set and NTLMv2 is compiled in, use NTLMv2: the idea here is to use the maximum security level as a server that accepts extended security for NTLMv1 is likely to understand NTLMv2 too. If NTLMv2 is not compiled in, but extended security is, use NTLMv1 + extended security.
3. Less securely mitigated: if NTLMFLAG_NEGOTIATE_NTLM2_KEY is set and extended security is compiled in, use NTLMV1 with extended security. If not compiled in, use NTLMv2 if available.
4. Never use NTLMv2: if NTLMFLAG_NEGOTIATE_NTLM2_KEY is set and extended security is compiled in, use NTLMV1 with extended security.
5. Never use extended security: Always use NTLMv1 without extended security.
6. Something else, as a new CURLOPT option for example (needs to set the default anyway).

In all cases, if NTLMFLAG_NEGOTIATE_NTLM2_KEY is clear or neither NTLMv2 nor extended security are available, use NTLMv1 without extended security.
NTLMFLAG_NEGOTIATE_NTLM_KEY is not tested anymore.

A local build with fix 2 applied has been tested against saslauthd successfully.

Whatever fix is adopted, the test suite would have to be adapted accordingly. Real tests against a M$ server are also mandatory.

Questions:

• Which is the preferred fix ?
• As I don't have any M$ software, how can I test in real conditions ?

[dfandrich]

Aren't there NTLM libraries out there that do this correctly that curl can leverage? It's always seemed suspect to me that we roll our own implementation of this.

[monnerat]

Yes, there are. But I don't know if they were available at the time NTLM was introduced in curl.
As cited in #6813 (comment), I checked cyrus-sasl and libgsasl/libntlm: both uses NTLMv1 without extended security (the lowest security level). Is this what we want ?
NTLMv1 with extended security as well as NTLMv2 feature a client challenge, while NTLMv1 does not.
In all cases, the biggest work will be about testing.

[MarcelRaad]

I like option 2. It's not only more secure, but also the smaller change in behavior.