ntlm: unsupported hash type md4

[nicolaibaralmueller]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

AWX can suddenly not connect to Windows hosts showing the error "ntlm: unsupported hash type md4".

Nothing has been changed on AWX.

AWX version

21.2.0

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

quay.io/ansible/awx-ee:latest

Operating system

Ubuntu 22.04 LTS

Web browser

No response

Steps to reproduce

Using VMware Inventory:

ansible_connection: winrm
ansible_port: 5986
ansible_winrm_transport: ntlm
ansible_winrm_server_cert_validation: ignore

Ansible Powershell WinRM script has been executed again. Also tested the credential login which also works.

Expected results

Should work as usual.

Actual results

<172.21.130.247> WINRM CONNECTION ERROR: unsupported hash type md4
Traceback (most recent call last):
  File "/usr/lib64/python3.9/hashlib.py", line 164, in __hash_new
    return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] unsupported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/ansible/plugins/connection/winrm.py", line 451, in _winrm_connect
    self.shell_id = protocol.open_shell(codepage=65001)  # UTF-8
  File "/usr/local/lib/python3.9/site-packages/winrm/protocol.py", line 166, in open_shell
    res = self.send_message(xmltodict.unparse(req))
  File "/usr/local/lib/python3.9/site-packages/winrm/protocol.py", line 243, in send_message
    resp = self.transport.send_message(message)
  F…
fatal: [WindowsHost]: UNREACHABLE! => {
    "changed": false,
    "msg": "ntlm: unsupported hash type md4",
    "unreachable": true
}

Additional information

No response

[shanemcd]

We recently merged this PR into awx-ee: ansible/awx-ee#160 Prior to this we were using the unmaintained quay.io/ansible/ansible-runner:latest base image.

Maybe try using quay.io/ansible/awx-ee:21.11.0 and see if that fixes it. If it's does, it means something has changed in either ansible-core or one of our dependencies.

[nitzmahone]

IIRC either the underlying OS libs or a newer Python stdlib removed the builtin md4 impl, which pywinrm's NTLM impl requires. @jborean93 fixed it in pypsrp (really the pyspnego subpackage) by rolling a private pure-Python md4 impl, but it's a little harder to transplant back to pywinrm since all that stuff is inline.

[nitzmahone]

OK, digging around some more the details are coming back.

pywinrm currently relies on requests-ntlm, which relies on Jordan's (now deprecated) pure-Python ntlm-auth (which relies on hashlib for md4- newer OpenSSL has dropped that since it's been deprecated forever). There's been an open PR on requests-ntlm for quite awhile to switch it over to Jordan's newer and actively-maintained pyspnego, but nobody's looked at it. I have maintainer privs on a couple of other requests projects, but not that one (and I haven't carefully reviewed that PR anyway).

Ideally, we'd just get that PR merged and all is well (well, downstream would also have to pick up the new deps, but I think we already have pyspnego anyway for pypsrp). Worst case, we could either vendor requests-ntlm + a patched ntlm-auth for md4 support into pywinrm, or do the same with the patched version of requests-ntlm that uses pyspnego.

[shrutebattlestargalactica]

same issue here with our instances

[nitzmahone]

So regardless of how we address this going forward for winrm @nicolaibaralmueller, since you're already using a recent awx-ee, the bits to use the newer psrp connection plugin should already be there. You should be able to just change ansible_connection: winrm to ansible_connection: psrp, rename ansible_winrm_server_cert_validation to ansible_psrp_cert_validation, and remove the port and transport connection vars altogether (since the psrp defaults will work).

for reference: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/psrp_connection.html#ansible-collections-ansible-builtin-psrp-connection

[lucapxl]

hopefully this can be helpful to somebody else: I had the exact same issue, but with @nitzmahone suggestion I was able to make my jobs work again.

my winrm configuration was:

[WindowsServers:vars]
ansible_connection=winrm
ansible_port=5985
ansible_winrm_message_encryption=always
ansible_winrm_scheme=http
ansible_winrm_server_cert_validation=ignore
ansible_winrm_transport=ntlm

my working psrp configuration is now:

[WindowsServers:vars]
ansible_connection=psrp
ansible_psrp_cert_validation=ignore
ansible_psrp_protocol=http

[nicolaibaralmueller]

Thanks for the help @nitzmahone @lucapxl @shanemcd.

Using 21.11.0 execution environment works with winrm and changing to psrp configuration works with latest AWX EE.

Problem solved!

[nitzmahone]

We're still talking through the best way to get things working again in pywinrm on newer OpenSSL, but I think we'll have a solution in place upstream for that soonish.

[nitzmahone]

OK, Jordan dusted off an open PR on requests-ntlm to convert it to use pyspnego internally, which should Just Fix the NTLM missing hash issues on newer OpenSSL. @shanemcd @AlanCoding if we pick up requests-ntlm>=1.2.0 in the EE defs, the pywinrm hash issues should be magically solved on all distributions.

[relrod]

if we pick up requests-ntlm>=1.2.0 in the EE defs, the pywinrm hash issues should be magically solved on all distributions.

I can confirm the latest EE has this.

bash-5.1$ pip list | grep ntlm
requests-ntlm                      1.2.0

Anything left to do here?

[silviolcf]

I`m facing the same error.
in Windows i can do with no problem a Request using Requests-ntlm2, but in a ubuntu server the same method just broken.

[mcinicola]

modify /etc/ssl/openssl.cnf

"# List of providers to load"
[provider_sect]
default = default_sect
legacy = legacy_sect <---- Add this

"# problems including inability to remotely access the system."
[default_sect]
activate = 1
[legacy_sect] <----add this
activate = 1 <---- add this