7.7+ issue with ssl-cert returning certificate when http-title returns "401 Authorization Required"

[jr00000]

On version 6.4 ssl-cert (nmap -sV -sC) will return a certificate when the http-title returns "401 Authorization Required" but on verison 7.7 or 7.8 it stops after the title and does not return a certificate. This occurs on every device that has a pop-up sign in window before you can access it's HTTP page. I have to continue running version 6.4 until this is corrected because this severely limits Nmaps abilities.

Example of 'nmap -sV -sC'
Version 6.4:

PORT STATE SERVICE VERSION
21/tcp open ftp oftpd
| ftp-anon: Anonymous FTP login allowed (FTP code 202)
|Can't get directory listing: Can't parse PASV response: "Command not implemented."
23/tcp open telnet?
80/tcp open http?
| http-auth:
| HTTP/1.0 401 Unauthorized
| Digest nonce=5ff518ef19b57b3722f6253570cf155a stale=FALSE algorithm=MD5 opaque= realm=Use 'live' as User Name in order to log in to the respective level qop=auth
|_http-methods: No Allow or Public header in OPTIONS response (status code 400)
|_http-title: 401 Authorization Required
443/tcp open ssl/openvas OpenVAS server
|_http-methods: No Allow or Public header in OPTIONS response (status code 400)
|http-title: 401 Authorization Required
| ssl-cert: Subject: commonName=local.myboschcam.net/organizationName=Bosch Sicherheitssysteme GmbH/stateOrProvinceName=Bayern/countryName=DE/localityName=Grasbrunn
| Issuer: commonName=local.myboschcam.net/organizationName=Bosch Sicherheitssysteme GmbH/stateOrProvinceName=Bayern/countryName=DE/localityName=Grasbrunn
| Public Key type: rsa
| Public Key bits: 1024
| Not valid before: 2012-11-05T13:05:29+00:00
| Not valid after: 2015-11-05T13:05:29+00:00
| MD5: a013 caa0 6b80 0788 c507 93e3 29fd 04b2
| SHA-1: 7c82 82ae 3d92 e609 75da 3e46 9d0f f957 896c 3512
| -----BEGIN CERTIFICATE-----
| MIICwDCCAimgAwIBAgIJAIl4oj8F35HTMA0GCSqGSIb3DQEBBQUAMHkxCzAJBgNV
| BAYTAkRFMQ8wDQYDVQQIDAZCYXllcm4xEjAQBgNVBAcMCUdyYXNicnVubjEmMCQG
| A1UECgwdQm9zY2ggU2ljaGVyaGVpdHNzeXN0ZW1lIEdtYkgxHTAbBgNVBAMMFGxv
| Y2FsLm15Ym9zY2hjYW0ubmV0MB4XDTEyMTEwNTEzMDUyOVoXDTE1MTEwNTEzMDUy
| OVoweTELMAkGA1UEBhMCREUxDzANBgNVBAgMBkJheWVybjESMBAGA1UEBwwJR3Jh
| c2JydW5uMSYwJAYDVQQKDB1Cb3NjaCBTaWNoZXJoZWl0c3N5c3RlbWUgR21iSDEd
| MBsGA1UEAwwUbG9jYWwubXlib3NjaGNhbS5uZXQwgZ8wDQYJKoZIhvcNAQEBBQAD
| gY0AMIGJAoGBALNVdJWLOjs6zTjYkB8zNdWppgdBzIwTYtLGSmEqe2l0i5asbHIM
| DSZk+9OcuVh0kasup4AO6w7Q9hG3E2gX2mnNWIhOXnIJ9No3EsseWgLYN0R6Vda1
| srjsjAFHcfsZoFibmybsOtOPR61uPWzlikJ1HSXoLXY8MSq/vHWhBnszAgMBAAGj
| UDBOMB0GA1UdDgQWBBTK669jgSwEYeAjMR1YpmkR6v9y7zAfBgNVHSMEGDAWgBTK
| 669jgSwEYeAjMR1YpmkR6v9y7zAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUA
| A4GBACuMVPYOlnXaWJWxRD87WoFHBrbfLMocgTAoahbpH0kzL9MCDbWSCx3cS5LT
| lu6ty0Vuh2ih1pauTWfzdKJnYt0znVxuHNs4vzgOqv8ntjzVkJT5105pdF8zQOQ2
| F+0TAeLeIUKBXwzhwVnPvFzPg2ZAVnBlw3pKj3PgRSEQAdFI
|-----END CERTIFICATE-----
554/tcp open rtsp Apple AirTunes rtspd
|rtsp-methods: ERROR: Script execution failed (use -d to debug)
3260/tcp open iscsi?
| iscsi-info:
| Target: iqn.2005-12.com.bosch:unit00075f7f9419.trg-000-sdcard
| Address: xxxxx:3260,1
| Authentication: Authentication failure
2 services unrecognized despite returning data. If you know the service/version, please submit the following fingerprints at http://www.insecure.org/cgi-bin/servicefp-submit.cgi :

Version 7.8:

PORT STATE SERVICE REASON VERSION
21/tcp open ftp syn-ack ttl 61 oftpd
|ftp-anon: ERROR: Script execution failed (use -d to debug)
|ftp-bounce: ERROR: Script execution failed (use -d to debug)
|ftp-syst: ERROR: Script execution failed (use -d to debug)
23/tcp open telnet syn-ack ttl 61 HP LaserJet telnetd
| fingerprint-strings:
| GenericLines:
| Welcome to NDC-265-P xx.xx.xx.xx from xx.xx.xx.xx
| telnet last socket error 0x00000133
| telnet -1 !!!
| invalid username
| enter username ->
| GetRequest:
| Welcome to NDC-265-P xx.xx.xx.xx from xx.xx.xx.xx
| second telnet session from xx.xx.xx.xx discarded
| HTTP
| input too long
| again:
| Help:
| Welcome to NDC-265-P xx.xx.xx.xx from xx.xx.xx.xx
| second telnet session from xx.xx.xx.xx discarded
| HELP
| invalid username
| enter username ->
| NCP:
| Welcome to NDC-265-P xx.xx.xx.xx from xx.xx.xx.xx
| second telnet session from xx.xx.xx.xx discarded
| DmdT
| NULL:
| Welcome to NDC-265-P xx.xx.xx.xx from xx.xx.xx.xx
| telnet last socket error 0x00000133
| telnet -1 !!!
80/tcp open http syn-ack ttl 61 Bosch DINION IP Bullet 5000 webcam http admin
| http-auth:
| HTTP/1.0 401 Unauthorized\x0D
| Digest stale=FALSE algorithm=MD5 realm=Use 'live' as User Name in order to log in to the respective level nonce=347e12f33487322b7f6522720a2d5433 opaque= qop=auth
| http-methods:
| Supported Methods: GET POST
|_http-title: 401 Authorization Required
443/tcp open ssl/openvas syn-ack ttl 61 OpenVAS server
|_ssl-date: 2019-08-11T09:00:08+00:00; -5h07m37s from scanner time.
554/tcp open rtsp syn-ack ttl 61 Apple AirTunes rtspd
|rtsp-methods: ERROR: Script execution failed (use -d to debug)
3260/tcp open iscsi? syn-ack ttl 61
| iscsi-info:
| iqn.2005-12.com.bosch:unit00075f7f9419.trg-000-sdcard:
| Address: xx.xx.xx.xx:3260,1
| Authentication: required
| Auth reason: Authentication failure
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service :

[jr00000]

Here are the scans with more detailed debug parameters:
nmap -n -T3 -d9 -Pn -sS -p 443 --script ssl-cert ip address

Attached as txt files:
Version 6.4: Scan6dot4Good.txt
Scan6dot4Good.txt

Version 7.8: Scan7dot8Error.txt
Scan7dot8Error.txt

[jr00000]

I installed every version of Nmap and determined 6.47 is the first version the issue begins to occur.

[nnposter]

I have sampled a handful of sites with the following scan:

nmap -n -Pn -sT -T2 -p443 --script ssl-cert -d <target>

All of the sites return HTTP status 401 while script ssl-cert worked as expected, returning the certificate.

The test environments were:

• nmap 7.70 and 7.80 on Ubuntu 18.04 x64 (Azure)
• nmap 7.70SVN on Ubuntu 18.04 x64 (VMware)
• nmap 7.70SVN on Win7 x64 (physical)

These results suggest that the issue is not universal but rather related to your scanning environment or specific targets.

The following part of your output is the key:

NSOCK DEBUG FULL [30.2900s] epoll_loop(): wait for events
NSOCK DEBUG FULL [30.2910s] epoll_loop(): wait for events
NSOCK DEBUG FULL [30.2910s] epoll_loop(): wait for events
NSOCK DEBUG FULL [30.2910s] epoll_loop(): wait for events
NSOCK DEBUG FULL [30.2910s] process_event(): Processing event 9 (timeout in 0ms, done=0)
NSOCK DEBUG FULL [30.2910s] process_event(): NSE #9: Sending event
NSOCK INFO [30.2910s] nsock_trace_handler_callback(): Callback: SSL-CONNECT TIMEOUT for EID 9 [xx.xx.xx.xx:443]

This shows that the HTTP status plays no role here. The TLS connection never completed the initial handshake.

[jr00000]

What is odd is that version 6.46 works but 6.47+ does not.

curl -vvIk https://_ipaddress_
and
openssl s_client -connect _ipaddress_:443
both work as expected all the time.

I am running CentOS 7.6.1810 on VMware. I have also tried it on Windows 10 x64 physical.

I can provide -d 9 if that would help determine the cause.

[nnposter]

The detailed log will probably not help. I have casually reviewed the SVN commits between 6.46 and 6.47, noting these culprit candidates:

• The OpenSSL library got upgraded
• Cipher suites offered during the TLS handshake have changed

I would suggest the following:

1. Run nmap 7.80 with -sT, instead of -sS, just to make sure that basic TCP connectivity works.
2. Assuming it does, run nmap script ssl-enum-ciphers, determining which cipher suites and TLS/SSL versions are supported by the target.
3. If your target is publicly facing you might also use third-party services, such as SSL Labs, for the same.
4. Compare these results with what nmap uses when making TLS connections.

[jr00000]

Below is the result of the ssl-enum-ciphers. Is there a list of what cipher suites are offered?

PORT     STATE SERVICE REASON
21/tcp   open  ftp     syn-ack ttl 61
23/tcp   open  telnet  syn-ack ttl 61
80/tcp   open  http    syn-ack ttl 61
443/tcp  open  https   syn-ack ttl 61
| ssl-enum-ciphers: 
|   TLSv1.0: 
|     ciphers: 
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 1024) - A
|       TLS_RSA_WITH_DES_CBC_SHA (rsa 1024) - D
|     compressors: 
|       NULL
|     cipher preference: client
|     warnings: 
|       64-bit block cipher DES vulnerable to SWEET32 attack
|       Weak certificate signature: SHA1
|_  least strength: D
554/tcp  open  rtsp    syn-ack ttl 61
3260/tcp open  iscsi   syn-ack ttl 61

[nnposter]

At first glance the cipher suites do not appear to be a problem. The first one, TLS_RSA_WITH_AES_128_CBC_SHA, is normally offered by Nmap.

There are two known issues related to connecting to some older TLS implementations:

• Some servers do not work properly when a client proposes TLS 1.2 but the server only supports 1.0.
• Some servers choke up when a client offers "too many" cipher suites.

The second case you can generally test for by trying to connect with OpenSSL. This typically works:

openssl s_client -connect <target>

However, the following does not:

openssl s_client -connect <target> -cipher ALL

For the first case you need to inspect the Nmap TLS handshake in Wireshark, looking for symptoms described in issue #511.

[jr00000]

Ok I tested those two issues. The second case had 0 errors with both openssl connections. The first case I am seeing Client Hello with no Server Hello. I attached a screenshot below. Does this mean that I will be unable to use Nmap 6.47+ if I would like to retrieve certificates from hosts that only support TLS 1.0?

[nnposter]

No, it does not. If a client proposes TLS 1.2 then the server is free to reply with any version up to 1.2, such as 1.0. It is then up to the client to decide if it is willing to continue with lower version. These days Nmap typically accept TLS 1.0, 1.1, and 1.2. Support for SSLv3 is spotty, depending on the OpenSSL library used.

What it does mean is that some TLS implementations are not fully compliant with the TLS standard and Nmap is not compensating for all the quirks that are out there.

[jr00000]

That doesn't make sense though. Nmap is using OpenSSL, which has no issues getting the certificate from this host. I have captured Wireshark while running OpenSSL and Nmap on the same host. OpenSSL which Client Hello and Server Hello while Nmap has issues. This was on Windows 10 x64.

This is OpenSSL 1.1.1c running openssl s_client -connect <target> -cipher ALL :

This is Nmap 7.70 running nmap -sV -p 443 --script ssl-cert ipaddress :

If Nmap is using OpenSSL then why does it have an issue with this? Beyond that, Nmap is a tool used for finding vulnerabilities so wouldn't it make sense to at least report or exploit that TLS is out of date?

[jr00000]

I can actually see the certificate inside of Wireshark when I run nmap -sV -p 443 --script ssl-cert ipaddress.

[nnposter]

I do not believe I can help with throubleshooting the issue any further without having access to the target. I have validated that the original claim of "This (issue) occurs on every device" is not true.

[jr00000]

@nnposter I would be willing to provide you with network access to the host, or mail you the hardware as it's very small if you would like to help further.

[nnposter]

I cannot make promises that I will be able to help but feel free to send me a pertinent IP address and port.

[nnposter]

Please let me know if you have further questions.

[jr00000]

@nnposter Can you provide an email address I can contact you at? I would still like to continue working on this issue.

[nnposter]

Just scroll down to the bottom of this README

[nnposter]

I have tested nmap against Bosch NDC-265-P, which is the target reported by the OP.

One clear issue is that Debian and a few derivatives, including Kali, are shipping with pretty unforgiving OpenSSL configuration. The connectivity with with this target fails because the minimum acceptable TLS version is 1.2, while the target supports only 1.0. Once this restriction is removed then nmap is performing as expected. To confirm that your environment is suffering from the same issue you can check with Wireshark if nmap responds to the Server Hello message with a "Protocol Version" fatal alert. You can then adjust the OpenSSL configuration as needed.

Another observed issue is that the target is taking 7 to 8 seconds to complete the TLS handshake, which is something that you need to be mindful of when tuning your scans.

[jr00000]

I don't understand why that would be a clear issue if installing nmap version 6.46 work perfectly fine but 6.47 and up do not. Doesn't matter what OS either, Windows 10, Debian, or Red Hat. I have not had any issues with response timeout while using version 6.46.

This obviously isn't getting fixed anytime soon so I am going to continue using version 6.46.

[nnposter]

I am not saying that this OpenSSL minimum TLS protocol issue is your issue. It is just one clear issue you might be running into. I have also offered a second possible explanation.

Your initial claim (as stated in the title of this issue) has turned out to be false. I have then offered to troubleshoot the issue further if I get access to one of the problematic targets. You have declined and instead disclosed which target hardware is not working for you. I have tested this hardware, identifying two potential scan dependencies listed above, and confirming that nmap is working correctly even with this specific hardware.

At this moment I do not believe that I can assist you any further.

[dmiller-nmap]

I did make a slightly related change today that may help in situations where the linked OpenSSL routines are being too strict to complete the handshake. In those situations, ssl-cert will fall back to a raw TCP connection and attempt to use the handshake routines from tls.lua to extract a certificate. See 81ceee4