OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: dh key too small

[wchen-r7]

I run into the following error while attempting to run glassfish_login or glassfish_deployer against a Glassfish 3.1.2.2 (Win Server 2003) test box:

[04/15/2016 03:31:57] [i(2)] core: Reloading auxiliary module scanner/http/glassfish_login. Ambiguous module warnings are safe to ignore
[04/15/2016 03:32:03] [e(0)] core: Error running against host 192.168.1.109: SSL_connect returned=1 errno=0 state=error: dh key too small
/Users/wchen/rapid7/msf/lib/rex/socket/ssl_tcp.rb:136:in `connect'
/Users/wchen/rapid7/msf/lib/rex/socket/ssl_tcp.rb:136:in `block in initsock_with_ssl_version'
/Users/wchen/.rvm/rubies/ruby-2.1.9/lib/ruby/2.1.0/timeout.rb:75:in `timeout'
/Users/wchen/rapid7/msf/lib/rex/socket/ssl_tcp.rb:134:in `initsock_with_ssl_version'
/Users/wchen/rapid7/msf/lib/rex/socket/ssl_tcp.rb:88:in `initsock'
/Users/wchen/rapid7/msf/lib/rex/socket/comm/local.rb:339:in `create_by_type'
/Users/wchen/rapid7/msf/lib/rex/socket/comm/local.rb:33:in `create'
/Users/wchen/rapid7/msf/lib/rex/socket.rb:47:in `create_param'
/Users/wchen/rapid7/msf/lib/rex/socket/tcp.rb:37:in `create_param'
/Users/wchen/rapid7/msf/lib/rex/socket/tcp.rb:28:in `create'
/Users/wchen/rapid7/msf/lib/rex/proto/http/client.rb:181:in `connect'
/Users/wchen/rapid7/msf/lib/metasploit/framework/login_scanner/glassfish.rb:66:in `send_request'
/Users/wchen/rapid7/msf/lib/metasploit/framework/login_scanner/glassfish.rb:36:in `check_setup'
/Users/wchen/rapid7/msf/modules/auxiliary/scanner/http/glassfish_login.rb:176:in `run_host'
/Users/wchen/rapid7/msf/lib/msf/core/auxiliary/scanner.rb:121:in `block (2 levels) in run'
/Users/wchen/rapid7/msf/lib/msf/core/thread_manager.rb:100:in `call'
/Users/wchen/rapid7/msf/lib/msf/core/thread_manager.rb:100:in `block in spawn'

Note that this used to work (#3716), so the bug came in after that.

[wchen-r7]

It looks like it's not specific to GlassFish.

Here's me testing https://metasploit.com w/ curl:

$ curl -v https://metasploit.com
* Rebuilt URL to: https://metasploit.com/
*   Trying 54.192.137.71...
* Connected to metasploit.com (54.192.137.71) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: www.metasploit.com
* Server certificate: Go Daddy Secure Certificate Authority - G2
* Server certificate: Go Daddy Root Certificate Authority - G2
> GET / HTTP/1.1
> Host: metasploit.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK

And then the same using auxiliary/scanner/http/ssl:

msf auxiliary(ssl) > set SSLVersion TLS1.2
SSLVersion => TLS1.2
msf auxiliary(ssl) > set SSL true
SSL => true
msf auxiliary(ssl) > run

[*] Error: 54.192.137.71: OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: sslv3 alert handshake failure
[*] Scanned 1 of 1 hosts (100% complete)
[*] Auxiliary module execution completed
msf auxiliary(ssl) > 

[firefart]

@wchen-r7 https://bugzilla.redhat.com/show_bug.cgi?id=1232110 and https://bugzilla.redhat.com/show_bug.cgi?id=1255417

looks like your openssl version is too new :) It looks like they dropped support for small DH keys which does not really help on pentesting machines :(

[wchen-r7]

ahhhh, so annoying.

Thanks @firefart

[busterb]

Sort of feels like we need to create an alternate ruby SSL wrapper to deal with old stuff. We can't really rely on the system ruby for any particular feature, especially old and insecure ones.

[firefart]

@busterb right but reimplementing the SSL stack is kinda over the top :) I think the small DH keys can only be enabled on openssl compile time. The same problem exists with browsers refusing connections to old, insecure SSL protocols.

Is it possible to ship a custom compiled openssl version with metasploit? where all the old stuff like SSLv2, export ciphers and small diffie helman keys are enabled?

[busterb]

That's more like what I meant; an alternate linked implementation. We do ship a local version with the Omnibus installers, but Kali currently ships whatever happens to come with Debian upstream. We need a solution more like sslscan, which statically links a 'kitchen sink' openssl build: https://github.com/rbsec/sslscan#openssl-issues

[firefart]

@busterb so would it be possible to ship openssl as a rubygem / precompiled in a special folder, manipulate the PATH variable on start of msf* to include the path in the first position and trick ruby into using our own openssl version?

[wchen-r7]

Do you guys want me to reopen this ticket? It feels many people can run into this problem.

[busterb]

Yeah, we should address it somehow, since only more people will begin seeing it.

[sekritskwurl]

Bump.
I ran into this issue with tomcat_mgr_deploy. I'm going to give recompiling Openssl to support weaker DH groups a shot so I can get the exploit to work, but it is far from the ideal solution.

[bcook-r7]

perhaps this broken-out version of the ruby openssl extension could be repurposed for use internally by metasploit: https://github.com/rubysl/rubysl-openssl

[naterobbified]

Anyone find a way to resolve this error? Got the same error while attempting a different metasploit exploit recently.

[karllll]

One possible bypass is to set the Net::HTTP ciphers option, excluding Diffie Hellman ciphers as described here ( https://zackhobson.com/2014/02/10/ssl_and_ruby_part_2/ ). I used http.ciphers = ['AES128-SHA'].

[twinvega]

Not a fix, but you could proxy your traffic through something like Burp which wouldn't have the issue.

[jepunband]

hi did anyone solve this please??

[-] Auxiliary failed: OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed
[-] Call stack:
[-] /opt/metasploit-framework/embedded/lib/ruby/2.4.0/net/protocol.rb:44:in connect_nonblock' [-] /opt/metasploit-framework/embedded/lib/ruby/2.4.0/net/protocol.rb:44:in ssl_socket_connect'
[-] /opt/metasploit-framework/embedded/lib/ruby/2.4.0/net/http.rb:948:in connect' [-] /opt/metasploit-framework/embedded/lib/ruby/2.4.0/net/http.rb:887:in do_start'
[-] /opt/metasploit-framework/embedded/lib/ruby/2.4.0/net/http.rb:876:in start' [-] /opt/metasploit-framework/embedded/lib/ruby/2.4.0/net/http.rb:1407:in request'
[-] /opt/metasploit-framework/embedded/framework/modules/auxiliary/gather/shodan_search.rb:59:in shodan_query' [-] /opt/metasploit-framework/embedded/framework/modules/auxiliary/gather/shodan_search.rb:109:in run'
[*] Auxiliary module execution completed

[egypt]

Issue still exists on recent Kali. A coworker hit it with ibm_websphere_java_deserialize.

[BeanBagKing]

Hit this issue on scanner/http/http_login

Edit: For anyone looking for a workaround, proxy the traffic through Burp, which will ignore the certificates, e.g. set Proxies http:127.0.0.1:8080 - I would imagine this would work for any module getting this error.

[coblax]

have anybody was solve this problem

[Raphaeangelo]

Bump!
I'm getting

Exploit failed [unreachable]: OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: dh key too small

[cnotin]

Same here with an old server that only supports RC4 and DES which aren't supported anymore since OpenSSL 1.1.0...
See this discussion on the Kali bug tracker: https://bugs.kali.org/view.php?id=5158
Right now Kali rolling uses OpenSSL 1.1.1d-2 but the unsafeopenssl package (see limitations in post) was created and offers version 1.0.2t-0kali1

[lesterpotter]

I'm getting...
Exploit failed [unreachable]: OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: sslv3 alert handshake failure

I'm working around it using Burp with...
set Proxies 127.0.0.1:8080

[pich4ya]

workaround

$ sed -i 's,^\(MinProtocol[ ]*=\).*,\1'TLSv1.0',g' /etc/ssl/openssl.cnf 
$ sed -i 's,^\(CipherString[ ]*=\).*,\1'DEFAULT@SECLEVEL=1',g' /etc/ssl/openssl.cnf

[github-actions]

Hi!

This issue has been left open with no activity for a while now.

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 30 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request.

[BeanBagKing]

As far as I know, this issue still exists and the report should probably be left open.

[Z3d6d]

set SSL false before running the exploit
Make sure you remove the proxy before doing this.

[victorlapshev]

set SSL false before running the exploit
Make sure you remove the proxy before doing this.

But what if target uses SSL? Will it still works if I will disable SSL option?

[BeanBagKing]

set SSL false before running the exploit
Make sure you remove the proxy before doing this.

But what if target uses SSL? Will it still works if I will disable SSL option?

No, I'm pretty sure Z3d6d is misunderstanding what the issue is here.

[bmilliron67]

This is still an issue. I'm getting this error with the f5_icontrol_rce module. I've tried changing the settings in the /etc/ssl/openssl.cnf to no avail. Looks like metasploit has it's own version at /usr/share/metasploit-framework/config/openssl.conf however this doesn't appear to be configured correctly. Nessus ships with it's own version of openssl binary. I think metasploit should do the same.

[adfoster-r7]

Looks like metasploit has it's own version at /usr/share/metasploit-framework/config/openssl.conf however this doesn't appear to be configured correctly

@bmilliron67 When this issue was created, Metasploit didn't have its own openssl conf file - it's only been added recently for openssl 3 support. If there's additional changes to make to the config file now - pull requests are more than welcome 👍

Nessus ships with it's own version of openssl binary. I think metasploit should do the same.

If you install Metasploit via the Rapid7 nightly installer, it should come bundled with openssl 1.1.1 - https://docs.metasploit.com/docs/using-metasploit/getting-started/nightly-installers.html

[mzpqnxow]

As far as I can tell, this is still an issue

It's not an issue with openssl.cnf nor does bundling openssl 1.1.1 address it. The DH key size check has been in place since before 1.1.0 (I'm not sure which exact version added it, but probably somewhere in 1.0.x)

If I remember correctly, it's hard-coded into the openssl SSL/TLS code and can not be turned off- except (maybe) by using a low level custom callback

As another on this thread mentioned, one approach to a workaround would be instructing metasploit to not choose a DH-enabled cipher-suite (as long as some are available on the server)

... however, I can't get this to work. metasploit seems to ignore it without throwing any error as to why (which should perhaps be a new GH issue)

After confirming a specific remote host has a non-DH kex cipher-suite using testssl.sh (which has excellent backwards compatibility):

TLS 1
 x39     DHE-RSA-AES256-SHA                DH 512     AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA                   
 x3a     ADH-AES256-SHA                    DH 512     AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA                   
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA                       
 x33     DHE-RSA-AES128-SHA                DH 512     AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA                   
 x34     ADH-AES128-SHA                    DH 512     AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA                   
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA                       
 x07     IDEA-CBC-SHA                      RSA        IDEA        128      TLS_RSA_WITH_IDEA_CBC_SHA                          
 x18     ADH-RC4-MD5                       DH 512     RC4         128      TLS_DH_anon_WITH_RC4_128_MD5                       
 x05     RC4-SHA                           RSA        RC4         128      TLS_RSA_WITH_RC4_128_SHA                           
 x04     RC4-MD5                           RSA        RC4         128      TLS_RSA_WITH_RC4_128_MD5                           
 x16     EDH-RSA-DES-CBC3-SHA              DH 512     3DES        168      TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA                  
 x1b     ADH-DES-CBC3-SHA                  DH 512     3DES        168      TLS_DH_anon_WITH_3DES_EDE_CBC_SHA                  
 x0a     DES-CBC3-SHA                      RSA        3DES        168      TLS_RSA_WITH_3DES_EDE_CBC_SHA                      
 x15     EDH-RSA-DES-CBC-SHA               DH 512     DES         56       TLS_DHE_RSA_WITH_DES_CBC_SHA                       
 x1a     ADH-DES-CBC-SHA                   DH 512     DES         56       TLS_DH_anon_WITH_DES_CBC_SHA                       
 x09     DES-CBC-SHA                       RSA        DES         56       TLS_RSA_WITH_DES_CBC_SHA    

In msfconsole I then use the following, to force it to a non-DH cipher-suite on TLS1.0:

set SSLCIPHER AES128-SHA
set SSLVERSION TLS1
run

I still end up with:

Exploit failed [unreachable]: OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: dh key too small

I also tried using the IANA/RFC name for the cipher ( TLS_RSA_WITH_AES_256_CBC_SHA) but had the same behavior. Perhaps I'm not using set SSLCIPHER correctly?

I ran into this years ago with an old Python interpreter build I had to maintain- the only way I was able to address it was by hand-patching the check in the openssl source

Should the issue I'm having with SSLCIPHER be a new issue? Or am I doing something wrong there? Surely I should at least be getting a warning that it didn't choose the cipher-suite I told it too, right?

EDIT: Oops, forgot to mention the version- from nightly-installers as of 2/16/2023, msfconsole -v reporting 6.3.3-dev-

[adfoster-r7]

@mzpqnxow Thanks for taking a closer look; Do you have exact replication steps I could follow?

[mzpqnxow]

@mzpqnxow Thanks for taking a closer look; Do you have exact replication steps I could follow?

I can give you commands or a script but I'm away at the moment (on mobile) but some notes for reference if you want to take a stab at it (or to remind me later)

All you should need is a 512 bit dhparams file and a TLS listener to reproduce the DH size issue ("openssl dhparams help" for the first, "openssl s_server help" for the second)

Coincidental timing on this post which is somewhat related

Anyway, to reproduce the issue of DH params too small, generate weak DH params file, generate any self-signed RSA key, run a listener that offers at least one DH Kex cipher-suite and one non-DH kex (you can explicitly specify these using -cipher but the default set should have this) and connect to it with (I assume) any msf exploit/tool (or at least the one from my example) and specify to msf the non-DH cipher-suite and you should still get the DH size error

To reproduce the issue of small RSA keys, generate a certificate with a 512bit RSA keypair. Then generate a proper ECDSA certificate and keypair. Then specify both to s_server and do roughly the same as in the previous example- first specify a cipher-suite in msf that is RSA-based to get the error. Then specify an available EC-based cipher-suite to see that it doesn't actually choose it like it should, and will still throw an RSA key length error

openssl cli is one of the least user-friendly tools I'm aware of if you don't use it often so I can attach commands or a script when I'm back at my desk to save you any hassle

If you have any issues with the cli refusing to generate weak keys, you can use the patched openssl binary from the testssl.sh project on github (in bin/)

Incidentally, I recommend using that patched openssl (which they maintain) for the openssl embedded in metasploit. It's forward and backward compatible; it can do TLS1.3 + ESNI for example, while also maintaining compatibility with SSLv2 (!) and small DH params and RSA keys. It will save you a lot of hassle as they do a great job at maintaining it. It's the best long term solution for your project IMO

EDIT: I may be conflating the RSA key length issue with the DH issue- one may not require any patch, but I'm certain one does. Regardless, the solution is going to require a (deliberately insecure) portable openssl build, and we can work out which is true with testing

[mzpqnxow]

OK, as much as I would rather not pick a random host from Shodan with a wacky configuration, I'm not sure I have the patience (or really, the time, at the moment) to create a self-contained reproduction. So please bear with me as the innocent host 112.95.225.18 experiences an unsolicited TLS connection (joking of course, I don't think this crosses any ethical boundaries)

1. Pick a random exploit that supports TLS (multi/http/hyperic_hq_script_console) for example (preferably one that couldn't accidentally exploit the target, which in this case seems to be a Cisco device)
2. Set the target as SSL, 112.95.225.18:10001
3. Use run

msf6 > use exploit/multi/http/hyperic_hq_script_console
[*] No payload configured, defaulting to linux/x86/meterpreter/reverse_tcp
msf6 exploit(multi/http/hyperic_hq_script_console) > set RHOSTS 112.95.225.18
RHOSTS => 112.95.225.18
msf6 exploit(multi/http/hyperic_hq_script_console) > set RPORT 10001
RPORT => 10001
msf6 exploit(multi/http/hyperic_hq_script_console) > set SSL true
SSL => true
msf6 exploit(multi/http/hyperic_hq_script_console) > run

[-] Exploit failed [unreachable]: OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: dh key too small
[*] Exploit completed, but no session was created.
msf6 exploit(multi/http/hyperic_hq_script_console) > 

OK, it doesn't like the DH key. Let's see what protocols and cipher-suites that endpoint is offering, using testssl.sh

 $ ./testssl.sh -E 112.95.225.18:10001

###########################################################
    testssl.sh       3.2rc2 from https://testssl.sh/dev/
    (88763f4 2023-02-20 20:29:14)

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.0.2-bad (1.0.2k-dev)" [~179 ciphers]
 on xxx:./bin/openssl.Linux.x86_64
 (built: "Sep  1 14:03:44 2022", platform: "linux-x86_64")


 Start 2023-02-21 14:54:52        -->> 112.95.225.18:10001 (112.95.225.18) <<--

 rDNS (112.95.225.18):   --
 Service detected:       HTTP



 Testing ciphers per protocol via OpenSSL plus sockets against the server, ordered by encryption strength 

Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
 - 
SSLv3
 x39     DHE-RSA-AES256-SHA                DH 512     AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA                   
 x3a     ADH-AES256-SHA                    DH 512     AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA                   
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA                       
 x33     DHE-RSA-AES128-SHA                DH 512     AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA                   
 x34     ADH-AES128-SHA                    DH 512     AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA                   
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA                       
 x07     IDEA-CBC-SHA                      RSA        IDEA        128      TLS_RSA_WITH_IDEA_CBC_SHA                          
 x18     ADH-RC4-MD5                       DH 512     RC4         128      TLS_DH_anon_WITH_RC4_128_MD5                       
 x05     RC4-SHA                           RSA        RC4         128      TLS_RSA_WITH_RC4_128_SHA                           
 x04     RC4-MD5                           RSA        RC4         128      TLS_RSA_WITH_RC4_128_MD5                           
 x16     EDH-RSA-DES-CBC3-SHA              DH 512     3DES        168      TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA                  
 x1b     ADH-DES-CBC3-SHA                  DH 512     3DES        168      TLS_DH_anon_WITH_3DES_EDE_CBC_SHA                  
 x0a     DES-CBC3-SHA                      RSA        3DES        168      TLS_RSA_WITH_3DES_EDE_CBC_SHA                      
 x15     EDH-RSA-DES-CBC-SHA               DH 512     DES         56       TLS_DHE_RSA_WITH_DES_CBC_SHA                       
 x1a     ADH-DES-CBC-SHA                   DH 512     DES         56       TLS_DH_anon_WITH_DES_CBC_SHA                       
 x09     DES-CBC-SHA                       RSA        DES         56       TLS_RSA_WITH_DES_CBC_SHA                           
TLS 1
 x39     DHE-RSA-AES256-SHA                DH 512     AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA                   
 x3a     ADH-AES256-SHA                    DH 512     AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA                   
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA                       
 x33     DHE-RSA-AES128-SHA                DH 512     AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA                   
 x34     ADH-AES128-SHA                    DH 512     AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA                   
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA                       
 x07     IDEA-CBC-SHA                      RSA        IDEA        128      TLS_RSA_WITH_IDEA_CBC_SHA                          
 x18     ADH-RC4-MD5                       DH 512     RC4         128      TLS_DH_anon_WITH_RC4_128_MD5                       
 x05     RC4-SHA                           RSA        RC4         128      TLS_RSA_WITH_RC4_128_SHA                           
 x04     RC4-MD5                           RSA        RC4         128      TLS_RSA_WITH_RC4_128_MD5                           
 x16     EDH-RSA-DES-CBC3-SHA              DH 512     3DES        168      TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA                  
 x1b     ADH-DES-CBC3-SHA                  DH 512     3DES        168      TLS_DH_anon_WITH_3DES_EDE_CBC_SHA                  
 x0a     DES-CBC3-SHA                      RSA        3DES        168      TLS_RSA_WITH_3DES_EDE_CBC_SHA                      
 x15     EDH-RSA-DES-CBC-SHA               DH 512     DES         56       TLS_DHE_RSA_WITH_DES_CBC_SHA                       
 x1a     ADH-DES-CBC-SHA                   DH 512     DES         56       TLS_DH_anon_WITH_DES_CBC_SHA                       
 x09     DES-CBC-SHA                       RSA        DES         56       TLS_RSA_WITH_DES_CBC_SHA                           
TLS 1.1
 -
TLS 1.2
 - 
TLS 1.3
 - 

 Done 2023-02-21 14:56:28 [ 102s] -->> 112.95.225.18:10001 (112.95.225.18) <<--

Plenty of cipher-suites to choose, though limited to only TLSv1.0 and SSLv3

If the issue is with the DH key size, let's choose a cipher-suite that has a non-DH kex (how about AES128-SHA, which is available on both SSLv3 and TLS1.0, and doesn't use an ephemeral kex):

msf6 exploit(multi/http/hyperic_hq_script_console) > set SSLCIPHER AES128-SHA
SSLCIPHER => AES128-SHA
msf6 exploit(multi/http/hyperic_hq_script_console) > run

[-] Exploit failed [unreachable]: OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: dh key too small
[*] Exploit completed, but no session was created.

Strange, it still seems to have use a DH kex... we can try to also specify the protocol:

msf6 exploit(multi/http/hyperic_hq_script_console) > set SSLVERSION SSL3
SSLVERSION => SSL3
msf6 exploit(multi/http/hyperic_hq_script_console) > run

[-] Exploit failed [unreachable]: OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: no protocols available
[*] Exploit completed, but no session was created.

No dice. Trying TLS1.0:

msf6 exploit(multi/http/hyperic_hq_script_console) > set SSLVERSION TLS1
SSLVERSION => TLS1
msf6 exploit(multi/http/hyperic_hq_script_console) > run

[-] Exploit failed [unreachable]: OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: dh key too small
[*] Exploit completed, but no session was created.
msf6 exploit(multi/http/hyperic_hq_script_console) > 

For good measure, making sure that msf doesn't want the IANA name for the cipher-suite instead (TLS_RSA_WITH_AES_128_CBC_SHA in this case)

msf6 exploit(multi/http/hyperic_hq_script_console) > set SSLCIPHER TLS_RSA_WITH_AES_128_CBC_SHA
SSLCIPHER => TLS_RSA_WITH_AES_128_CBC_SHA
msf6 exploit(multi/http/hyperic_hq_script_console) > run

[-] Exploit failed [unreachable]: OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: dh key too small
[*] Exploit completed, but no session was created.

Nope

So the issue here is (at least, I think) that msf isn't honoring the SSLCIPHER parameter. I don't know if you want to handle this as a separate issue or not

Control - openssl.Linux.x86_64 binary from testssl.sh

This works OK, it establishes the session and hangs until I either send an HTTP request or control-c

 $ OPENSSL_CONF= ../bin/openssl.Linux.x86_64 s_client -connect 112.95.225.18:10001 -ssl3               
WARNING: can't open config file: 
CONNECTED(00000003)
depth=0 CN = e8:b7:48:83:66:80, OU = RV082, O = "Cisco Systems, Inc.", L = Irvine, C = US, SN = Califomia
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = e8:b7:48:83:66:80, OU = RV082, O = "Cisco Systems, Inc.", L = Irvine, C = US, SN = Califomia
verify error:num=10:certificate has expired
notAfter=Dec 30 08:00:02 2019 GMT
verify return:1
depth=0 CN = e8:b7:48:83:66:80, OU = RV082, O = "Cisco Systems, Inc.", L = Irvine, C = US, SN = Califomia
notAfter=Dec 30 08:00:02 2019 GMT
verify return:1
---
Certificate chain
 0 s:/CN=e8:b7:48:83:66:80/OU=RV082/O=Cisco Systems, Inc./L=Irvine/C=US/SN=Califomia
   i:/CN=e8:b7:48:83:66:80/OU=RV082/O=Cisco Systems, Inc./L=Irvine/C=US/SN=Califomia
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDWTCCAsKgAwIBAgIJANNLXtUkgh2FMA0GCSqGSIb3DQEBBQUAMHwxGjAYBgNV
BAMTEWU4OmI3OjQ4OjgzOjY2OjgwMQ4wDAYDVQQLEwVSVjA4MjEcMBoGA1UEChMT
Q2lzY28gU3lzdGVtcywgSW5jLjEPMA0GA1UEBxMGSXJ2aW5lMQswCQYDVQQGEwJV
UzESMBAGA1UEBBMJQ2FsaWZvbWlhMB4XDTEwMDEwMTA4MDAwMloXDTE5MTIzMDA4
MDAwMlowfDEaMBgGA1UEAxMRZTg6Yjc6NDg6ODM6NjY6ODAxDjAMBgNVBAsTBVJW
MDgyMRwwGgYDVQQKExNDaXNjbyBTeXN0ZW1zLCBJbmMuMQ8wDQYDVQQHEwZJcnZp
bmUxCzAJBgNVBAYTAlVTMRIwEAYDVQQEEwlDYWxpZm9taWEwgZ8wDQYJKoZIhvcN
AQEBBQADgY0AMIGJAoGBAM8DXrm1l3ZO9VUbVZYzC5KQN+VUveaNf4XeQ0utp0y6
n5qhH+38urPjnzeNNMZ4h6ujRmW6pnK+mkVBLlKbkARtpIdreWcPKUHiJXSsugBB
IseFh8Ohjr8zZ8SqnZdr4cX5MRudV3fuanZg0sB7mO+U0ahRwFV6BwHLfXAuGPLR
AgMBAAGjgeIwgd8wHQYDVR0OBBYEFGuZHHmOORM+biEomvQQqQSPD7rEMIGvBgNV
HSMEgacwgaSAFGuZHHmOORM+biEomvQQqQSPD7rEoYGApH4wfDEaMBgGA1UEAxMR
ZTg6Yjc6NDg6ODM6NjY6ODAxDjAMBgNVBAsTBVJWMDgyMRwwGgYDVQQKExNDaXNj
byBTeXN0ZW1zLCBJbmMuMQ8wDQYDVQQHEwZJcnZpbmUxCzAJBgNVBAYTAlVTMRIw
EAYDVQQEEwlDYWxpZm9taWGCCQDTS17VJIIdhTAMBgNVHRMEBTADAQH/MA0GCSqG
SIb3DQEBBQUAA4GBAD6yXn1NxC2fO9XHoCWLRHEMApKNfB31Spu6hU44xcMNRt85
qAmp35lr10Nw9/CJ30S+eU2UTgqBzzCko+21COkOb9qY5fCaEoSOBIYtbN5PJh0V
oZ4L+In5EzyEzwixKu4a9sAIXO2FsdaxyrsQRYWSyI7oKdCcrGM0jZqeRWWZ
-----END CERTIFICATE-----
subject=/CN=e8:b7:48:83:66:80/OU=RV082/O=Cisco Systems, Inc./L=Irvine/C=US/SN=Califomia
issuer=/CN=e8:b7:48:83:66:80/OU=RV082/O=Cisco Systems, Inc./L=Irvine/C=US/SN=Califomia
---
No client certificate CA names sent
Server Temp Key: DH, 512 bits
---
SSL handshake has read 1281 bytes and written 309 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 1A7430F161E4C9F3A877A7FA05B676F92CF7E932DF8AD0A54060A83BABFDE5320FDC1BB7BB7CD586B7C0999C13796C96
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1676992357
    Timeout   : 7200 (sec)
    Verify return code: 10 (certificate has expired)
---
^C

Control - using the openssl binary embedded into metasploit

The openssl binary embedded into metasploit succeeds, but only when the cipher is explicitly set:

 $ /opt/metasploit-framework/embedded/bin/openssl s_client -connect 112.95.225.18:10001 -tls1 -no_check_time -cipher AES256-SHA
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = e8:b7:48:83:66:80, OU = RV082, O = "Cisco Systems, Inc.", L = Irvine, C = US, SN = Califomia
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = e8:b7:48:83:66:80, OU = RV082, O = "Cisco Systems, Inc.", L = Irvine, C = US, SN = Califomia
verify return:1
---
Certificate chain
 0 s:CN = e8:b7:48:83:66:80, OU = RV082, O = "Cisco Systems, Inc.", L = Irvine, C = US, SN = Califomia
   i:CN = e8:b7:48:83:66:80, OU = RV082, O = "Cisco Systems, Inc.", L = Irvine, C = US, SN = Califomia
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDWTCCAsKgAwIBAgIJANNLXtUkgh2FMA0GCSqGSIb3DQEBBQUAMHwxGjAYBgNV
BAMTEWU4OmI3OjQ4OjgzOjY2OjgwMQ4wDAYDVQQLEwVSVjA4MjEcMBoGA1UEChMT
Q2lzY28gU3lzdGVtcywgSW5jLjEPMA0GA1UEBxMGSXJ2aW5lMQswCQYDVQQGEwJV
UzESMBAGA1UEBBMJQ2FsaWZvbWlhMB4XDTEwMDEwMTA4MDAwMloXDTE5MTIzMDA4
MDAwMlowfDEaMBgGA1UEAxMRZTg6Yjc6NDg6ODM6NjY6ODAxDjAMBgNVBAsTBVJW
MDgyMRwwGgYDVQQKExNDaXNjbyBTeXN0ZW1zLCBJbmMuMQ8wDQYDVQQHEwZJcnZp
bmUxCzAJBgNVBAYTAlVTMRIwEAYDVQQEEwlDYWxpZm9taWEwgZ8wDQYJKoZIhvcN
AQEBBQADgY0AMIGJAoGBAM8DXrm1l3ZO9VUbVZYzC5KQN+VUveaNf4XeQ0utp0y6
n5qhH+38urPjnzeNNMZ4h6ujRmW6pnK+mkVBLlKbkARtpIdreWcPKUHiJXSsugBB
IseFh8Ohjr8zZ8SqnZdr4cX5MRudV3fuanZg0sB7mO+U0ahRwFV6BwHLfXAuGPLR
AgMBAAGjgeIwgd8wHQYDVR0OBBYEFGuZHHmOORM+biEomvQQqQSPD7rEMIGvBgNV
HSMEgacwgaSAFGuZHHmOORM+biEomvQQqQSPD7rEoYGApH4wfDEaMBgGA1UEAxMR
ZTg6Yjc6NDg6ODM6NjY6ODAxDjAMBgNVBAsTBVJWMDgyMRwwGgYDVQQKExNDaXNj
byBTeXN0ZW1zLCBJbmMuMQ8wDQYDVQQHEwZJcnZpbmUxCzAJBgNVBAYTAlVTMRIw
EAYDVQQEEwlDYWxpZm9taWGCCQDTS17VJIIdhTAMBgNVHRMEBTADAQH/MA0GCSqG
SIb3DQEBBQUAA4GBAD6yXn1NxC2fO9XHoCWLRHEMApKNfB31Spu6hU44xcMNRt85
qAmp35lr10Nw9/CJ30S+eU2UTgqBzzCko+21COkOb9qY5fCaEoSOBIYtbN5PJh0V
oZ4L+In5EzyEzwixKu4a9sAIXO2FsdaxyrsQRYWSyI7oKdCcrGM0jZqeRWWZ
-----END CERTIFICATE-----
subject=CN = e8:b7:48:83:66:80, OU = RV082, O = "Cisco Systems, Inc.", L = Irvine, C = US, SN = Califomia

issuer=CN = e8:b7:48:83:66:80, OU = RV082, O = "Cisco Systems, Inc.", L = Irvine, C = US, SN = Califomia

---
No client certificate CA names sent
---
SSL handshake has read 991 bytes and written 264 bytes
Verification error: self signed certificate
---
New, SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS NOT supported
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: B8D118BC735532B8D8D3CF2FA223E28B63F0947EB1F79F537A0C809AC643FB0616D5E052B4C11169BFF5FAECAEDAED15
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1676992925
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
---
^C

When the cipher isn't explicitly set to a non-DH kex, it fails with the dh key too small message, which (I think) is suggestive of the original issue (that the embedded OpenSSL does not support short DH keys and may be needing a patch). Here's the output when the embedded openssl is permitted to choose a DH-kex cipher-suite:

  $ /opt/metasploit-framework/embedded/bin/openssl s_client -connect 112.95.225.18:10001 -tls1 -no_check_time -debug -msg >/dev/null
Can't use SSL_get_servername
depth=0 CN = e8:b7:48:83:66:80, OU = RV082, O = "Cisco Systems, Inc.", L = Irvine, C = US, SN = Califomia
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = e8:b7:48:83:66:80, OU = RV082, O = "Cisco Systems, Inc.", L = Irvine, C = US, SN = Califomia
verify return:1
139853909251136:error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small:ssl/statem/statem_clnt.c:2150:

Sorry, I know this is a lot of info, and it's not an ideal "environment" for you to reproduce against. If I have time to rig up the s_server commandline (and generate a similar certificate as this host has) then I will but I can't promise anything. I really just stumbled upon this issue by accident and it's not a big problem for me

[mzpqnxow]

I want to add what may be a very important detail here, about this specific endpoint- the certificate is a bit unusual in that it has:

            X509v3 Basic Constraints: critical
                CA:FALSE

This may be an edge-case within the edge-case for the small DH key issue, I'm not sure having not looked at the code in long time