Multi-threaded check command, plus aux scanner support #2922

wchen-r7 · 2014-01-26T18:57:22Z

This PR addresses the following redmine tickets:

Multi-threaded support for check: http://dev.metasploit.com/redmine/issues/8752
Aux scanner support: http://dev.metasploit.com/redmine/issues/8753
Deprecate ms08_067_check: http://dev.metasploit.com/redmine/issues/8755

It is also the final completion of this ticket:
http://dev.metasploit.com/redmine/issues/8737

Usage:

Basically there are two ways to use this command:

Example 1:

check [an IP or an IP range]

Example 2:

> set [RHOST or RHOSTS] IP
> check

If you wish to have more threads, then do:

set threads [number of threads]

Please note: Having more threads may result in missed opportunities (meaning that even though a host is vulnerable, it may not be flagged).

Verifications

Before you begin, please make sure you have the following for testing:

A Windows XP box vulnerable to ms08-067
A fake web server: ruby -run -e httpd . -p 8181
Have aux_scanner_check_test.rb for testing: https://gist.github.com/wchen-r7/f297c96881f073336208, you should probably place it under: auxiliary/scanner/http because this is a scanner module.

And now let's move on to these scenarios:

Scenario 1: With ms08_067_netapi, set the RHOST and run check

Start msfconsole
Enter command: use exploit/windows/smb/ms08_067_netapi
Enter command: set rhost [IP]
Enter command: check
You should see the check run and flag your test box as vulnerable

Scenario 2: With ms08_067_netapi, check multiple hosts

Start msfconsole
Enter command: use exploit/windows/smb/ms08_067_netapi
Enter command: check [range] (keep the range small or it will run for a long time)
You should see the check run, and eventually flag your test box as vulnerable

Scenario 3: With ms08_067_netapi, set threads, and run multiple hosts

Start msfconsole
Enter command: use exploit/windows/smb/ms08_067_netapi
Enter command: set threads 10
Enter command: check [range] (ideally try to keep the range small for faster testing)
You should see the check to run faster.
You should also see your test box flagged as vulnerable. However, if you set the threads too high, you might not see that.

Scenario 4: With ms08_067_netapi, abort scanning while on a multi-threaded scan

Start msfconsole
Enter command: use exploit/windows/smb/ms08_067_netapi
Enter command: set threads 10
Enter command: check [range] (ideally try to keep the range small for faster testing)
When the check starts to run, press [CTRL]+[C] to abort
You should see "Caught interrupt from the console", and no more scan messages.

Scenario 5: With ms08_067_netapi, run check without any arguments

Start msfconsole
Enter command: use exploit/windows/smb/ms08_067_netapi
Enter command: check
You should see an option validation error about "RHOST"

Scenario 6: Run nodejs_pipelining.rb against the fake web server

Start msfconsole
Enter command: use auxiliary/dos/http/nodejs_pipelining
Enter command: set rport [fake web server PORT]
Enter command: check [fake web server IP]
You should see a check message

Scenario 7: Run aux_scanner_check_test against the web server

Start msfconsole
Enter command: use [path to aux_scanner_check_test.rb]
Enter command: set rport [fake web server rport]
Enter command: check [web server IP]
You should see a check message

Scenario 8: With aux_scanner_check_test, run check without any arguments

Start msfconsole
Enter command: use [path to aux_scanner_check_test.rb]
Enter command: check
You should see an option validation error about "RHOSTS"

Scenario 9: Run a module that does not support check at all

Start msfconsole
Enter command: use auxiliary/scanner/http/soap_xml
Enter command: set rhosts [IP]
Enter command: check
You should see: "This module does not support check."

Auxiliary modules can use check, too. Not just exploits.

If threads aren't killed, then when the user triggers interrupt, the console will keep the threads (vuln checks) running, which looks weird.

Code changes address these feature requests: [SeeRM rapid7#8737] [SeeRM rapid7#8752] [SeeRM rapid7#8755]

[SeeRM rapid7#8755]

limhoff-r7 · 2014-01-26T20:47:31Z

lib/msf/core/auxiliary/scanner.rb

@@ -32,6 +32,17 @@ def initialize(info = {})
 end


+def check


The returns are unnecessary as the begin rescue will be treated as a single statement and if there is no error then the check_host's return will be return value and if there is an error then the rescue's value will be the return value. Also remove the unnecessary self when calling replicant.

def check nmod = replicant begin nmod.check_host(datastore['RHOST]) rescue NoMethodError Exploit::CheckCode::Unsupported end end

Applied in eec01e7

limhoff-r7 · 2014-01-27T02:39:47Z

lib/msf/ui/console/module_command_dispatcher.rb

+        break unless ip
+
+        @tl << framework.threads.spawn("CheckHost-#{ip}", false, ip.dup) { |tip|
+          mod.datastore['RHOST'] = tip


This is still not thread-safe, but github's hiding our discussion because it was on an outdated diff. To fix this you need to make the threads not share mutable data or only share read-only data. The easiest way I can think of to do this is to create a replicant on line 85 in the parent thread and then use that replicant in the child thread block of 87 to 88. This will be inefficient since check_simple itself calls replicant internally, but modifying the datastore has to happen on a separate module instance for each thread:

instance = mod.replicant instance.datastore['RHOST'] = ip.dup thread = framework.threads.spawn("CheckHost-#{ip}", false) { instance.check_simple } @tl << thread

I moved setting the datastore outside the thread since it doesn't take that long and doesn't really need to be threaded.

Hmm... actually calling replicant should not affect the original mod, but could take some time since it's duping a lot of variables, so it's probably better to move it into the thread.

thread = framework.threads.spawn("CheckHost-#{ip}", false) { instance = mod.replicant instance.datastore['RHOST'] = ip.dup instance.check_simple } @tl << thread

The ip.dup is still necessary in case any of the code that uses 'RHOST' uses mutators like strip! that would modify the String object instead of just replacing it in the data store.

So this solution doesn't actually work because "instance" doesn't have check_simple(). Luke, what was the other solution you were talking about at the airport? Did you say extend a module in there? So check_simple() exists in both exploit and auxiliary, not sure how you're supposed to extend that in a more universal way.... could you please repeat what you said? Thanks.

You need to emulate what happens when you call Msf::ModuleSet#create: The instance is initialized from the class and then the creation event is sent to the framework:

instance = mod.replicant instance.datastore['RHOST'] = ip.dup framework.events.on_module_created(instance) instance.check_simple

The framework.events.on_module_created is implemented as Msf::Simple::Framework#on_module_created:

def on_module_created(instance) Msf::Simple::Framework.simplify_module(instance) end

Msf::Simple::Framework.simplify_module extends the instance with a module_type specific Simple module:

instance.extend(ModuleSimplifiers[instance.type])

It is this extend that finally gives your instance the check_simple method.

And, no, it wasn't immediately obvious that the event system would add the simple methods to me either. I only figured it out when I left out the framework.events.on_module_created call on my module caching branches. The Simple namespace in lib/msf/base should probably be merged into lib/msf/core's normal modules and classes since nothing really works with non-simplified modules and frameworks, but that's for later for me at least after rounds 1 and 2 of the module cache are complete.

Ah yes, that works. Thank you!

Applied in 9f669a8

wchen-r7 · 2014-01-31T00:36:01Z

So if I'm not mistaken, all concerns are addressed at this point. I'm gonna go ahead and retest everything now. And Luke, if you see any more (or things that still aren't properly addressed), please let me know. And thanks again.

wchen-r7 · 2014-01-31T01:21:59Z

I finished testing and only had to fix ::Interrupt exception handling. Also passed Travis again. This PR is back on track.

wvu · 2014-02-03T18:09:45Z

Testing...

wvu · 2014-02-03T18:13:36Z

Checking boxes, rather. :)

wvu · 2014-02-03T18:18:41Z

I checked all the boxes. Phew.

wvu · 2014-02-03T19:26:25Z

Scenario 1:

msf > use exploit/windows/smb/ms08_067_netapi 
msf exploit(ms08_067_netapi) > set RHOST 172.16.126.128
RHOST => 172.16.126.128
msf exploit(ms08_067_netapi) > check
[+] 172.16.126.128:445 - The target is vulnerable.
msf exploit(ms08_067_netapi) >

Scenario 2:

msf exploit(ms08_067_netapi) > unset RHOST 
Unsetting RHOST...
msf exploit(ms08_067_netapi) > check 172.16.126.127-130
[*] 172.16.126.127:445 - Cannot reliably check exploitability.
[+] 172.16.126.128:445 - The target is vulnerable.
[*] 172.16.126.129:445 - The target is not exploitable.
[*] 172.16.126.130:445 - Cannot reliably check exploitability.
msf exploit(ms08_067_netapi) >

Scenario 3:

msf exploit(ms08_067_netapi) > set THREADS 10
THREADS => 10
msf exploit(ms08_067_netapi) > check 172.16.126.127-130
[*] 172.16.126.129:445 - The target is not exploitable.
[+] 172.16.126.128:445 - The target is vulnerable.
[*] 172.16.126.130:445 - Cannot reliably check exploitability.
[*] 172.16.126.127:445 - Cannot reliably check exploitability.
msf exploit(ms08_067_netapi) >

Scenario 4:

msf exploit(ms08_067_netapi) > check 172.16.126.127-130
[*] 172.16.126.129:445 - The target is not exploitable.
[+] 172.16.126.128:445 - The target is vulnerable.
^C[*] Caught interrupt from the console...
msf exploit(ms08_067_netapi) >

Scenario 5:

msf exploit(ms08_067_netapi) > check
[-] Check failed: The following options failed to validate: RHOST.
msf exploit(ms08_067_netapi) >

Scenario 6:

msf exploit(ms08_067_netapi) > use auxiliary/dos/http/nodejs_pipelining 
msf auxiliary(nodejs_pipelining) > set RPORT 8181
RPORT => 8181
msf auxiliary(nodejs_pipelining) > check 127.0.0.2
[*] 127.0.0.2:8181 - The target appears to be vulnerable.
msf auxiliary(nodejs_pipelining) >

Scenario 7:

msf auxiliary(nodejs_pipelining) > use auxiliary/scanner/http/aux_scanner_check_test 
msf auxiliary(aux_scanner_check_test) > set RPORT 8181
RPORT => 8181
msf auxiliary(aux_scanner_check_test) > check 127.0.0.2
[+] 127.0.0.2:8181 - The target is vulnerable.
[*] Checked 1 of 1 hosts (100% complete)
msf auxiliary(aux_scanner_check_test) >

Scenario 8:

msf auxiliary(aux_scanner_check_test) > check
[-] Check failed: The following options failed to validate: RHOSTS.
msf auxiliary(aux_scanner_check_test) >

Scenario 9:

msf auxiliary(aux_scanner_check_test) > use auxiliary/scanner/http/soap_xml 
msf auxiliary(soap_xml) > set RHOSTS 127.0.0.2
RHOSTS => 127.0.0.2
msf auxiliary(soap_xml) > check
[*] 127.0.0.2:80 - This module does not support check.
[*] Checked 1 of 1 hosts (100% complete)
msf auxiliary(soap_xml) >

wchen-r7 · 2014-02-03T20:28:33Z

well done sir.

wchen-r7 · 2014-02-03T20:33:05Z

Looks like testing is complete. Please merge. Thanks.

wchen-r7 · 2014-02-03T20:50:30Z

Talked to Tod. Since there's an issue with other things in framework, we're holding this one to make sure we have a safe upcoming release. Should probably go in the same time as #2930, since they're two non-small effort features.

wvu · 2014-02-03T21:47:57Z

Roger.

wvu · 2014-02-04T17:25:22Z

Thanks!!

wchen-r7 added 14 commits January 24, 2014 17:54

aux scanner support

93fa58e

Save progress

216fa45

This one looks like is working

2046209

Merge the broken module fix because it's annoying

9cc7a88

Change default thread count

7dfd4ab

Was playing with ms08_067 check and realized I forgot this print

cc4dea7

Check argument

3bb17da

NoMethod check and stuff

2d12c0a

Fix option validation

60f1688

Change Unsupported message

6ffb750

Auxiliary modules can use check, too. Not just exploits.

Make sure all threads are killed after interrupt

f0ebd13

If threads aren't killed, then when the user triggers interrupt, the console will keep the threads (vuln checks) running, which looks weird.

Show warning

a14dddd

Last commit before PR

48836b4

Code changes address these feature requests: [SeeRM rapid7#8737] [SeeRM rapid7#8752] [SeeRM rapid7#8755]

ms08_067_check.rb is deprecated.

f471f50

[SeeRM rapid7#8755]

limhoff-r7 reviewed Jan 26, 2014
View reviewed changes

wchen-r7 added 4 commits January 26, 2014 16:25

No explicit "return"

eec01e7

Loop do it

45bb336

{ } block this

0ffacc3

loop do this too

6435ddd

limhoff-r7 reviewed Jan 27, 2014
View reviewed changes

Make check_multiple() thread-safe

9f669a8

Fix ::Interrupt exception handling

4d008ca

wvu added a commit that referenced this pull request Feb 4, 2014

Land #2922, multithreaded check command

a58698c

wvu merged commit 4d008ca into rapid7:master Feb 4, 2014

wchen-r7 mentioned this pull request May 24, 2016

Fix check command with an IP or IP range #6907

Merged

4 tasks

wchen-r7 deleted the check branch August 22, 2016 16:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-threaded check command, plus aux scanner support #2922

Multi-threaded check command, plus aux scanner support #2922

wchen-r7 commented Jan 26, 2014

limhoff-r7 Jan 26, 2014

wchen-r7 Jan 26, 2014

wchen-r7 Jan 30, 2014

limhoff-r7 Jan 27, 2014

limhoff-r7 Jan 27, 2014

wchen-r7 Jan 30, 2014

limhoff-r7 Jan 30, 2014

limhoff-r7 Jan 30, 2014

wchen-r7 Jan 30, 2014

wchen-r7 Jan 30, 2014

wchen-r7 commented Jan 31, 2014

wchen-r7 commented Jan 31, 2014

wvu commented Feb 3, 2014

wvu commented Feb 3, 2014

wvu commented Feb 3, 2014

wvu commented Feb 3, 2014

wchen-r7 commented Feb 3, 2014

wchen-r7 commented Feb 3, 2014

wchen-r7 commented Feb 3, 2014

wvu commented Feb 3, 2014

wvu commented Feb 4, 2014

Multi-threaded check command, plus aux scanner support #2922

Multi-threaded check command, plus aux scanner support #2922

Conversation

wchen-r7 commented Jan 26, 2014

Usage:

Verifications

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wchen-r7 commented Jan 31, 2014

wchen-r7 commented Jan 31, 2014

wvu commented Feb 3, 2014

wvu commented Feb 3, 2014

wvu commented Feb 3, 2014

wvu commented Feb 3, 2014

wchen-r7 commented Feb 3, 2014

wchen-r7 commented Feb 3, 2014

wchen-r7 commented Feb 3, 2014

wvu commented Feb 3, 2014

wvu commented Feb 4, 2014