Reduce Unknown commands handling from 1 second to 0.5 seconds

[agalway-r7]

Cuts down unknown command time from 1 second to 0.5 seconds

Creates a class instance key for signing JARs. This cuts down on creation time for new android modules and as a result speeds up boot and incorrect command reset times by 0.5 seconds.

Verification

• Start msfconsole
• jimbob
• [-] Unknown command: jimbob. Should be output instantly
• exploit/windows/misc/crosschex_device_bof
• [-] Unknown command: exploit/windows/misc/crosschex_device_bof.
• This is a module we can load. Do you want to use exploit/windows/misc/crosschex_device_bof? [y/N] Should be output instantly

TL;DR

• Implementing a constant signing key of size 2048 cuts down incorrect command time down by half. Still feels a little slow but it’s a definite improvement.
• Constant signing key of size 1024 seems to improve speed by roughly 0.1s over size 2048. Whether it’s worth it depends on what we get out of the larger key. IMO I’d keep the larger key, with all the difference cutting down the size makes in time.
• Using the key lookup is by far the fastest method, by virtue of avoiding module creation and JAR signing altogether. However I’ve no idea if that breaks any functionality so can’t recommend it atm. Most likely another PR.

[adfoster-r7]

The function that takes ~1 second to execute is the PayloadSet's recalculate method. There's a comment on the code that this approach may not work as we expect:

metasploit-framework/lib/msf/core/payload_set.rb

Lines 7 to 17 in 4c06146

	###
	#
	# This class is a special case of the generic module set class because
	# payloads are generated in terms of combinations between various
	# components, such as a stager and a stage. As such, the payload set
	# needs to be built on the fly and cannot be simply matched one-to-one
	# with a payload module. Yeah, the term module is kind of overloaded
	# here, but eat it!
	#
	###
	class PayloadSet < ModuleSet

We might need a different approach to the above, but it's worth double checking.

[adfoster-r7]

Poking further at this; I'm concerned that this approach breaks the law of demeter, and may not work well with some of the assumptions that are already in place within framework. In particular it seems like module set already has some nil handling in place:

metasploit-framework/lib/msf/core/module_set.rb

Lines 42 to 50 in 4c06146

	# If there is no module associated with this class, then try to demand
	# load it.
	if klass.nil? or klass == Msf::SymbolicModule
	framework.modules.load_cached_module(module_type, reference_name)
	recalculate
	klass = fetch(reference_name, nil)
	end

Which is actually why the slow recalculate method gets called in the first place. It might be worth doing a bit of git fu to learn why that logic was added in the first place, to make sure this approach doesn't break existing logic etc.

[adfoster-r7]

Cross referencing some discussion on bootup/missing command performance - #7292 (comment)

[adfoster-r7]

Thoughts on making this lazy instantiated? i.e. Loading this module doesn't create a key, but the first access will:

def signing_key
  @@signing_key ||= OpenSSL::PKey::RSA.new(2048)
end

[agalway-r7]

Looks good, will add it in 👍

[adfoster-r7]

Hm; Maybe drop the new_key argument here until there's a valid use case for it? 🤔

Just thinking: If we let this becomes configurable, and every android payload starts generating its own key - then we're back to the same position as before.

[agalway-r7]

This is true ^^ I'll save it for when it's 100% needed

[agalway-r7]

Using this to time creation of all module types:

start = Time.now
module_instance = set.create(module_reference_name)
puts Time.now - start

These are the unchanged times from master:

0.001121
1.009482 <-- Payload Set (Includes JAR Signing Code)
0.002953
0.001575
0.001481
0.001276
0.000813

Time with cached 2048 JAR signing key:

0.000993
0.497179 <-- Payload Set (Includes JAR Signing Code)
0.001081
0.000687
0.000395
0.000522
0.000399

Time with cached 1024 JAR signing key:

0.001161
0.350745 <-- Payload Set (Includes JAR Signing Code)
0.001392
0.000904
0.000475
0.000554
0.000436

Time with using keys in payload_set and module_set to determine if a path points to a module (Signing Key size doesn’t matter here):

4.0e-06
3.0e-06 <-- Payload Set (Includes JAR Signing Code)
2.0e-06
2.0e-06
2.0e-06
2.0e-06
1.0e-06

Conclusions:

• Implementing a constant signing key of size 2048 cuts down incorrect command time down by half. Still feels a little slow but it’s a definite improvement.
• Constant signing key of size 1024 seems to improve speed by roughly 0.1s over size 2048. Whether it’s worth it depends on what we get out of the larger key. IMO I’d keep the larger key, with all the difference cutting down the size makes in time.
• Using the key lookup is by far the fastest method, by virtue of avoiding module creation and JAR signing altogether. However I’ve no idea if that breaks any functionality so can’t recommend it atm. Most likely another PR.

[timwr]

Great work! This seems to reduce the startup time (but it's similar to #13171 )

$ git checkout fb06bc0
$ time for i in $(seq 1 5); do msfconsole -qx "exit"; done

real    0m29.137s
user    0m25.340s
sys     0m3.491s
$ git checkout master
Previous HEAD position was fb06bc09f5 remove new_key option and lazy loads static key
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
$ time for i in $(seq 1 5); do msfconsole -qx "exit"; done

real    0m33.000s
user    0m28.664s
sys     0m4.001s
$ git checkout fix_android_pkey
Switched to branch 'fix_android_pkey'
$ time for i in $(seq 1 5); do msfconsole -qx "exit"; done

real    0m29.917s
user    0m25.692s
sys     0m3.917s
$

I'm going to land this. If after landing this #13171 still makes a noticeable difference we can consider landing that too.

[cdelafuente-r7]

Release Notes

This reduces unknown commands handling from 1 second to 0.5 seconds for Android payloads.