Leverage the module metadata cache in the module_sets #18704

dwelch-r7 · 2024-01-15T15:25:12Z

This PR resolves an issues introduced when the defer-module-loads feature was added into framework
essentially the issue was the module sets (payloads, exploits, post...etc) weren't being populated with the the module data that several places were referencing in framework, they would only contain any modules that had been actively loaded by the user

The fix I've made for this is to make the module sets aware of the module metadata cache and to refactor some calls using the module sets so we don't accidentally force all modules to be loaded again slowing our boot times
I was pretty conservative in doing this so there's definitely some performance left on the table here but I don't think it's much

Verification steps

CI passes
use an evasion module like windows/applocker_evasion_install_util and do run -n <tab><tab> and you should see the available nop modules auto completing
repeat with run -p for payloads and run -e for encoders
Spin up the RPC service
check that calls to module.exploits module.nops etc. return all the module refnames for that type

example rpc call:

curl --request POST \
  --url http://127.0.0.1:8081/api/v1/json-rpc \
  --data '{
	"jsonrpc": "2.0",
	"method": "module.nops",
	"id": 1,
	"params": []
}'

response:

{
	"jsonrpc": "2.0",
	"result": {
		"modules": [
			"aarch64/simple",
			"armle/simple",
			"cmd/generic",
			"mipsbe/better",
			"php/generic",
			"ppc/simple",
			"sparc/random",
			"tty/generic",
			"x64/simple",
			"x86/opty2",
			"x86/single_byte"
		]
	},
	"id": 1
}

lib/msf/core/module/platform_list.rb

adfoster-r7 · 2024-01-16T11:36:27Z

Can we add an extra test slice with the feature flag enabled, similar to

https://github.com/rapid7/metasploit-framework/blob/57f97ac79ed6c163f3b67de01fdda6af9fb60706/.github/workflows/verify.yml#L76C1-L79

metasploit-framework/spec/spec_helper.rb

Lines 162 to 166 in 57f97ac

    
           if ENV['DATASTORE_FALLBACKS'] 
        
             config.before(:suite) do 
        
               Msf::FeatureManager.instance.set(Msf::FeatureManager::DATASTORE_FALLBACKS, true) 
        
             end 
        
           end

I didn't implement anything generic at the time, but maybe we should? 🤷
Potentially a naming convention of MSF_FEATURE_DATASTORE_FALLBACKS or something? Not a blocker for me. Copy/pasta works.

It might make sense to add this to the acceptance tests too, your call -

metasploit-framework/.github/workflows/acceptance.yml

Lines 58 to 77 in 57f97ac

    
           meterpreter: 
        
             # Python 
        
             - { name: python, runtime_version: 3.6 } 
        
             - { name: python, runtime_version: 3.11 } 
        
             # Java - newer versions of Java are not supported currently: https://github.com/rapid7/metasploit-payloads/issues/647 
        
             - { name: java, runtime_version: 8 } 
        
             # PHP 
        
             - { name: php, runtime_version: 5.3 } 
        
             - { name: php, runtime_version: 7.4 } 
        
             - { name: php, runtime_version: 8.2 } 
        
           include: 
        
             # Windows Meterpreter 
        
             - { meterpreter: { name: windows_meterpreter }, os: windows-2019 } 
        
             - { meterpreter: { name: windows_meterpreter }, os: windows-2022 } 
        
             # Mettle 
        
             - { meterpreter: { name: mettle }, os: macos-11 } 
        
             - { meterpreter: { name: mettle }, os: ubuntu-20.04 }

Maybe another Java latest and greatest slice would work here

spec/lib/msf/core/module_set_spec.rb

test/modules/post/test/all.rb

tools/modules/module_ports.rb

adfoster-r7 · 2024-01-16T11:51:42Z

After we wire up those feature flags to be set during tests, we can regenerate the hyperfine before/after timings afterwards to confirm there's not a shift in performance

adfoster-r7 · 2024-01-18T13:15:37Z

spec/lib/msf/core/encoded_payload_spec.rb

@@ -17,6 +17,11 @@
      module_type: 'encoder',
      modules_path: modules_path,
    )
+    allow(framework.encoders).to receive(:each_module_ranked)
+                                   .and_yield('x86/shikata_ga_nai', framework.encoders['x86/shikata_ga_nai'])


Is there any additional context to this change

Just wondering if it's a a workaround for an implementation change that we're fixing in tests? 👀

sure, so the test was originally just loading these 4 modules into the encoders module set, and since those were the only 4 loaded only they would be available, I'm just updating the mocking to only have these same 4 modules being returned from each_module_ranked which now uses the metadata to get its list of modules rather than from what modules are loaded into the module set

Oh so expect_to_load_module_ancestors actually has the side effect of forcing modules to load

I don't feel super comfortable having the each_module_ranked call mocked here, as it feels like the existing test is verifying a lot of metasploit (if we're treating this as more of an integration test). Not a hard blocker for me, it's just we don't have existing tests for each_module_ranked from what I can see from a 3 second look 🕵️

would you be more okay with mocking out the call to getting the metadata instead?

What are your thoughts on this? As well as changing the spec/lib/msf/core/module_set_spec.rb test to verify that ranked_modules is actually deterministic

diff --git a/lib/msf/core/module_set.rb b/lib/msf/core/module_set.rb index 175a32adc4..7ad1207411 100644 --- a/lib/msf/core/module_set.rb +++ b/lib/msf/core/module_set.rb @@ -274,7 +274,7 @@ class Msf::ModuleSet < Hash # @return [Array<Array<String, Class>>] Array of arrays where the inner array is a pair of the module reference name # and the module class. def rank_modules - module_metadata.sort_by { |refname, _metadata| module_rank(refname) }.reverse! + module_metadata.sort_by { |refname, _metadata| "#{module_rank(refname)}-#{refname}" }.reverse! end # Retrieves the rank from a loaded, not-yet-loaded, or unloadable Metasploit Module. diff --git a/spec/lib/msf/core/encoded_payload_spec.rb b/spec/lib/msf/core/encoded_payload_spec.rb index 88f9dcb823..cbea72498d 100644 --- a/spec/lib/msf/core/encoded_payload_spec.rb +++ b/spec/lib/msf/core/encoded_payload_spec.rb @@ -5,23 +5,28 @@ RSpec.describe Msf::EncodedPayload do include_context 'Msf::Simple::Framework#modules loading' before do + ancestor_reference_names = [ + # Excellent rank + 'x86/shikata_ga_nai', + # Great rank + 'x86/call4_dword_xor', + 'x86/xor_dynamic', + 'generic/none', + ] expect_to_load_module_ancestors( - ancestor_reference_names: [ - # Excellent rank - 'x86/shikata_ga_nai', - # Great rank - 'x86/call4_dword_xor', - 'x86/xor_dynamic', - 'generic/none', - ], + ancestor_reference_names: ancestor_reference_names, module_type: 'encoder', modules_path: modules_path, ) - allow(framework.encoders).to receive(:each_module_ranked) - .and_yield('x86/shikata_ga_nai', framework.encoders['x86/shikata_ga_nai']) - .and_yield('x86/xor_dynamic', framework.encoders['x86/xor_dynamic']) - .and_yield('x86/call4_dword_xor', framework.encoders['x86/call4_dword_xor']) - .and_yield('generic/none', framework.encoders['generic/none']) + + # Improve test performance - return only the test modules that we're interested in + allow(framework.encoders).to receive(:rank_modules).and_wrap_original do |original, *args| + ranked_modules = original.call(*args) + + ranked_modules.select do |ref_name, _metadata| + ancestor_reference_names.include?(ref_name) + end + end end let(:ancestor_reference_names) {

🤔

totally agree it's an improvement, have applied the patch

cgranleese-r7 · 2024-01-18T14:01:54Z

Before

Module types relevant to the flag were not tab completed:

msf6 evasion(windows/applocker_evasion_install_util) > run -p
run -p ConsoleLogging=     run -p LogLevel=           run -p PAYLOAD=            run -p PromptTimeFormat=   run -p TARGET=             run -p WORKSPACE=
run -p ENCODER=            run -p MeterpreterPrompt=  run -p Prompt=             run -p SessionLogging=     run -p TimestampOutput=
run -p FILENAME=           run -p MinimumRank=        run -p PromptChar=         run -p SessionTlvLogging=  run -p VERBOSE=

After

Auto completion

`handler -e`

msf6 payload(windows/x64/meterpreter_reverse_tcp) > handler -e
handler -e cmd/brace                     handler -e mipsbe/longxor                handler -e x64/xor_dynamic               handler -e x86/context_cpuid             handler -e x86/shikata_ga_nai
handler -e cmd/echo                      handler -e mipsle/byte_xori              handler -e x64/zutto_dekiru              handler -e x86/context_stat              handler -e x86/single_static_bit
handler -e cmd/generic_sh                handler -e mipsle/longxor                handler -e x86/add_sub                   handler -e x86/context_time              handler -e x86/unicode_mixed
handler -e cmd/ifs                       handler -e php/base64                    handler -e x86/alpha_mixed               handler -e x86/countdown                 handler -e x86/unicode_upper
handler -e cmd/perl                      handler -e ppc/longxor                   handler -e x86/alpha_upper               handler -e x86/fnstenv_mov               handler -e x86/xor_dynamic
handler -e cmd/powershell_base64         handler -e ppc/longxor_tag               handler -e x86/avoid_underscore_tolower  handler -e x86/jmp_call_additive         handler -e x86/xor_poly
handler -e cmd/printf_php_mq             handler -e ruby/base64                   handler -e x86/avoid_utf8_tolower        handler -e x86/nonalpha
handler -e generic/eicar                 handler -e sparc/longxor_tag             handler -e x86/bloxor                    handler -e x86/nonupper
handler -e generic/none                  handler -e x64/xor                       handler -e x86/bmp_polyglot              handler -e x86/opt_sub
handler -e mipsbe/byte_xori              handler -e x64/xor_context               handler -e x86/call4_dword_xor           handler -e x86/service

`handler -p`

msf6 payload(windows/x64/meterpreter_reverse_tcp) > handler -p
Display all 1388 possibilities? (y or n)
handler -p aix/ppc/shell_bind_tcp                                             handler -p cmd/windows/powershell/x64/custom/reverse_tcp_rc4
handler -p aix/ppc/shell_find_port                                            handler -p cmd/windows/powershell/x64/custom/reverse_tcp_uuid
handler -p aix/ppc/shell_interact                                             handler -p cmd/windows/powershell/x64/custom/reverse_winhttp
handler -p aix/ppc/shell_reverse_tcp                                          handler -p cmd/windows/powershell/x64/custom/reverse_winhttps
handler -p android/meterpreter/reverse_http                                   handler -p cmd/windows/powershell/x64/encrypted_shell/reverse_tcp
handler -p android/meterpreter/reverse_https                                  handler -p cmd/windows/powershell/x64/exec
handler -p android/meterpreter/reverse_tcp                                    handler -p cmd/windows/powershell/x64/loadlibrary
handler -p android/meterpreter_reverse_http                                   handler -p cmd/windows/powershell/x64/messagebox
handler -p android/meterpreter_reverse_https                                  handler -p cmd/windows/powershell/x64/meterpreter/bind_ipv6_tcp
handler -p android/meterpreter_reverse_tcp                                    handler -p cmd/windows/powershell/x64/meterpreter/bind_ipv6_tcp_uuid
handler -p android/shell/reverse_http                                         handler -p cmd/windows/powershell/x64/meterpreter/bind_named_pipe
handler -p android/shell/reverse_https                                        handler -p cmd/windows/powershell/x64/meterpreter/bind_tcp

`exploit -n`

msf6 exploit(windows/smb/ms17_010_psexec) > exploit -n
Display all 165 possibilities? (y or n)
exploit -n ALLOW_GUEST=                             exploit -n MSI::Custom=                             exploit -n Powershell::sub_funcs=                   exploit -n SMBDomain=
exploit -n AutoLoadStdapi=                          exploit -n MSI::EICAR=                              exploit -n Powershell::sub_vars=                    exploit -n SMBName=
exploit -n AutoRunScript=                           exploit -n MSI::Path=                               exploit -n Powershell::wrap_double_quotes=          exploit -n SMBPass=
exploit -n AutoSystemInfo=                          exploit -n MSI::Template=                           exploit -n PrependMigrate=                          exploit -n SMBUser=
exploit -n AutoUnhookProcess=                       exploit -n MSI::UAC=                                exploit -n PrependMigrateProc=                      exploit -n SSL=
exploit -n AutoVerifySessionTimeout=                exploit -n MeterpreterDebugBuild=                   exploit -n Prompt=                                  exploit -n SSLCipher=
exploit -n CHOST=                                   exploit -n MeterpreterDebugLogging=                 exploit -n PromptChar=                              exploit -n SSLServerNameIndication=
exploit -n CMD::DELAY=                              exploit -n MeterpreterPrompt=                       exploit -n PromptTimeFormat=                        exploit -n SSLVerifyMode=
exploit -n CPORT=                                   exploit -n MinimumRank=                             exploit -n Proxies=                                 exploit -n SSLVersion=
exploit -n CheckModule=                             exploit -n NAMEDPIPE=                               exploit -n RHOSTS=                                  exploit -n SessionCommunicationTimeout=
exploit -n ConnectTimeout=                          exploit -n NAMED_PIPES=                             exploit -n RPORT=                                   exploit -n SessionExpirationTimeout=
exploit -n ConsoleLogging=                          exploit -n NOP=                                     exploit -n ReverseAllowProxy=                       exploit -n SessionLogging=
exploit -n ContextInformationFile=                  exploit -n NTLM::SendLM=                            exploit -n ReverseListenerBindAddress=              exploit -n SessionRetryTotal=
exploit -n DBGTRACE=                                exploit -n NTLM::SendNTLM=                          exploit -n ReverseListenerBindPort=                 exploit -n SessionRetryWait=
exploit -n DCERPC::ReadTimeout=                     exploit -n NTLM::SendSPN=                           exploit -n ReverseListenerComm=                     exploit -n SessionTlvLogging=
exploit -n DCERPC::fake_bind_multi=                 exploit -n NTLM::UseLMKey=                          exploit -n ReverseListenerThreaded=                 exploit -n StageEncoder=
exploit -n DCERPC::fake_bind_multi_append=          exploit -n NTLM::UseNTLM2_session=                  exploit -n Rhostname=                               exploit -n StageEncoderSaveRegisters=
exploit -n DCERPC::fake_bind_multi_prepend=         exploit -n NTLM::UseNTLMv2=                         exploit -n SERVICE_DESCRIPTION=                     exploit -n StageEncodingFallback=
exploit -n DCERPC::max_frag_size=                   exploit -n PASSWORD=                                exploit -n SERVICE_DISPLAY_NAME=                    exploit -n StagerRetryCount=
exploit -n DCERPC::smb_pipeio=                      exploit -n PAYLOAD=                                 exploit -n SERVICE_FILENAME=                        exploit -n StagerRetryWait=
exploit -n DOMAIN=                                  exploit -n PSH_PATH=                                exploit -n SERVICE_NAME=                            exploit -n TARGET=
exploit -n DisablePayloadHandler=                   exploit -n PayloadBindPort=                         exploit -n SERVICE_PERSIST=                         exploit -n TCP::max_send_size=
exploit -n DomainControllerRhost=                   exploit -n PayloadProcessCommandLine=               exploit -n SERVICE_STUB_ENCODER=                    exploit -n TCP::send_delay=
exploit -n ENCODER=                                 exploit -n PayloadUUIDName=                         exploit -n SHARE=                                   exploit -n TimestampOutput=
exploit -n EXE::Custom=                             exploit -n PayloadUUIDRaw=                          exploit -n SMB::AlwaysEncrypt=                      exploit -n USERNAME=
exploit -n EXE::EICAR=                              exploit -n PayloadUUIDSeed=                         exploit -n SMB::Auth=                               exploit -n VERBOSE=
exploit -n EXE::FallBack=                           exploit -n PayloadUUIDTracking=                     exploit -n SMB::ChunkSize=                          exploit -n WORKSPACE=
exploit -n EXE::Inject=                             exploit -n PingbackRetries=                         exploit -n SMB::Krb5Ccname=                         exploit -n WfsDelay=
exploit -n EXE::OldMethod=                          exploit -n PingbackSleep=                           exploit -n SMB::KrbOfferedEncryptionTypes=          exploit -n aarch64/simple
exploit -n EXE::Path=                               exploit -n Powershell::encode_final_payload=        exploit -n SMB::Native_LM=                          exploit -n armle/simple
exploit -n EXE::Template=                           exploit -n Powershell::encode_inner_payload=        exploit -n SMB::Native_OS=                          exploit -n cmd/generic
exploit -n EXITFUNC=                                exploit -n Powershell::exec_in_place=               exploit -n SMB::Rhostname=                          exploit -n mipsbe/better
exploit -n EnableContextEncoding=                   exploit -n Powershell::exec_rc4=                    exploit -n SMB::VerifySignature=                    exploit -n php/generic
exploit -n EnableStageEncoding=                     exploit -n Powershell::method=                      exploit -n SMB::obscure_trans_pipe_level=           exploit -n ppc/simple
exploit -n EnableUnicodeEncoding=                   exploit -n Powershell::no_equals=                   exploit -n SMB::pad_data_level=                     exploit -n sparc/random
exploit -n HandlerSSLCert=                          exploit -n Powershell::noninteractive=              exploit -n SMB::pad_file_level=                     exploit -n tty/generic
exploit -n InitialAutoRunScript=                    exploit -n Powershell::persist=                     exploit -n SMB::pipe_evasion=                       exploit -n x64/simple
exploit -n KrbCacheMode=                            exploit -n Powershell::prepend_protections_bypass=  exploit -n SMB::pipe_read_max_size=                 exploit -n x86/opty2
exploit -n LEAKATTEMPTS=                            exploit -n Powershell::prepend_sleep=               exploit -n SMB::pipe_read_min_size=                 exploit -n x86/single_byte
exploit -n LHOST=                                   exploit -n Powershell::remove_comspec=              exploit -n SMB::pipe_write_max_size=
exploit -n LPORT=                                   exploit -n Powershell::strip_comments=              exploit -n SMB::pipe_write_min_size=
exploit -n LogLevel=                                exploit -n Powershell::strip_whitespace=            exploit -n SMBDirect=

`exploit -p`

msf6 exploit(windows/smb/ms17_010_psexec) > exploit -p
Display all 1542 possibilities? (y or n)
exploit -p ALLOW_GUEST=                                                       exploit -p cmd/windows/powershell/shell/bind_hidden_tcp
exploit -p AutoLoadStdapi=                                                    exploit -p cmd/windows/powershell/shell/bind_ipv6_tcp
exploit -p AutoRunScript=                                                     exploit -p cmd/windows/powershell/shell/bind_ipv6_tcp_uuid
exploit -p AutoSystemInfo=                                                    exploit -p cmd/windows/powershell/shell/bind_named_pipe
exploit -p AutoUnhookProcess=                                                 exploit -p cmd/windows/powershell/shell/bind_nonx_tcp
exploit -p AutoVerifySessionTimeout=                                          exploit -p cmd/windows/powershell/shell/bind_tcp
exploit -p CHOST=                                                             exploit -p cmd/windows/powershell/shell/bind_tcp_rc4
exploit -p CMD::DELAY=                                                        exploit -p cmd/windows/powershell/shell/bind_tcp_uuid
exploit -p CPORT=                                                             exploit -p cmd/windows/powershell/shell/find_tag
exploit -p CheckModule=                                                       exploit -p cmd/windows/powershell/shell/reverse_ipv6_tcp
exploit -p ConnectTimeout=                                                    exploit -p cmd/windows/powershell/shell/reverse_nonx_tcp
exploit -p ConsoleLogging=                                                    exploit -p cmd/windows/powershell/shell/reverse_ord_tcp
exploit -p ContextInformationFile=                                            exploit -p cmd/windows/powershell/shell/reverse_tcp
exploit -p DBGTRACE=                                                          exploit -p cmd/windows/powershell/shell/reverse_tcp_allports
exploit -p DCERPC::ReadTimeout=                                               exploit -p cmd/windows/powershell/shell/reverse_tcp_dns
exploit -p DCERPC::fake_bind_multi=                                           exploit -p cmd/windows/powershell/shell/reverse_tcp_rc4
exploit -p DCERPC::fake_bind_multi_append=                                    exploit -p cmd/windows/powershell/shell/reverse_tcp_rc4_dns
exploit -p DCERPC::fake_bind_multi_prepend=                                   exploit -p cmd/windows/powershell/shell/reverse_tcp_uuid
exploit -p DCERPC::max_frag_size=                                             exploit -p cmd/windows/powershell/shell/reverse_udp
exploit -p DCERPC::smb_pipeio=                                                exploit -p cmd/windows/powershell/shell_bind_tcp
exploit -p DOMAIN=                                                            exploit -p cmd/windows/powershell/shell_bind_tcp_xpfw
exploit -p DisablePayloadHandler=                                             exploit -p cmd/windows/powershell/shell_hidden_bind_tcp
exploit -p DomainControllerRhost=                                             exploit -p cmd/windows/powershell/shell_reverse_tcp
exploit -p ENCODER=                                                           exploit -p cmd/windows/powershell/speak_pwned
exploit -p EXE::Custom=                                                       exploit -p cmd/windows/powershell/upexec/bind_hidden_ipknock_tcp
exploit -p EXE::EICAR=                                                        exploit -p cmd/windows/powershell/upexec/bind_hidden_tcp
exploit -p EXE::FallBack=                                                     exploit -p cmd/windows/powershell/upexec/bind_ipv6_tcp
exploit -p EXE::Inject=                                                       exploit -p cmd/windows/powershell/upexec/bind_ipv6_tcp_uuid
exploit -p EXE::OldMethod=                                                    exploit -p cmd/windows/powershell/upexec/bind_named_pipe

`exploit -e`

msf6 exploit(windows/smb/ms17_010_psexec) > exploit -e
Display all 200 possibilities? (y or n)
exploit -e ALLOW_GUEST=                             exploit -e MinimumRank=                             exploit -e Rhostname=                               exploit -e USERNAME=
exploit -e AutoLoadStdapi=                          exploit -e NAMEDPIPE=                               exploit -e SERVICE_DESCRIPTION=                     exploit -e VERBOSE=
exploit -e AutoRunScript=                           exploit -e NAMED_PIPES=                             exploit -e SERVICE_DISPLAY_NAME=                    exploit -e WORKSPACE=
exploit -e AutoSystemInfo=                          exploit -e NOP=                                     exploit -e SERVICE_FILENAME=                        exploit -e WfsDelay=
exploit -e AutoUnhookProcess=                       exploit -e NTLM::SendLM=                            exploit -e SERVICE_NAME=                            exploit -e cmd/brace
exploit -e AutoVerifySessionTimeout=                exploit -e NTLM::SendNTLM=                          exploit -e SERVICE_PERSIST=                         exploit -e cmd/echo
exploit -e CHOST=                                   exploit -e NTLM::SendSPN=                           exploit -e SERVICE_STUB_ENCODER=                    exploit -e cmd/generic_sh
exploit -e CMD::DELAY=                              exploit -e NTLM::UseLMKey=                          exploit -e SHARE=                                   exploit -e cmd/ifs
exploit -e CPORT=                                   exploit -e NTLM::UseNTLM2_session=                  exploit -e SMB::AlwaysEncrypt=                      exploit -e cmd/perl
exploit -e CheckModule=                             exploit -e NTLM::UseNTLMv2=                         exploit -e SMB::Auth=                               exploit -e cmd/powershell_base64
exploit -e ConnectTimeout=                          exploit -e PASSWORD=                                exploit -e SMB::ChunkSize=                          exploit -e cmd/printf_php_mq
exploit -e ConsoleLogging=                          exploit -e PAYLOAD=                                 exploit -e SMB::Krb5Ccname=                         exploit -e generic/eicar
exploit -e ContextInformationFile=                  exploit -e PSH_PATH=                                exploit -e SMB::KrbOfferedEncryptionTypes=          exploit -e generic/none
exploit -e DBGTRACE=                                exploit -e PayloadBindPort=                         exploit -e SMB::Native_LM=                          exploit -e mipsbe/byte_xori
exploit -e DCERPC::ReadTimeout=                     exploit -e PayloadProcessCommandLine=               exploit -e SMB::Native_OS=                          exploit -e mipsbe/longxor
exploit -e DCERPC::fake_bind_multi=                 exploit -e PayloadUUIDName=                         exploit -e SMB::Rhostname=                          exploit -e mipsle/byte_xori
exploit -e DCERPC::fake_bind_multi_append=          exploit -e PayloadUUIDRaw=                          exploit -e SMB::VerifySignature=                    exploit -e mipsle/longxor
exploit -e DCERPC::fake_bind_multi_prepend=         exploit -e PayloadUUIDSeed=                         exploit -e SMB::obscure_trans_pipe_level=           exploit -e php/base64
exploit -e DCERPC::max_frag_size=                   exploit -e PayloadUUIDTracking=                     exploit -e SMB::pad_data_level=                     exploit -e ppc/longxor
exploit -e DCERPC::smb_pipeio=                      exploit -e PingbackRetries=                         exploit -e SMB::pad_file_level=                     exploit -e ppc/longxor_tag

`generate -e`

msf6 payload(windows/x64/meterpreter_reverse_tcp) > generate -e
generate -e AutoLoadStdapi=               generate -e PayloadUUIDTracking=          generate -e cmd/brace                     generate -e x86/alpha_upper
generate -e AutoRunScript=                generate -e PingbackRetries=              generate -e cmd/echo                      generate -e x86/avoid_underscore_tolower
generate -e AutoSystemInfo=               generate -e PingbackSleep=                generate -e cmd/generic_sh                generate -e x86/avoid_utf8_tolower
generate -e AutoUnhookProcess=            generate -e PrependMigrate=               generate -e cmd/ifs                       generate -e x86/bloxor
generate -e AutoVerifySessionTimeout=     generate -e PrependMigrateProc=           generate -e cmd/perl                      generate -e x86/bmp_polyglot
generate -e ConsoleLogging=               generate -e Prompt=                       generate -e cmd/powershell_base64         generate -e x86/call4_dword_xor
generate -e ENCODER=                      generate -e PromptChar=                   generate -e cmd/printf_php_mq             generate -e x86/context_cpuid
generate -e EXITFUNC=                     generate -e PromptTimeFormat=             generate -e generic/eicar                 generate -e x86/context_stat
generate -e EXTENSIONS=                   generate -e ReverseAllowProxy=            generate -e generic/none                  generate -e x86/context_time
generate -e EXTINIT=                      generate -e ReverseListenerBindAddress=   generate -e mipsbe/byte_xori              generate -e x86/countdown
generate -e EnableUnicodeEncoding=        generate -e ReverseListenerBindPort=      generate -e mipsbe/longxor                generate -e x86/fnstenv_mov
generate -e HandlerSSLCert=               generate -e ReverseListenerComm=          generate -e mipsle/byte_xori              generate -e x86/jmp_call_additive
generate -e InitialAutoRunScript=         generate -e ReverseListenerThreaded=      generate -e mipsle/longxor                generate -e x86/nonalpha
generate -e LHOST=                        generate -e SessionCommunicationTimeout=  generate -e php/base64                    generate -e x86/nonupper
generate -e LPORT=                        generate -e SessionExpirationTimeout=     generate -e ppc/longxor                   generate -e x86/opt_sub
generate -e LogLevel=                     generate -e SessionLogging=               generate -e ppc/longxor_tag               generate -e x86/service
generate -e MeterpreterDebugBuild=        generate -e SessionRetryTotal=            generate -e ruby/base64                   generate -e x86/shikata_ga_nai
generate -e MeterpreterDebugLogging=      generate -e SessionRetryWait=             generate -e sparc/longxor_tag             generate -e x86/single_static_bit
generate -e MeterpreterPrompt=            generate -e SessionTlvLogging=            generate -e x64/xor                       generate -e x86/unicode_mixed
generate -e MinimumRank=                  generate -e StagerRetryCount=             generate -e x64/xor_context               generate -e x86/unicode_upper
generate -e PayloadProcessCommandLine=    generate -e StagerRetryWait=              generate -e x64/xor_dynamic               generate -e x86/xor_dynamic
generate -e PayloadUUIDName=              generate -e TimestampOutput=              generate -e x64/zutto_dekiru              generate -e x86/xor_poly
generate -e PayloadUUIDRaw=               generate -e VERBOSE=                      generate -e x86/add_sub
generate -e PayloadUUIDSeed=              generate -e WORKSPACE=                    generate -e x86/alpha_mixed

`run -p`

msf6 evasion(windows/applocker_evasion_install_util) > run -p
Display all 1404 possibilities? (y or n)
run -p ConsoleLogging=                                                    run -p cmd/windows/powershell/x64/custom/bind_named_pipe
run -p ENCODER=                                                           run -p cmd/windows/powershell/x64/custom/bind_tcp
run -p FILENAME=                                                          run -p cmd/windows/powershell/x64/custom/bind_tcp_rc4
run -p LogLevel=                                                          run -p cmd/windows/powershell/x64/custom/bind_tcp_uuid
run -p MeterpreterPrompt=                                                 run -p cmd/windows/powershell/x64/custom/reverse_http
run -p MinimumRank=                                                       run -p cmd/windows/powershell/x64/custom/reverse_https
run -p PAYLOAD=                                                           run -p cmd/windows/powershell/x64/custom/reverse_named_pipe
run -p Prompt=                                                            run -p cmd/windows/powershell/x64/custom/reverse_tcp
run -p PromptChar=                                                        run -p cmd/windows/powershell/x64/custom/reverse_tcp_rc4
run -p PromptTimeFormat=                                                  run -p cmd/windows/powershell/x64/custom/reverse_tcp_uuid
run -p SessionLogging=                                                    run -p cmd/windows/powershell/x64/custom/reverse_winhttp
run -p SessionTlvLogging=                                                 run -p cmd/windows/powershell/x64/custom/reverse_winhttps
run -p TARGET=                                                            run -p cmd/windows/powershell/x64/encrypted_shell/reverse_tcp
run -p TimestampOutput=                                                   run -p cmd/windows/powershell/x64/exec
run -p VERBOSE=                                                           run -p cmd/windows/powershell/x64/loadlibrary
run -p WORKSPACE=                                                         run -p cmd/windows/powershell/x64/messagebox

`run -n`

msf6 evasion(windows/applocker_evasion_install_util) > run -n
run -n ConsoleLogging=     run -n MeterpreterPrompt=  run -n PromptChar=         run -n TARGET=             run -n aarch64/simple      run -n php/generic         run -n x64/simple
run -n ENCODER=            run -n MinimumRank=        run -n PromptTimeFormat=   run -n TimestampOutput=    run -n armle/simple        run -n ppc/simple          run -n x86/opty2
run -n FILENAME=           run -n PAYLOAD=            run -n SessionLogging=     run -n VERBOSE=            run -n cmd/generic         run -n sparc/random        run -n x86/single_byte
run -n LogLevel=           run -n Prompt=             run -n SessionTlvLogging=  run -n WORKSPACE=          run -n mipsbe/better       run -n tty/generic

`run -e`

msf6 evasion(windows/applocker_evasion_install_util) > run -e
run -e ConsoleLogging=               run -e TimestampOutput=              run -e mipsbe/longxor                run -e x86/alpha_mixed               run -e x86/nonalpha
run -e ENCODER=                      run -e VERBOSE=                      run -e mipsle/byte_xori              run -e x86/alpha_upper               run -e x86/nonupper
run -e FILENAME=                     run -e WORKSPACE=                    run -e mipsle/longxor                run -e x86/avoid_underscore_tolower  run -e x86/opt_sub
run -e LogLevel=                     run -e cmd/brace                     run -e php/base64                    run -e x86/avoid_utf8_tolower        run -e x86/service
run -e MeterpreterPrompt=            run -e cmd/echo                      run -e ppc/longxor                   run -e x86/bloxor                    run -e x86/shikata_ga_nai
run -e MinimumRank=                  run -e cmd/generic_sh                run -e ppc/longxor_tag               run -e x86/bmp_polyglot              run -e x86/single_static_bit
run -e PAYLOAD=                      run -e cmd/ifs                       run -e ruby/base64                   run -e x86/call4_dword_xor           run -e x86/unicode_mixed
run -e Prompt=                       run -e cmd/perl                      run -e sparc/longxor_tag             run -e x86/context_cpuid             run -e x86/unicode_upper
run -e PromptChar=                   run -e cmd/powershell_base64         run -e x64/xor                       run -e x86/context_stat              run -e x86/xor_dynamic
run -e PromptTimeFormat=             run -e cmd/printf_php_mq             run -e x64/xor_context               run -e x86/context_time              run -e x86/xor_poly
run -e SessionLogging=               run -e generic/eicar                 run -e x64/xor_dynamic               run -e x86/countdown
run -e SessionTlvLogging=            run -e generic/none                  run -e x64/zutto_dekiru              run -e x86/fnstenv_mov
run -e TARGET=                       run -e mipsbe/byte_xori              run -e x86/add_sub                   run -e x86/jmp_call_additive

RPC

metasploit-framework (46273d5) [$] via  desktop-linux via  v3.0.5 via ⍱ took 5m17s
❯ curl --request POST \
  --url http://127.0.0.1:8081/api/v1/json-rpc \
  --data '{
        "jsonrpc": "2.0",
        "method": "module.nops",
        "id": 1,
        "params": []
}'
{"jsonrpc":"2.0","result":{"modules":["aarch64/simple","armle/simple","cmd/generic","mipsbe/better","php/generic","ppc/simple","sparc/random","tty/generic","x64/simple","x86/opty2","x86/single_byte"]},"id":1}%

lib/msf/core/encoded_payload.rb

lib/msf/core/exploit/remote/browser_autopwn2.rb

lib/msf/core/module/platform.rb

cgranleese-r7 · 2024-01-19T16:22:11Z

lib/msf/core/modules/metadata/cache.rb

+  def module_metadata(type)
+    @mutex.synchronize do
+      wait_for_load
+      # TODO: Should probably figure out a way to cache this


Can this be removed?

cgranleese-r7 · 2024-01-19T16:23:10Z

spec/lib/msf/core/module_set_spec.rb

+        allow(subject).to receive(:create).with(module_refname)
+      end
+
+      # TODO: it's unexpected that `fetch` and `[]` would act this differently


Same as above.

adfoster-r7 · 2024-01-19T21:36:27Z

lib/msf/core/module_set.rb

@@ -304,33 +274,31 @@ def each_module_list(ary, opts, &block)
  # @return [Array<Array<String, Class>>] Array of arrays where the inner array is a pair of the module reference name
  #   and the module class.
  def rank_modules
-    self.sort_by { |pair| module_rank(*pair) }.reverse!
+    module_metadata.sort_by { |refname, _metadata| "#{module_rank(refname)}-#{refname}" }.reverse!


Looks like this is super slow now; it seems like the sort block calls off to module_rank(refname) which then does a lookup on the module metadata - which is now behind a mutex lock and creation of an array:

# This file def rank_modules module_metadata.sort_by { |refname, _metadata| "#{module_rank(refname)}-#{refname}" }.reverse! end def module_rank(reference_name) module_metadata[reference_name].rank || Msf::NormalRanking end # Elsewhere def module_metadata(type) @mutex.synchronize do wait_for_load @module_metadata_cache.filter_map { |_, metadata| [metadata.ref_name, metadata] if metadata.type == type }.to_h end end

I believe this should be faster, but it's worth confirming it's semantically the same:

module_metadata.sort_by { |refname, metadata| [metadata.rank, refname] }.reverse!

Tested with:

require 'benchmark/ips' require 'msfenv' framework = Msf::Simple::Framework.create Benchmark.ips do |x| # Configure the number of seconds used during # the warmup phase (default 2) and calculation phase (default 5) x.config(:time => 15, :warmup => 4) # These parameters can also be configured this way x.time = 5 x.warmup = 2 x.report("rank_modules_string") do framework.modules.payloads.rank_modules_string.length end x.report("rank_modules_array") do framework.modules.payloads.rank_modules_array.length end x.report("rank_modules_array_without_lock") do framework.modules.payloads.rank_modules_array_without_lock.length end x.compare! end

Results:

Warming up -------------------------------------- rank_modules_string 1.000 i/100ms rank_modules_array 1.000 i/100ms rank_modules_array_without_lock 15.000 i/100ms Calculating ------------------------------------- rank_modules_string 0.392 (± 0.0%) i/s - 3.000 in 7.716404s rank_modules_array 0.426 (± 0.0%) i/s - 3.000 in 7.055821s rank_modules_array_without_lock 151.488 (± 5.9%) i/s - 765.000 in 5.068616s Comparison: rank_modules_array_without_lock: 151.5 i/s rank_modules_array: 0.4 i/s - 355.61x slower rank_modules_string: 0.4 i/s - 386.61x slower

Code diff that I tested with:

diff --git a/lib/msf/core/module_set.rb b/lib/msf/core/module_set.rb index 175a32adc4..a6b297a22a 100644 --- a/lib/msf/core/module_set.rb +++ b/lib/msf/core/module_set.rb @@ -196,7 +196,6 @@ class Msf::ModuleSet < Hash module_metadata.keys end - protected # Enumerates the modules in the supplied array with possible limiting factors. # @@ -268,13 +267,25 @@ class Msf::ModuleSet < Hash # @return [String] type of modules attr_writer :module_type + def rank_modules + rank_modules_array_without_lock + end + # Ranks modules based on their constant rank value, if they have one. Modules without a Rank are treated as if they # had {Msf::NormalRanking} for Rank. # # @return [Array<Array<String, Class>>] Array of arrays where the inner array is a pair of the module reference name # and the module class. - def rank_modules - module_metadata.sort_by { |refname, _metadata| module_rank(refname) }.reverse! + def rank_modules_string + module_metadata.sort_by { |refname, _metadata| "#{module_rank(refname)}-#{refname}" }.reverse! + end + + def rank_modules_array + module_metadata.sort_by { |refname, _metadata| [module_rank(refname), refname] }.reverse! + end + + def rank_modules_array_without_lock + module_metadata.sort_by { |refname, metadata| [metadata.rank, refname] }.reverse! end # Retrieves the rank from a loaded, not-yet-loaded, or unloadable Metasploit Module.

adfoster-r7 · 2024-01-19T22:14:23Z

lib/msf/core/modules/metadata/obj.rb

  attr_reader :platform
+  # @return [Msf::Module::PlatformList]
+  attr_reader :platform_list


Suggested change

attr_reader :platform_list

attr_reader :platforms

Edit: I see, module instance has called it platform - but here platform is already defined as a string

Hm. Looks like all possible options are bad 😞

you can also call platforms on a PlatformList so calling it platforms leads to platforms.platforms which I didn't like either 😅

adfoster-r7 · 2024-01-19T22:14:55Z

lib/msf/core/modules/metadata/obj.rb

@@ -28,8 +28,10 @@ class Obj
  attr_reader :description
  # @return [Array<String>]
  attr_reader :references
-  # @return [Boolean]
+  # @return [String]


Not a blocker; Should we make this deprecated? 🤔

adfoster-r7 · 2024-02-02T14:16:35Z

I think this might need a few more tweaks for performance, but it looks like a good stepping stone for now 👍

adfoster-r7 · 2024-02-02T14:17:38Z

Release Notes

Fixes a bug with framework having 0 registered nop modules when the defer-module-loads feature was enabled.

Leverage the module metadata cache in the module_sets

2cf045d

dwelch-r7 added the rn-fix release notes fix label Jan 15, 2024

adfoster-r7 reviewed Jan 16, 2024

View reviewed changes

lib/msf/core/module/platform_list.rb Outdated Show resolved Hide resolved

adfoster-r7 reviewed Jan 16, 2024

View reviewed changes

spec/lib/msf/core/module_set_spec.rb Outdated Show resolved Hide resolved

adfoster-r7 reviewed Jan 16, 2024

View reviewed changes

test/modules/post/test/all.rb Outdated Show resolved Hide resolved

adfoster-r7 reviewed Jan 16, 2024

View reviewed changes

tools/modules/module_ports.rb Outdated Show resolved Hide resolved

dwelch-r7 added 3 commits January 16, 2024 13:31

Fix encoded payloads test

1c4258f

Parse platform list within module metadata object

236f885

Add test run with defer module loads feature enabled

46273d5

cgranleese-r7 self-assigned this Jan 17, 2024

adfoster-r7 reviewed Jan 18, 2024

View reviewed changes

bwatters-r7 reviewed Jan 18, 2024

View reviewed changes

lib/msf/core/encoded_payload.rb Outdated Show resolved Hide resolved

lib/msf/core/exploit/remote/browser_autopwn2.rb Outdated Show resolved Hide resolved

lib/msf/core/module/platform.rb Outdated Show resolved Hide resolved

shuffle platform parsing and code quality

391bc4e

cgranleese-r7 reviewed Jan 19, 2024

View reviewed changes

dwelch-r7 added 2 commits January 19, 2024 17:50

rank modules consistently

334a469

Add test to ensure module ranking is consistent

763a97d

adfoster-r7 reviewed Jan 19, 2024

View reviewed changes

Improve module ranking performance

82e9c27

adfoster-r7 merged commit 48221e5 into rapid7:master Feb 2, 2024
58 checks passed

This was referenced Feb 7, 2024

Remove all references to Msf::SymbolicModule #18803

Merged

Revert change to each_module for compatible nops #18828

Merged

cgranleese-r7 mentioned this pull request Mar 25, 2024

Fixes WMAP plugin loading failures #19006

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Leverage the module metadata cache in the module_sets #18704

Leverage the module metadata cache in the module_sets #18704

dwelch-r7 commented Jan 15, 2024 •

edited by cgranleese-r7

adfoster-r7 commented Jan 16, 2024 •

edited

adfoster-r7 commented Jan 16, 2024

adfoster-r7 Jan 18, 2024

dwelch-r7 Jan 18, 2024

adfoster-r7 Jan 18, 2024

dwelch-r7 Jan 18, 2024

adfoster-r7 Jan 19, 2024 •

edited

dwelch-r7 Jan 19, 2024

cgranleese-r7 commented Jan 18, 2024

cgranleese-r7 Jan 19, 2024

cgranleese-r7 Jan 19, 2024

adfoster-r7 Jan 19, 2024

adfoster-r7 Jan 19, 2024 •

edited

dwelch-r7 Jan 22, 2024

adfoster-r7 Jan 19, 2024

adfoster-r7 commented Feb 2, 2024

adfoster-r7 commented Feb 2, 2024

Leverage the module metadata cache in the module_sets #18704

Leverage the module metadata cache in the module_sets #18704

Conversation

dwelch-r7 commented Jan 15, 2024 • edited by cgranleese-r7

Verification steps

adfoster-r7 commented Jan 16, 2024 • edited

adfoster-r7 commented Jan 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adfoster-r7 Jan 19, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgranleese-r7 commented Jan 18, 2024

Before

After

Auto completion

handler -e

handler -p

exploit -n

exploit -p

exploit -e

generate -e

run -p

run -n

run -e

RPC

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adfoster-r7 Jan 19, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adfoster-r7 commented Feb 2, 2024

adfoster-r7 commented Feb 2, 2024

Release Notes

dwelch-r7 commented Jan 15, 2024 •

edited by cgranleese-r7

adfoster-r7 commented Jan 16, 2024 •

edited

adfoster-r7 Jan 19, 2024 •

edited

`handler -e`

`handler -p`

`exploit -n`

`exploit -p`

`exploit -e`

`generate -e`

`run -p`

`run -n`

`run -e`

adfoster-r7 Jan 19, 2024 •

edited