Fix external module encoding

[sjanusz-r7]

Fixes #15577

This PR properly handles encoding of UTF-8 characters.

It also makes a small correction to one of the relevant Python modules which was necessary to run the module.

Before

[-] Auxiliary failed: Encoding::UndefinedConversionError "\xE4" from ASCII-8BIT to UTF-8
[-] Call stack:
[-]   /opt/metasploit-framework/embedded/lib/ruby/gems/2.7.0/gems/json-2.5.1/lib/json/common.rb:312:in `generate'
[-]   /opt/metasploit-framework/embedded/lib/ruby/gems/2.7.0/gems/json-2.5.1/lib/json/common.rb:312:in `generate'
[-]   /opt/metasploit-framework/embedded/framework/lib/msf/core/modules/external/message.rb:37:in `to_json'
[-]   /opt/metasploit-framework/embedded/framework/lib/msf/core/modules/external/bridge.rb:70:in `send'
[-]   /opt/metasploit-framework/embedded/framework/lib/msf/core/modules/external/bridge.rb:18:in `exec'
[-]   /opt/metasploit-framework/embedded/framework/lib/msf/core/modules/external.rb:25:in `exec'
[-]   /opt/metasploit-framework/embedded/framework/lib/msf/core/module/external.rb:8:in `execute_module'

After

msf6 auxiliary(gather/python_gather) > run

[*] Starting server...
[*] 漢字
[*] Auxiliary module execution completed

In this example the UTF-8 encoded characters are simply output to the terminal for testing purposes.

Verification

• Follow the steps from Auxiliary failed: Encoding::UndefinedConversionError "\xE4" from ASCII-8BIT to UTF-8 #15577

[adfoster-r7]

We traced down the original use of this method to #10002 which allows datastore values to be a nested array, which I'm not sure is something we'd want

[gwillcox-r7]

Testing this now and will land this if all checks pass

[gwillcox-r7]

Before Patch:

 ~/git/metasploit-framework/m/a/gather │ master ?22  nano test.py                                                 ✔ │ 2.7.2 Ruby 
 ~/git/metasploit-framework/m/a/gather │ master ?23  ./msfconsole                                            ✔ │ 5s │ 2.7.2 Ruby 
zsh: no such file or directory: ./msfconsole
 ~/git/metasploit-framework/m/a/gather │ master ?23  chmod +x test.py                                         127 х │ 2.7.2 Ruby 
 ~/git/metasploit-framework/m/a/gather │ master ?23  ls                                                           ✔ │ 2.7.2 Ruby 
advantech_webaccess_creds.rb                f5_bigip_cookie_disclosure.rb           oats_downloadservlet_traversal.rb
alienvault_iso27001_sqli.rb                 firefox_pdfjs_file_theft.rb             office365userenum.py
alienvault_newpolicyform_sqli.rb            flash_rosetta_jsonp_url_disclosure.rb   opennms_xxe.rb
android_browser_file_theft.rb               fortios_vpnssl_traversal_creds_leak.rb  peplink_bauth_sqli.rb
android_browser_new_tab_cookie_theft.rb     get_user_spns.py                        pimcore_creds_sqli.rb
android_htmlfileprovider.rb                 hp_enum_perfd.rb                        pulse_secure_file_disclosure.rb
android_object_tag_webview_uxss.rb          hp_snac_domain_creds.rb                 qnap_backtrace_admin_hash.rb
android_stock_browser_uxss.rb               http_pdf_authors.rb                     qnap_lfi.rb
apache_rave_creds.rb                        huawei_wifi_info.rb                     rails_doubletap_file_read.rb
apple_safari_ftp_url_cookie_theft.rb        ibm_bigfix_sites_packages_enum.rb       redis_extractor.rb
apple_safari_webarchive_uxss.rb             ibm_sametime_enumerate_users.rb         safari_file_url_navigation.rb
asterisk_creds.rb                           ibm_sametime_room_brute.rb              saltstack_salt_root_key.rb
avtech744_dvr_accounts.rb                   ibm_sametime_version.rb                 samsung_browser_sop_bypass.rb
browser_info.rb                             ie_sandbox_findfiles.rb                 search_email_collector.rb
browser_lanipleak.rb                        ie_uxss_injection.rb                    searchengine_subdomains_collector.rb
c2s_dvr_password_disclosure.rb              impersonate_ssl.rb                      shodan_honeyscore.rb
censys_search.rb                            ipcamera_password_disclosure.rb         shodan_host.rb
cerberus_helpdesk_hash_disclosure.rb        java_rmi_registry.rb                    shodan_search.rb
checkpoint_hostname.rb                      jenkins_cred_recovery.rb                snare_registry.rb
chrome_debugger.rb                          joomla_com_realestatemanager_sqli.rb    solarwinds_orion_sqli.rb
cisco_rv320_config.rb                       joomla_contenthistory_sqli.rb           ssllabs_scan.rb
citrix_published_applications.rb            joomla_weblinks_sqli.rb                 teamtalk_creds.rb
citrix_published_bruteforce.rb              kerberos_enumusers.rb                   test.py
cloud_lookup.rb                             konica_minolta_pwd_extract.rb           trackit_sql_domain_creds.rb
coldfusion_pwd_props.rb                     lansweeper_collector.rb                 vbulletin_getindexablecontent_sqli.rb
corpwatch_lookup_id.rb                      ldap_hashdump.rb                        vbulletin_vote_sqli.rb
corpwatch_lookup_name.rb                    mantisbt_admin_sqli.rb                  vmware_vcenter_vmdir_ldap.rb
cve_2021_27850_apache_tapestry_hmac_key.rb  mcafee_epo_xxe.rb                       windows_deployment_services_shares.rb
d20pass.rb                                  memcached_extractor.rb                  windows_secrets_dump.rb
darkcomet_filedownloader.rb                 mikrotik_winbox_fileread.py             wp_all_in_one_migration_export.rb
dolibarr_creds_sqli.rb                      mongodb_js_inject_collection_enum.rb    wp_ultimate_csv_importer_user_extract.rb
doliwamp_traversal_creds.rb                 ms14_052_xmldom.rb                      wp_w3_total_cache_hash_extract.rb
drupal_openid_xxe.rb                        mybb_db_fingerprint.rb                  xbmc_traversal.rb
eaton_nsm_creds.rb                          natpmp_external_address.rb              xerox_pwd_extract.rb
emc_cta_xxe.rb                              netgear_password_disclosure.rb          xerox_workcentre_5xxx_ldap.rb
enum_dns.rb                                 nis_bootparamd_domain.rb                xymon_info.rb
eventlog_cred_disclosure.rb                 nis_ypserv_map.rb                       zabbix_toggleids_sqli.rb
exchange_proxylogon_collector.rb            nuuo_cms_bruteforce.rb                  zookeeper_info_disclosure.rb
external_ip.rb                              nuuo_cms_file_download.rb               zoomeye_search.rb
 ~/git/metasploit-framework/m/a/gather │ master ?23  ./msfconsole                                                 ✔ │ 2.7.2 Ruby 
zsh: no such file or directory: ./msfconsole
 ~/git/metasploit-framework/m/a/gather │ master ?23  cd ..                                                    127 х │ 2.7.2 Ruby 
cd%                                                                                                                                ~/git/metasploit-framework/m/auxiliary │ master ?23  cd ..                                                       ✔ │ 2.7.2 Ruby 
c%                                                                                                                                 ~/git/metasploit-framework/modules │ master ?23  cd ..                                                           ✔ │ 2.7.2 Ruby 
 ~/git/metasploit-framework │ master ?23  ./msfconsole                                                            ✔ │ 2.7.2 Ruby 
[!] The following modules could not be loaded!..\
[!] 	/home/gwillcox/git/metasploit-framework/modules/auxiliary/gather/office365userenum.py
[!] Please see /home/gwillcox/.msf4/logs/framework.log for details.
                                                  
                                   ____________
 [%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%| $a,        |%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%]
 [%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%| $S`?a,     |%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%]
 [%%%%%%%%%%%%%%%%%%%%__%%%%%%%%%%|       `?a, |%%%%%%%%__%%%%%%%%%__%%__ %%%%]
 [% .--------..-----.|  |_ .---.-.|       .,a$%|.-----.|  |.-----.|__||  |_ %%]
 [% |        ||  -__||   _||  _  ||  ,,aS$""`  ||  _  ||  ||  _  ||  ||   _|%%]
 [% |__|__|__||_____||____||___._||%$P"`       ||   __||__||_____||__||____|%%]
 [%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%| `"a,       ||__|%%%%%%%%%%%%%%%%%%%%%%%%%%]
 [%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%|____`"a,$$__|%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%]
 [%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%        `"$   %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%]
 [%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%]


       =[ metasploit v6.1.3-dev-6ca7e00b63                ]
+ -- --=[ 2161 exploits - 1147 auxiliary - 367 post       ]
+ -- --=[ 592 payloads - 45 encoders - 10 nops            ]
+ -- --=[ 8 evasion                                       ]

Metasploit tip: Metasploit can be configured at startup, see 
msfconsole --help to learn more

msf6 > use auxiliary/gather/test 
msf6 auxiliary(gather/test) > show options

Module options (auxiliary/gather/test):

   Name   Current Setting  Required  Description
   ----   ---------------  --------  -----------
   query                   no        search something

msf6 auxiliary(gather/test) > set query 汉字/漢字
query => 汉字/漢字
msf6 auxiliary(gather/test) > run

[*] Starting server...
[-] Auxiliary failed: Encoding::UndefinedConversionError "\xE6" from ASCII-8BIT to UTF-8
[-] Call stack:
[-]   /home/gwillcox/.rvm/gems/ruby-2.7.2@metasploit-framework/gems/json-2.5.1/lib/json/common.rb:312:in `generate'
[-]   /home/gwillcox/.rvm/gems/ruby-2.7.2@metasploit-framework/gems/json-2.5.1/lib/json/common.rb:312:in `generate'
[-]   /home/gwillcox/git/metasploit-framework/lib/msf/core/modules/external/message.rb:37:in `to_json'
[-]   /home/gwillcox/git/metasploit-framework/lib/msf/core/modules/external/bridge.rb:70:in `send'
[-]   /home/gwillcox/git/metasploit-framework/lib/msf/core/modules/external/bridge.rb:18:in `exec'
[-]   /home/gwillcox/git/metasploit-framework/lib/msf/core/modules/external.rb:25:in `exec'
[-]   /home/gwillcox/git/metasploit-framework/lib/msf/core/module/external.rb:8:in `execute_module'
[-]   /home/gwillcox/git/metasploit-framework/modules/auxiliary/gather/test.py:34:in `run'
[*] Auxiliary module execution completed
msf6 auxiliary(gather/test) > 

[gwillcox-r7]

After patch:

msf6 > use auxiliary/gather/test
msf6 auxiliary(gather/test) > set query 汉字/漢字
query => 汉字/漢字
msf6 auxiliary(gather/test) > run

[*] Starting server...
[*] 汉字/漢字
[*] Auxiliary module execution completed
msf6 auxiliary(gather/test) > 

Will land this now

[gwillcox-r7]

Release Notes

A bug has been fixed that prevented external modules from properly handling the encoding of UTF-8 characters.