add. new gather module cloud_lookup

[mekhalleh]

This module replace the previously pulls:

• add. new auxiliary module: behind_cloudflare #10481 (behind_cloudflare)
• add. new auxiliary module: behind_ingensec #10486 (behind_ingensec)
• add. new auxiliary module: behind_cloudfront #11204 (behind_cloudfront)

This module can be useful if you need to test the security of your server and your
website behind a solution Cloud based. By discovering the origin IP address of the
targeted host.

More precisely, I use multiple data sources (in order ViewDNS.info, DNS enumeration and Censys)
to collect assigned (or have been assigned) IP addresses from the targeted site or domain
that uses the following:
Amazon Cloudflare, Amazon CloudFront, ArvanCloud, Envoy Proxy, Fastly, Stackpath Fireblade,
Stackpath MaxCDN, Imperva Incapsula, InGen Security (BinarySec EasyWAF), KeyCDN, Netlify
and Sucuri.

Verification Steps

1. Install the module as usual
2. Start msfconsole
3. Do: use auxiliary/gather/cloud_lookup
4. Do: set hostname www.zataz.com
5. Do: run

Options

CENSYS_SECRET

Your Censys API SECRET.

CENSYS_UID

Your Censys API UID.

COMPSTR

You can use a custom string to perform the comparison.

HOSTNAME

This is the hostname [fqdn] on which the website responds. But this can also be a domain.

msf5 auxiliary(gather/cloud_lookup) > set hostname www.zataz.com
--or--
msf5 auxiliary(gather/cloud_lookup) > set hostname discordapp.com

Proxies

A proxy chain of format type:host:port[,type:host:port][...]. It's optional.

RPORT

The target TCP port on which the protected website responds. Default: 443

SSL

Negotiate SSL/TLS for outgoing connections. Default: true

THREADS

Number of concurent threads needed for DNS enumeration. Default: 8

URIPATH

The URI path on which to perform the page comparison. Default: '/'

WORDLIST

Name list required for DNS enumeration. Default: ~/metasploit-framework/data/wordlists/namelist.txt

Advanced options

DNSENUM

Set DNS enumeration as optional. Default: true

NS

Specify the nameserver to use for queries. Default: is system DNS

REPORT_LEAKS

Set to write leaked ip addresses in notes. Default: false

USERAGENT

Specify a personalized User-Agent header in HTTP requests. Default: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0

TAG

Specify the HTML tag in which you want to find the fingerprint. Default: title
Useful when combined with the CMPSTR option.

TIMEOUT

HTTP(s) request timeout. Default: 5

VERBOSE

You can also enable the verbose mode to have more information displayed in the console.

Scenarios

For auditing purpose

If successful, you must be able to obtain the IP(s) address of the website as follows:

msf5 auxiliary(gather/cloud_lookup) > set verbose true
verbose => true
msf5 auxiliary(gather/cloud_lookup) > run

[*] Selected action: Amazon CloudFlare
[*] Passive gathering information...
[*]  * ViewDNS.info: 17 IP address found(s).
[*]  * DNS Enumeration: 6 IP address found(s).
[*] Clean Amazon CloudFlare server(s)...
[*]  * TOTAL: 10 IP address found(s) after cleaning.
[*]
[*] Bypass Automatic is in progress...
[*]  * Initial request to the original server for <title> comparison
[*]  * Trying: http://XXX.XXX.XXX.XXX:80/
[+] A direct-connect IP address was found: http://XXX.XXX.XXX.XXX:80/
[*]  * Trying: https://XXX.XXX.XXX.XXX:443/
    --> responded with an unhandled HTTP status code: 504
[*]  * Trying: http://XXX.XXX.XXX.XXX:80/
[*]  * Trying: https://XXX.XXX.XXX.XXX:443/
[*]  * Trying: http://XXX.XXX.XXX.XXX:80/
[+] A direct-connect IP address was found: http://XXX.XXX.XXX.XXX:80/
[*]  * Trying: https://XXX.XXX.XXX.XXX:443/
    --> responded with an unhandled HTTP status code: 504
[*]  * Trying: http://XXX.XXX.XXX.XXX:80/
[+] A direct-connect IP address was found: http://XXX.XXX.XXX.XXX:80/
[*]  * Trying: https://XXX.XXX.XXX.XXX:443/
    --> responded with an unhandled HTTP status code: 403
[*] Auxiliary module execution completed

In this case 'A direct-connect IP address was found' is reported.

However, some disreputable administrators used a simple redircetion (301 and 302)
to force the passage through the WAF. This makes the IP address leak in the 'location'
parameter of the HTTP header.

For exemple:

msf5 auxiliary(gather/cloud_lookup) > set hostname www.exodata.fr
hostname => www.exodata.fr
msf5 auxiliary(gather/cloud_lookup) > run

[*] Selected action: Amazon CloudFlare
[*] Passive gathering information...
[*]  * ViewDNS.info: 3 IP address found(s).
[*]  * DNS Enumeration: 12 IP address found(s).
[*] Clean Amazon CloudFlare server(s)...
[*]  * TOTAL: 4 IP address found(s) after cleaning.
[*]
[*] Bypass Automatic is in progress...
[*]  * Initial request to the original server for <title> comparison
[*]  * Trying: http://41.213.135.13:80/
[*]  * Trying: https://41.213.135.13:443/
      --> responded with HTTP status code: 302 to http://www.exodata.fr/
[!] A leaked IP address was found: https://41.213.135.13:443/
[*]  * Trying: http://185.161.8.26:80/
      --> responded with HTTP status code: 302 to https://www.exodata.fr/
[!] A leaked IP address was found: http://185.161.8.26:80/
[*]  * Trying: https://185.161.8.26:443/
[-] No direct-connect IP address found :-(
[*] Auxiliary module execution completed

--or--

msf5 auxiliary(gather/cloud_lookup) > set verbose false
verbose => false
msf5 auxiliary(gather/cloud_lookup) > set hostname www.ingensecurity.com
hostname => www.ingensecurity.com
msf5 auxiliary(gather/cloud_lookup) > run

[*] Passive gathering information...
[*]  * ViewDNS.info: 2 IP address found(s).
[*]  * DNS Enumeration: 8 IP address found(s).
[*] Clean InGen Security (BinarySec EasyWAF) server(s)...
[*]  * TOTAL: 4 IP address found(s) after cleaning.
[*]
[*] Bypass Automatic is in progress...
[*]  * Initial request to the original server for <title> comparison
[!] A leaked IP address was found: http://188.165.33.235:80/
[-] No direct-connect IP address found :-(
[*] Auxiliary module execution completed

In this case 'A leaked IP address was found' is displayed but the bypass is NOT effective.

You can also use the 'REPORT_LEAKS' option for write that in the notes.

For some reason you may need to change the URI path to interoperate with other than the index page.
To do this specific thing.

For example:

msf5 > use auxiliary/gather/cloud_lookup
msf5 auxiliary(gather/cloud_lookup) > set HOSTNAME www.zataz.com
hostname => www.zataz.com
msf5 auxiliary(gather/cloud_lookup) > set URIPATH /contacter/
uripath => /contacter/
msf5 auxiliary(gather/cloud_lookup) > set compstr Contacter ZATAZ
compstr => Contacter ZATAZ
msf5 auxiliary(gather/cloud_lookup) > run
...

--or--

msf5 > use auxiliary/gather/cloud_lookup
msf5 auxiliary(gather/cloud_lookup) > set HOSTNAME www.zataz.com
hostname => www.zataz.com
msf5 auxiliary(gather/cloud_lookup) > set URIPATH /contacter/
uripath => /contacter/
msf5 auxiliary(gather/cloud_lookup) > set compstr Contacter ZATAZ
compstr => Contacter ZATAZ
msf5 auxiliary(gather/cloud_lookup) > set tag html
tag => html
msf5 auxiliary(gather/cloud_lookup) > run
...

References

1. https://citadelo.com/en/blog/cloudflare-how-to-do-it-right-and-do-not-reveal-your-real-ip/

[mekhalleh]

@msjenkins-r7 test this please.

[space-r7]

It looks like Travis is failing due to some EOL spaces. You can run dev/tools/msftidy.rb on your module to check for any issues that may arise with Travis.

modules/auxiliary/gather/cloud_lookup.rb:322 - [WARNING] Spaces at EOL
modules/auxiliary/gather/cloud_lookup.rb:502 - [WARNING] Spaces at EOL
modules/auxiliary/gather/cloud_lookup.rb:509 - [WARNING] Spaces at EOL

[jmartin-tech]

Minor spelling comment.

[smcintyre-r7]

Hey thanks for this contribution! I went through and reviewed it just now and left quite a bit of feedback. Many of the suggestions I left use some Ruby features to condense redundant logic (and by extension reduce the lines of code) down.

A couple of reoccurring themes:

• The grammar around reporting IP addresses should use "IP address(es) found" never "IP address found(s)".
• You should prefer the == and != operators instead. You don't really need to use the .eql? method unless you're calling it on a hash. Additionally, simply if some_value is preferred from a syntax perspective vs unless some_value == false or if some_value != true and other variations. There was quite a few of these that should really be updated.

Lastly, if you could, would you please run rubocop on your module. It'll help you out by fixing alot of smaller syntax issues for you automatically. Just make sure to use our latest rubocop configuration .rubocop.yml (specify it with the -c argument).

Thank you again for your submission and let me know if you have any questions!

[smcintyre-r7]

It looks like you just about addressed all of the comments, thank you! Were you still planning on moving the DNS enumeration functions into a mixin? It looks like there's quite a bit of overlap with the enum_dns module that could be reduced. I would probably drop it in a new library in lib/msf/core/auxiliary/dns.rb then include Msf::Auxiliary::Dns in this and the original enum_dns modules. You already prefixed the methods with dns_ which is good.

[mekhalleh]

@smcintyre-r7 thanks for your remarks.

About moving the dns function in a mixin, if you want to move the DNS functions to avoid overlapping with enum_dns, yes would be great :) you have better control of the guideline than me.

I observed the query_async function in lib/msf/core/exploit/dns/client.rb which can be used for enumeration with thread, but I did not understand how I could work with it. If I had an example ;)

[smcintyre-r7]

To move the functions to a mixin, you need to:

1. Create a new file at lib/msf/core/auxiliary/dns.rb, setup the module structure (look at other examples in that directory)
2. Add your new entry to lib/msf/core/auxiliary/mixins.rb
3. For both this module and the original enum_dns module:
   1. Add include Msf::Auxiliary::Dns to the top of the module
   2. Start to move functions from the module into the new file that are the same between each module. Looks like there's about 5 functions:
      • dns_enumeration
      • dns_get_a
      • dns_get_mx
      • dns_query
      • save_note (This one could have the differences made into parameters or just leave it out)
   3. Create an initialize method that registers the datastore options used by the shared functions (listed above). Looks like that would just be the following two options:
      • OptAddress.new('NS', [false, 'Specify the nameserver to use for queries (default is system DNS)']),
      • OptPath.new('WORDLIST', [false, 'Wordlist of subdomains', ::File.join(Msf::Config.data_directory, 'wordlists', 'namelist.txt')])
4. Test both modules to ensure that they're working.

[Hi111111]

Please don't send any messages to my email
REDACTED
Thanks

[smcintyre-r7]

@Hi111111 You need to unsubscribe from this Issue thread and potentially unwatch the repository to avoid receiving unwanted messages in your email. We don't control who is notified (that's handled through GitHub and your configuration), but the notifications are essential for the collaboration with the involved parties.

[mekhalleh]

hello @smcintyre-r7 all is done.

It is now enough to move the other functions from dns_enum module in the mixin :)

Do you think it would be interesting to do the same with the censys_search module?

[smcintyre-r7]

You're almost there, the 5 methods you moved into the mixin just need to be removed from the original dns_enum module. You already included the mixin in the dns_enum module so you just need to add the dns_ prefix to the originals and remove their definitions. For example, references to get_a need to be updated to dns_get_a and then get_a needs to be removed.

After that I think this'll be ready for some testing and I can get it processed assuming nothing else comes up. Thanks!

[smcintyre-r7]

Quick update, I made some changes to this code that you can see reflected in this branch. Some notable things I did included:

• Moved the new mixin to Msf::Exploit::Remote::DNS::Enumeration and used the existing client in there. This introduced a few bugs that I submitted a fix for in CVE-2018-17463, Google Chrome < 70 Object.create exploit #12384.
• Removed the WORDLIST_TLD option from the enum_dns module. It was complicating testing and I'd like to get these changes ironed out and merged in. If you'd like to resubmit that feature, we can take a look at it in a future branch.

Do you have a true positive test case I can use with this? The results are censored in your documentation and previous PRs but I'd like to test this more thoroughly. If you'd like you can send the HOSTNAME setting to msfdev@metasploit.com.

[smcintyre-r7]

Alright I was able to find a true positive test case. Following your lead I won't post the information here though. I've tested this PR successfully on a true positive and multiple true negatives. I've also tested with CENSYS API information as well as the old DNS enumeration module. At this point everything looks good to me, I'm just going to wait for the unit tests to pass and I'll get this merged in.

Thank you for all of your work on this! 👍

[mekhalleh]

Nice :) and sorry for your last request (about test cases) but I was full on a new job.
Thanks you for your help ;)

[smcintyre-r7]

Release Notes

This adds an auxiliary module that attempts multiple techniques to fingerprint IP addresses that can be used for directly connecting to web servers that are supposed to be protected by cloud based solutions. This helps to identify a common class of misconfiguration vulnerabilities in these scenarios.