Fix reference URLs for several modules and update tools/modules/module_reference.rb code with fixes

[h00die]

fixes #4039

Updated tools/modules/module_reference.rb with a rubocop run, removed the SSL version reference to use the default. Made it so 300s now mark as down, and a CSV export.

This gave ~2600 URLs with issues (some seem to work and were false positives).

Mainly just either putting in the new URLs (if it forwards), looking for the real URL via google search, or worst case going with wayback machine references. If all those cases fail, I just leave it as is.

[h00die]

Linting will fail on this because i'm touching so many modules. I have no intention of doing rubocop on them. #dealwithit

[adfoster-r7]

@h00die To help with reviewing/landing, it'd be great to revert the rubocop autocorrections and keep this PR limited to the URL fixes 👀

Or if that's overhead, I can create a separate PR to rubocop these modules to help reduce the noise a bit

[h00die]

To save some code somewhere, i'm using a really quick and dirty python script to go through the csv output of module_reference.rb (csv output coming to this pr) to catch all the 300s and overwrite them in the module. MANY of the responses are just http to https or adding a / at the end, so this catches many of those. However, it isn't perfect as many sites will forward you to a home page and such. Manual review still required.

import csv
import requests

cache = {}


def fix(module, old, new):
    print(f"  Updating {module} from {old} -> {new}")
    try:
        with open(f"modules/{module}.rb") as r:
            data = r.read()
        data = data.replace(f"'old'", new)
        with open(f"modules/{module}.rb", "w") as w:
            w.write(data)
    except Exception as e:
        print(e)


with open("/tmp/all_url_refs.csv", newline="") as csvfile:
    reflist = csv.reader(csvfile)
    for row in reflist:
        module = row[0]
        status = row[1]
        badurl = row[2].replace("URL-", "")
        if status != "Down":
            continue
        print(f"Checking: {badurl}")
        if badurl in cache:
            print("  got a cache hit")
            fix(module, badurl, cache[badurl])
            continue
        try:
            r = requests.get(badurl, allow_redirects=True, timeout=5)
            if (
                r.status_code == 200
                and not "not found" in r.text.lower()
                and r.url != badurl
            ):
                cache[badurl] = r.url
                fix(module, badurl, cache[badurl])
        except Exception as e:
            print(e)

[h00die]

@bcoles don't start looking through this yet. I'll be more or less updating all of this hopefully today with the final run

[h00die]

This is now ready for review. Obvi it isn't exhaustive, but it does update quite a few old and really old links to work again.

I think the funniest one is http://www.smbloris.com/ is now an office furniture store or something.

[bcoles]

[...] csv output coming to this pr

I think this closes #4039 ? This issue is 7 years old and mostly done. It was re-opened until the results were "easy to prune". Presumably your CSV output is easy to parse.

[bcoles]

Should this be 'https://labs.f-secure.com/tools/sap-metasploit-modules/' or 'https://labs.f-secure.com/archive/mwr-sap-metasploit-modules/' like the others?

I downloaded the zip, but didn't find this module in there (perhaps the file name has changed).

[h00die]

I tried to do 1:1 replacements, investigating 2,600 broken URLs intelligently was entirely too time consuming. I'm keen to leave this as is since that's what was originally there

[bcoles]

I read through the URL changes.

I noticed that all r-7.co links were changed (presumably because they 30x redirected). Presumably, the purpose of this domain was for conventiant URL shortening. On the other hand, 30x status codes should be considered an error. 🤷

There are a few references like these which simply point to a vendor or researcher's domain/blog, rather than anything relevant to the module:

[ 'URL', 'https://blog.c22.cc' ]

[ 'URL', 'https://labs.f-secure.com/' ]

These references aren't useful and are more akin to advertising.

[h00die]

agree on the r-7.co shortlinks. I could go either way.

[github-actions]

Thanks for your pull request! Before this pull request can be merged, it must pass the checks of our automated linting tools.

We use Rubocop and msftidy to ensure the quality of our code. This can be ran from the root directory of Metasploit:

rubocop <directory or file>
tools/dev/msftidy.rb <directory or file>

You can automate most of these changes with the -a flag:

rubocop -a <directory or file>

Please update your branch after these have been made, and reach out if you have any problems.

[adfoster-r7]

@gwillcox-r7 Looks like in this scenario we should be good to ignore the linting failures. The pull request is just updating the URLs of the modules - and it would just muddy the waters to request linting 367 modules as part of this pull request 👀

[adfoster-r7]

@h00die If you're good for this to be reviewed/landed, it'd be great to squash this down to a single commit 🎉

[gwillcox-r7]

@gwillcox-r7 Looks like in this scenario we should be good to ignore the linting failures. The pull request is just updating the URLs of the modules - and it would just muddy the waters to request linting 367 modules as part of this pull request 👀

No problem I was just going through PRs that are open and was assigning tags to those that were open but didn't have labels attached to them yet. Wasn't sure if we needed that given the changes being made here so thanks for clarifying.

[h00die]

squishy squashy, all in one.

[gwillcox-r7]

Will need to rebase this as we removed the VSS modules so they don't need to be updated anymore, but rest looks like it should be easy to check and then land.

[gwillcox-r7]

Fixed up some of the errors encountered when running the tool in d7b442f

[gwillcox-r7]

Running a complete check now. Is taking a while (been nearly an hour now) but hopefully it should tell us if the latest changes work and if there are any remaining URLs that need fixing.

[gwillcox-r7]

I'm in disagreement with this and the prime reason here is that if the code is a redirect code, your essentially going to end up with hundreds of links that end up being reported as being "down" when in reality they aren't actually down.

Case in point I ran this against all URLs in the database, and ended up with this result:

post/windows/manage/sticky_  Down         URL-https://blog.carnal0wnage.com/2012/04/pri
  keys                                      vilege-escalation-via-sticky-keys.html
  post/windows/manage/vmdk_mo  Down         URL-http://www.shelliscoming.com/2017/05/post
  unt                                       -exploitation-mounting-vmdk-files.html
  post/windows/manage/vss      Alive        URL-https://web.archive.org/web/2020111121295
                                            2/https://securityweekly.com/2011/11/02/safel
                                            y-dumping-hashes-from-liv/
  post/windows/recon/outbound  Down         URL-http://www.shelliscoming.com/2014/11/gett
  _ports                                    ing-outbound-filtering-rules-by.html

Number of bad references found: 8663

Note here that http://www.shelliscoming.com/2017/05/post-exploitation-mounting-vmdk-files.html is considered down but in reality if you visit that page it just upgrades to a SSL version of that page so its not really "down" per say.

Now that being said I'm all for updating these old links to https links, but this labeling of a link as "down" doesn't really serve to help anyone and only causes more confusion.

[gwillcox-r7]

I also have concerns r.e the accuracy of this script given that https://blog.carnal0wnage.com/2012/04/privilege-escalation-via-sticky-keys.html was incorrectly labeled as being down. I'll have to look into this further.

[gwillcox-r7]

Alright so the response to the link above was HTTP/2 200 OK so its possible that maybe the handler doesn't handle HTTP/2 responses correctly? Idk.

[gwillcox-r7]

I also have concerns r.e the accuracy of this script given that https://blog.carnal0wnage.com/2012/04/privilege-escalation-via-sticky-keys.html was incorrectly labeled as being down. I'll have to look into this further.

This has been addressed with 550a625 which does a major rework in an attempt to support redirections by defining a new status type for redirects and updating the code to report the status appropriately whilst also still not labeling the URL as down.

The HTTP/2 issue still exists however.

[gwillcox-r7]

Our Rex library appears to be pulling some weird shenanigans on SSL versions that honestly is leading me scratching my head. Seems to default to SSLv3 if a version is not supplied but if I try change it then its still throwing errors about trying to perform a handshake with SSLv3 that failed.

Ideally as per some discussions with @smcintyre-r7 (sorry CCing you in here as I think this may be an issue with Rex itself), Rex would actually bother to renegotiate the SSL version but this doesn't seem to be the case right now.

[h00die]

yea i wasn't happy about calling redirects down, but was also trying to keep it more Boolean. I also figured if it isn't perfect, I'd rather call it out to get fixed. regardless, this is a vast improvement regardless.

I noticed the SSL issues as well, didn't want to go down an additional rabbit hole as this was already enough

[gwillcox-r7]

No worries @h00die, I think our main concern is if this could affect other modules if this is affecting the Rex library. Also personally I really don't like the fact that this module probably hasn't been working for a long time now so forgive my continuous editing to try get it working again 😄

[gwillcox-r7]

SSL related issues are now being tracked at #16213 as this appears to be a bigger issue affecting much more of our code base than originally anticipated. As this is an unrelated issue to this code though, which seems to be working fine minus that issue, I'm going to go ahead and land this.

[gwillcox-r7]

Fixed up last RuboCop issues cause it wasn't that many and was an easy fix. Will land this now, future work on SSL will be another PR as that is a larger job.

[gwillcox-r7]

Release Notes

A number of broken URL references have been fixed in Metasploit modules. In addition the tools/modules/module_reference.rb code has been updated to log redirects so that they can be appropriately triaged later and to support saving results to a CSV file. Finally, several modules had their code adjusted to conform to RuboCop standards.