check_external_scripts.rb add more files to check

[h00die]

This PR adds the sqlmap UDF and JTR files to the external scripts check (#10579). The external scripts checker is an external "lib" checker, to ensure all non-gem libraries, configs, and scripts that are used by MSF are up to date.

• I went ahead and included the JTR file updates, and tested them with the cracker modules, and found no issues.
• The UDF files were up to date after decloaking.
• I've also included the SharpHound.ps1 updates.

To Test:

• run tools/dev/check_external_scripts.rb and ensure it doesn't crash, and doesn't get any 404 or other related issues (non-200ish), and all files are up to date.
• check that the JTR modules (cracker) are still working. See https://github.com/rapid7/metasploit-framework/wiki/Hashes-and-Password-Cracking#example-hashes for some hashes to add and then crack them.
• Test the sharphound post module still works

[h00die]

Tested JTR changes. Everything still cracks the same pre/post. No real time change in cracking (on an i7 anyways).
pre, and post.

65.47user 4.27system 0:29.10elapsed 239%CPU (0avgtext+0avgdata 260008maxresident)k
0inputs+488outputs (0major+1341656minor)pagefaults 0swaps

65.78user 4.11system 0:29.17elapsed 239%CPU (0avgtext+0avgdata 260060maxresident)k
0inputs+488outputs (0major+1334142minor)pagefaults 0swaps

[smcintyre-r7]

It looks like this modified the binaries in data/exploits/mysql/. Are those related to this change? It's hard to tell what was changed about them on GitHub other than that the size went down.

[h00die]

Yes those are related. I last updated them like 2yrs ago, so not surprising there is some change, but unsure what all it was. I did not test the UDF file changes

[h00die]

Reverted the mysql ones. Looks like they changed something (obvi due to file size decrease). Reverted to the last ones we had and its working fine, but here was testing on the new version.

[*] Started reverse TCP handler on 1.1.1.1:4444 
[*] 2.2.2.2:3306 - Checking target architecture...
[*] 2.2.2.2:3306 - Checking for sys_exec()...
[*] 2.2.2.2:3306 - Checking target architecture...
[*] 2.2.2.2:3306 - Checking for MySQL plugin directory...
[*] 2.2.2.2:3306 - Target arch (linux64) and target path both okay.
[*] 2.2.2.2:3306 - Uploading lib_mysqludf_sys_64.so library to /usr/lib/mysql/plugin/MOLAdXgh.so...
[-] 2.2.2.2:3306 - MySQL Error: RbMysql::CantOpenLibrary Can't open shared library 'MOLAdXgh.so' (errno: 11 /usr/lib/mysql/plugin/MOLAdXgh.so: invalid ELF header)
[*] 2.2.2.2:3306 - Checking for sys_exec()...
[*] 2.2.2.2:3306 - MySQL function sys_exec() not available
[*] Exploit completed, but no session was created.

Bloodhound also had another update.

[h00die]

Ok, got the udf stuff figured out.
SQLMap has the cloaked versions, we use the de-cloaked version. Ref https://github.com/rapid7/metasploit-framework/pull/10428/files#diff-16c64629a8b04ecc1635cd9dddcd64edR6

@stamparm you last provided de-cloaked versions #9677 (comment) about 2yrs ago. Asking you for de-cloaked versions every once in a while isn't necessarily sustainable. Is there any way we can have those either added to the sqlmap git, or a diff repo so they could be automatically pulled?

I'm going to delay this till we can find a good way here. Either UDF files need to be removed from this script, or adjusted if @stamparm is willing to upload decloaked versions for us to pull.

[stamparm]

@h00die I am going to leave a generic "how-to" de-cloak here:

$ cd /tmp
$ git clone --depth=1 https://github.com/sqlmapproject/sqlmap.git
$ cd sqlmap
$ find data/udf -type f -iname '*_' -exec python extra/cloak/cloak.py -d -i '{}' \;
$ find data/udf -type f -iname '*.so'
data/udf/postgresql/linux/32/9.3/lib_postgresqludf_sys.so
data/udf/postgresql/linux/32/9.1/lib_postgresqludf_sys.so
data/udf/postgresql/linux/32/9.5/lib_postgresqludf_sys.so
data/udf/postgresql/linux/32/11/lib_postgresqludf_sys.so
data/udf/postgresql/linux/32/10/lib_postgresqludf_sys.so
data/udf/postgresql/linux/32/8.4/lib_postgresqludf_sys.so
data/udf/postgresql/linux/32/9.0/lib_postgresqludf_sys.so
data/udf/postgresql/linux/32/9.2/lib_postgresqludf_sys.so
data/udf/postgresql/linux/32/9.4/lib_postgresqludf_sys.so
data/udf/postgresql/linux/32/8.3/lib_postgresqludf_sys.so
data/udf/postgresql/linux/32/9.6/lib_postgresqludf_sys.so
data/udf/postgresql/linux/32/8.2/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/9.3/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/9.1/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/9.5/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/11/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/10/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/8.4/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/9.0/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/9.2/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/9.4/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/8.3/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/9.6/lib_postgresqludf_sys.so
data/udf/postgresql/linux/64/8.2/lib_postgresqludf_sys.so
data/udf/mysql/linux/32/lib_mysqludf_sys.so
data/udf/mysql/linux/64/lib_mysqludf_sys.so

p.s. "cloaking" is used because sqlmap (previously) regularly appeared on malware blacklists based on these compiled binaries

[h00die]

@stamparm worked great, thanks!

[h00die]

there was some back and forth on this, but its ready to be tested. In the end, the mysql UDF files are still current.

[h00die]

Will need an update when #13194 lands to add the shaphound exe. Depending which lands first....

[h00die]

since the bloodhound module got fixed, added sharphound.exe to the update list. Also updated all the shaprhound modules. This should be ready to review again since we're no longer in a chicken and egg situation

[gwillcox-r7]

A few minor changes though my bigger concern right now is that the SharpHound.ps1 file isn't the same one that is on GitHub, and I also have concerns regarding us importing DLLs and EXEs that have been compiled by someone else directly into Metasploit since we have a lot of rules against doing that in general.

Let me get back to you r.e the DLLs and EXEs, but the other fixes should be relatively easy to make. PS1 file should also be easy to update assuming that updating it won't break anything but let me know if this is not the case and we can look into it further.

[gwillcox-r7]

@h00die Also looks like some further changes to the script might be needed:

Downloading: SQLMap UDF - lib_mysqludf_sys_32.so
[INFO] Old Hash: df70cdb324a653a02d77c7b7cdc1e595852c5200
[INFO]  Performing decloaking
[ERROR] Destination not found, check path: /home/gwillcox/git/metasploit-framework/data/exploits/mysql/lib_mysqludf_sys_32.so
Downloading: SQLMap UDF - lib_mysqludf_sys_64.so
[INFO] Old Hash: 5ac015b797818474e64a57df9774bff984107dd5
[INFO]  Performing decloaking
fatal: destination path '/tmp/sqlmap_decloak' already exists and is not an empty directory.
[ERROR] Destination not found, check path: /home/gwillcox/git/metasploit-framework/data/exploits/mysql/lib_mysqludf_sys_64.so
Downloading: SQLMap UDF - lib_mysqludf_sys_32.dll
[INFO] Old Hash: a52d0798898a4cb34a8cf2ba2bbef8a7b18c6b84
[INFO]  Performing decloaking
fatal: destination path '/tmp/sqlmap_decloak' already exists and is not an empty directory.
[ERROR] Destination not found, check path: /home/gwillcox/git/metasploit-framework/data/exploits/mysql/lib_mysqludf_sys_32.dll
Downloading: SQLMap UDF - lib_mysqludf_sys_64.dll
[INFO] Old Hash: fea5159fd0a741e8c12f86756b04474ad8a25724
[INFO]  Performing decloaking
fatal: destination path '/tmp/sqlmap_decloak' already exists and is not an empty directory.
[ERROR] Destination not found, check path: /home/gwillcox/git/metasploit-framework/data/exploits/mysql/lib_mysqludf_sys_64.dll

Looks like the script is failing if /tmp/sqlmap_decloak already exists.

[gwillcox-r7]

Update: After further debugging I determined that the real problem was because the script assumes that git and python are already installed on the target system. This may not always be the case, such as in my case where I had git installed but whilst Python was installed, it was installed as python3, so python3 was a valid command, but python was not.

More to the point though, we should not be assuming that users will have Python or Git installed when running this script.

The next commit should fix this by adding in a few checks to ensure that the commands executed successfully before continuing. Aka git exists, the target files exist, and python or python3 exists.

Applied changes myself with last commit

[gwillcox-r7]

Hmm so one last change that I think might be good is to remove the need to keep cloning the repo again and again for every sqlmap related file we need to copy over. Would probably be better to just have a function that is called once at the start of the file downloads which removes any old folder at /tmp/sqlmap_decloak, and then clones the repo into it, another function which calls the decloak function on each file as needed, and a final cleanup function which simply deletes the /tmp/sqlmap_decloak folder once we are done decloaking all the files. This will remove a lot of extra overhead that is incurred by the git clone call that is not needed for what we are doing.

[gwillcox-r7]

Release Notes

Updated tools/dev/check_external_scripts.rb to include JohnTheRipper, SQLMap UDF, and SharpHound.ps1 related files, providing assurance that those related libraries and configuration files are kept up to date.