#156 Initial implementation of sqlite3 integration

[j-baines]

Implementation Details

This pull request attempts to address #156. This implementation introduces a new -db command line parameter that allows a user to provide an SQLite3 database. The database contains three tables (whose schema you can find defined in the const area in db/create.go):

1. metadata: Stores the schema version so we can detect incompatible schema versions in the future, and the creation time.
2. verified: Stores the results of our validateTarget function for a given ip/port/software, as well as the version string (when possible).
3. httpCache: Stores HTTP responses based on the host, port, and uri.

The verified logic should be totally transparent to framework users (meaning, it just automatically works). The HTTP caching is a little different. HTTP caching is... not a straight forward topic and I certainly didn't want to introduce anything weird in the hundreds of exploits we've written (e.g. if we accidentally cache a page that we need a fresh CSRF token from... that's bad). As such, I added a single function that the developer can use to tell the system to cache the page. That's protocol.HTTPGetCache.

I think protocol.HTTPGetCache will likely just be used to replace all the protocol.HTTPSendAndRecv in ValidateTarget and CheckVersion. For example, This is the diff I have for testing the changes in CVE-2024-31982:

diff --git a/feed/cve-2024-31982/cve-2024-31982.go b/feed/cve-2024-31982/cve-2024-31982.go
index 58f3776..2609651 100644
--- a/feed/cve-2024-31982/cve-2024-31982.go
+++ b/feed/cve-2024-31982/cve-2024-31982.go
@@ -26,7 +26,7 @@ type XWikiSearchInjection struct{}
 // and the xwiki div tag.
 func (sploit XWikiSearchInjection) ValidateTarget(conf *config.Config) bool {
        url := protocol.GenerateURL(conf.Rhost, conf.Rport, conf.SSL, "/")
-       _, body, ok := protocol.HTTPSendAndRecv("GET", url, "")
+       _, body, ok := protocol.HTTPGetCache(url)
        if !ok {
                return false
        }
@@ -39,7 +39,7 @@ func (sploit XWikiSearchInjection) ValidateTarget(conf *config.Config) bool {
 // it seemed about half exposed a version.
 func (sploit XWikiSearchInjection) CheckVersion(conf *config.Config) exploit.VersionCheckType {
        url := protocol.GenerateURL(conf.Rhost, conf.Rport, conf.SSL, "/")
-       _, body, ok := protocol.HTTPSendAndRecv("GET", url, "")
+       _, body, ok := protocol.HTTPGetCache(url)
        if !ok {
                return exploit.Unknown

The way the HTTP caching works is actually really simple. We just use Response.Write to serialize a response to a string and http.ReadResponse to serialize it back. There has never been any issues with de-serialization of stored data, so I'm sure it's great (this is sarcasm... but I'm not aware of this being a dangerous function).

The SQLite3 driver I ended up using was the modernc library. This is attractive because it's cgo-free, has a decent user base, and appears to be actively maintained. It is the slowest driver but I don't think that matters for what we are doing (like the win for a cache hit will always be huge).

Finally, I did add logic that would allow a user to cache no HTTP but still use the database. The flag is -cacheMax and is set to 10 MB by default. The user can set it down to 0 if they want.

Oddities

Other oddities in this patch:

• c2Flags got pulled out into its own function. My small addition had bumped up gocyclo too high.
• I fixed the backwards return in CreateRequest that @terrorbyte had commented on (and was documented in Document protocol.CreateRequest bool Return #138)

Finally, allow me to copy and paste the .md I wrote:

Database Usage

go-exploit supports the use of an SQLite3 database in order to facilite cross-exploit communication and HTTP caching. This is optional, but can greatly improve the performance of large scale scanning.

To use this feature, use the -db command line option. The provided file can be empty or a database created during previous runs of go-exploit. In the example below, we use the option -db vc.db to use the non-existent file vc.db:

albinolobster@mournland:~/initial-access/feed/cve-2024-31982$ ls vc.db
ls: cannot access 'vc.db': No such file or directory
albinolobster@mournland:~/initial-access/feed/cve-2024-31982$ ./build/cve-2024-31982_linux-arm64 -v -c -rhost 10.9.49.29 -rport 8080 -db vc.db -fll TRACE
time=2024-06-26T15:54:18.902-04:00 level=DEBUG msg="Using the HTTP User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
time=2024-06-26T15:54:18.911-04:00 level=STATUS msg="Starting target" index=0 host=10.9.49.29 port=8080 ssl=false "ssl auto"=false
time=2024-06-26T15:54:18.911-04:00 level=STATUS msg="Validating XWiki target" host=10.9.49.29 port=8080
time=2024-06-26T15:54:19.068-04:00 level=SUCCESS msg="Target verification succeeded!" host=10.9.49.29 port=8080 verified=true
time=2024-06-26T15:54:19.068-04:00 level=STATUS msg="Running a version check on the remote target" host=10.9.49.29 port=8080
time=2024-06-26T15:54:19.068-04:00 level=TRACE msg="HTTP cache hit: http://10.9.49.29:8080/"
time=2024-06-26T15:54:19.069-04:00 level=VERSION msg="The reported version is 14.10.7" host=10.9.49.29 port=8080 version=14.10.7
time=2024-06-26T15:54:19.070-04:00 level=SUCCESS msg="The target appears to be a vulnerable version!" host=10.9.49.29 port=8080 vulnerable=yes
albinolobster@mournland:~/initial-access/feed/cve-2024-31982$ ls vc.db 
vc.db

In the TRACE output we can see that there is an HTTP cache hit during version scanning, this is because the first request in target verfication was cached. If we run the exploit again we will see more TRACE output:

albinolobster@mournland:~/initial-access/feed/cve-2024-31982$ ./build/cve-2024-31982_linux-arm64 -v -c -rhost 10.9.49.29 -rport 8080 -db vc.db -fll TRACE
time=2024-06-26T15:55:57.566-04:00 level=DEBUG msg="Using the HTTP User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
time=2024-06-26T15:55:57.570-04:00 level=STATUS msg="Starting target" index=0 host=10.9.49.29 port=8080 ssl=false "ssl auto"=false
time=2024-06-26T15:55:57.570-04:00 level=STATUS msg="Validating XWiki target" host=10.9.49.29 port=8080
time=2024-06-26T15:55:57.570-04:00 level=TRACE msg="Verified software cache hit" result=true
time=2024-06-26T15:55:57.570-04:00 level=SUCCESS msg="Target verification succeeded!" host=10.9.49.29 port=8080 verified=true
time=2024-06-26T15:55:57.570-04:00 level=STATUS msg="Running a version check on the remote target" host=10.9.49.29 port=8080
time=2024-06-26T15:55:57.570-04:00 level=TRACE msg="HTTP cache hit: http://10.9.49.29:8080/"
time=2024-06-26T15:55:57.570-04:00 level=VERSION msg="The reported version is 14.10.7" host=10.9.49.29 port=8080 version=14.10.7

Note the first TRACE this time is for Verified software cache hit. That is because the result of the previous run was saved in the database, so this exploit knows that 10.9.49.29:8080 is XWiki.

albinolobster@mournland:~/initial-access/feed/cve-2024-31982$ sqlite3 vc.db 
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> select * from verified;
1|1719431659|XWiki|1|14.10.7|10.9.49.29|8080
sqlite> 

In fact, on this second run, no network traffic was generated. This has a variety of useful applications including improved speed, asset database generation, easy to create test databases, and "scanless" scans.

[j-baines]

Tested most recent version across Windows, Linux, and Darwin.