lookuper performs lookups against VirusTotal/ThreatExpert/Google Safe Browsing/HaveIBeenPnwed data for data types such as MD5, SHA256, URL, IP, Domains, Strings, Mutexes and email addresses
Data is cached in a local SQLite database. The VirusTotal functionality supports multiple API keys, so you can supply four public API keys and it will run continuously, iterating through the API keys.
- Data caching in SQLite database
- Support for multiple API keys (VT)
- Batched requesting (VT)
- Supports hashes (MD5 & SHA256), URL's, IP's and domains from VirusTotal (https://www.virustotal.com/en/documentation/public-api/)
- Supports hashes (MD5, strings) from ThreatExpert (http://www.threatexpert.com/)
- Supports domains/URL's from Google SafeBrowsing (https://developers.google.com/safe-browsing/v4/)
- Supports email addresses from HaveIBeenPwned (https://haveibeenpwned.com/API/v2)
The applications configuration is read from the lookuper.config file located in the same directory as the binary.
The configuration holds the following values:
- safe_browsing_api_key: The API key used for the Google Safe Browsing functionality
- virus_total_api_keys: The API key(s) used for the VirusTotal functionality
- max_hash_age: The maximium age that the data is held before being deemed as stale
The following is an example layout of the lookuper.config file:
safe_browsing_api_key: ABC.... virus_total_api_keys: - AAA... - BBB... - CCC... - DDD... max_data_age: 30
The application uses commands to perform work for the different data types:
NAME: lookuper - Looks stuff up... USAGE: lookuper [global options] command [command options] [arguments...] VERSION: 0.0.7 AUTHOR(S): Mark Woan <firstname.lastname@example.org> COMMANDS: resume Resumes an existing process clear Clears the work queue md5vt Check MD5 hashes via VirusTotal md5te Check MD5 hashes via ThreatExpert sha256vt Check SHA256 hashes via VirusTotal ipvt Check IP addresses via VirusTotal domainvt Check domains via VirusTotal urlvt Check URL's via VirusTotal stringte Check strings via ThreatExpert gsb Check Url's/Domains via Google Safe Browsing hibp Check email addresses via HaveIBeenPwned help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --help, -h show help --version, -v print the version
An example command line and output is shown below:
./lookuper md5vt -i md5-28.txt -o . 2016/09/06 15:18:23 Data type: MD5 (VT) 2016/09/06 15:18:23 Loading data 2016/09/06 15:18:24 Loaded No. items: 28 2016/09/06 15:18:24 Data type: MD5 (VT) 2016/09/06 15:18:24 API key: ABC....... 2016/09/06 15:18:24 Batch size: 4 2016/09/06 15:20:38 Complete 2016/09/06 15:20:38 Cache hits: 0
This document assumes that the golang tool set has been installed.
Golang applications always have a src directory which contains the applications source code, along with any associated projects that are referenced by the primary application. The following shows where the applications source code resides:
The project uses gb for building the project. gb allows for reproducible builds and vendoring so that all dependencies are kept with the project source.
Compile with gb
To compile the application use the following commands (assuming the same directory structure):
$ cd /lookuper/source $ gb build all
Compile under Windows
The go-sqlite3 database driver used to access the Sqlite database is a cgo package, therefore gcc is needed to build the application. On linux this is generally not an issue as most normal people :-) have gcc installed.
On Windows, that is not always the case. So to build on Windows (x64), perform the following steps:
- Download tdm-gcc (x64) and install. Accept the defaults.
- Open a command prompt and CD to:
- Execute the mingwvars.bat file which will get the correct environment variables
- Then CD to the lookuper source code and follow the same instructions for compiling using gb