31 October 2024
- Enable TLS connection to Postgres using Rustls.
- Optional server TLS support using Rustls. Server certificate file may be just the cert, or cert + CA cert in one file.
- Optional TLS support in the client. Client configuration may specify a custom CA certificate path.
30 September 2024
- Http compression support. Client requests Gzip, server supports Brotli, Gzip, Zstd.
31 August 2024
- Vt-client updated to support Gzip compression with Virus Total.
- Better VT key checks.
28 July 2024
- Much more inline documentation, should make docs.rs for MalwareDB more informative.
- Removed all references to SDHash. It likely won't come back unless progress is made on the Rust crate which implements SDHash.
27 June 2024
- Client binary may generate autocompletion scripts.
- Client may view, create, unpack CaRT files.
- Better bounds checking when parsing PE32.
- Store section, architecture for executable files in the database.
- Sources can be labeled malicious. No impact yet but might automatically mark samples from that source as malicious.
- Better bounds checking when parsing ELFs.
- Client may submit samples and refer to source by name or ID.
- Client binary shows uploaded information if user presses Ctrl-C.
- On server first run, some admin commands are suggested if compiled with
adminfeature. - Server can create empty config file.
- Admin command may toggle ability to send unknown files to VirusTotal.
- Admin command can try to find config file.
- Admin command may toggle compression anytime (was first-run only).
30 May 2024
- Fix broken MDB client login.
- Implement getting list of sources for a user, this was in
malwaredb-apibut not implemented in the server or client.
- Extract timestamps from MS Office "docfile" documents.
29 April 2024
- VirusTotal reports generated by malwaredb-virustotal may be used to load AV reports into MalwareDB.
- Client crate is now also a library so it may be tested in unit tests, and may be integrated with other applications.
- Breaking change: HTTP API uses a Header instead of a Post struct for providing the API token to the server.
21 March 2024
- Add total AVs available when storing VirusTotal results.
- Postgres uses a transaction when inserting records to help preserve data integrity.
28 February 2024
- Support for encrypting the samples on the server side, with options being XOR, RC4, or AES-128.
- Files are encrypted with the latest key type, but files encrypted with an older key are still retrievable.
- Admin GUI can update label information, and display a label's parent.
- Server can decode CaRT submissions only if the default CR4 key is used.
- While working with unit tests, it was observed that before encoding, an ELF was 5k, which became 2.8k after encoding, thanks to CaRT's use of zlib.
- The server binary can be used to encode, decode, and display information about a CaRT file. See
mdb_server cart --helpfor details.
29 January 2024
- Added a URL endpoint to download a sample as a CaRT file, which allows for safe handling of malware by ensuring the sample can't be accidentally run or trigger an anti-virus application. This uses the default RC4 key, which is Pi twice.
Version 0.0.4 skipped due to botched release. 30 December 2023
- Upload sample to VT if permitted.
- New VirusTotal client, which parses out scan/report details into a big struct for ease of use. Later (12/20) moved to its own repository, malwaredb/vt-client.
- Decompression of gzipped files when adding to MalwareDB with the client or admin.
- Server optionally may store the samples compressed with gzip.
- Display of label information in admin GUI.
- Admin GUI may edit group information.
28 October 2023
- MalwareDB can generate the configuration file to run as a service on Linux (systemd), macOS, and FreeBSD.
- Sources viewable in admin GUI.
- Preliminary identification of MS Office files (Office95/Docfile).
- Start of label support. Admin CLI and client can see labels, admin CLI can create labels, but they aren't applied yet to samples or sources.
- Add support for RTF files, more to be done on identifying useful information.
- CI checks for unused dependencies.
- Show group information in the admin GUI.
- Use
libmagicto get a description of the file type. This is often useful, but MalwareDB doesn't use it for decisions.
27 September 2023
- Added macOS and Windows to the build CI, test fails for Windows because the test deletes the SQLite DB file, but Windows won't allow that claiming the file is still in use.
- Removed the
ssdeepcrate in favour of thefuzzyhashcrate, sincessdeepwasn't building on Windows andfuzzyhashhas no dependencies. - Published latest changes to
malwaredb-lzjdwhich uses newer dependencies, making theCargo.locksmaller. - Updated readme docs which will appear in on Crates.io for the sub-crates when next published.
- Replaced fltk-rs with slint. The UI is easier to work with.
- Additional admin SQL functions and data in the
admin:Userstruct to be shown in the GUI. - Update CI so the GUI components are included when ensuring the project builds and is tested with Clippy.
- Trying to use Postgres in CI, but running into an issue where
psqlcan't connect, but the output from Docker shows Postgres listening on 0.0.0.0:5432. Further work needed. - Created the data repository, which will hold some SQL files for some data, starting with VirusShare zip file sources.
- Initial work to report back to the client if a submitted file was actually processed or not.
- Additional admin data: show number of files associated with Groups and Sources, number of Groups associated with a Source.
- Submit files into MalwareDB from a zip file from the server admin CLI and the client application.
21 August 2023
- MalwareDB is now on crates.io!
- Install the client:
cargo install malwaredb-client - Install the server:
cargo install malwaredb --features=admin,sqlite,admin-gui(remove any undesired features from the command).
- Install the client:
- Started work on admin GUI, installed with the
admin-guifeature on the server. - Uses fltk-rs & fltk since it's cross-platform and has the desired GUI widgets.
- Just shows some data, no CRUD (create, rename, update, delete) yet.
- Read from Zip files, including password-protected Zips.
- Dockerfile in progress.
- Client can walk one or more directories, including symbolic links.
- Added an option for maximum file size.
- Similarity search using some similarity metrics implemented.
- DB info reports if the extensions are installed (currently required for Postgres).
- Server now saves samples, if configured to do so.
- Files may now also be retrieved.
- Rust 1.70 adds
OnceLock, which allowed for the removal of the unsafe static variable for the global config! -- Removed, no longer needed. No more global variables. - Still need some unsafe code to efficiently parse PE32 files in the types crate, unless I switch to the goblin crate.
- Files can be inserted into the database! Still working out how to handle the trait for executable file types, and how to have that data inserted cleanly.
- Added a trait for
.entropy()for a vector of bytes. - Some better PDF parsing to see if there's Javascript, forms; get author, title, and number of pages.
- Started work on figuring out how to handle file type detection where the parsers and types in the
typessub-crate may not be the supported files in thefiletypedatabase table. - This so far is well decoupled, so the
typessub-crate doesn't know about databases, HTTP, etc. - The new
types::KnownTypeenum will expose the format-specific data to theservercrate so the extracted data can be saved to the database. - Still need to figure out the similarity hashing situation, so that LZJD, SSDeep, SDHash, TLSH, and possibly others can easily be used by Rust.
- Server finds it's config file in this order for all OSes:
~/.mdb_server/mdb_config.tomlfor instances where the config is desired to be in the service account's home directory/etc/mdb_server/mdb_config.tomlintended for Linux/usr/local/etc/mdb_server/mdb_config.tomlintended for FreeBSD- Run
./mdb_server run load <PATH_TO_FILE>to provide the config file path - Run
./mdb_server run config [-p <PORT>] [--ip <IP_ADDR>] [--dir <PATH_TO_FILES_DIR>] -db <DB_CONFIG_STRING>- Default IP is
127.0.0.1 - No storage directory means MalwareDB won't save the samples it receives
- The database config string format:
"postgres user=usernamehere password=passwordhere dbname=thedatabasename host=localhost
- Default IP is
- The config files are in a custom directory since we'll probably need to store SSL cert and key as well, possibly other information.
- Admin features so far (when compiled with
cargo build --features=admin):- Create & list users
- Create & list groups
- Create & list sources
- List supported data types
- Missing: adding, removing user accounts to groups; adding, removing access to sources to various groups.
- Client features so far:
- Login (get API key, save API key and server URL in a config file at
~/.mdb_client.toml) - Show information about self: user ID, user name, group memberships, usable sources
- Show some server information (total number of files, memory/disk used, component versions, uptime)
- Show supported data types
- Login (get API key, save API key and server URL in a config file at
- Some initial work is in place to handle files once received, but that's not implemented yet.
- Users can login, get the API key to the local MDB client config.
- Trait
HashPathfor easy creation of a relative path based on a SHA-256 hash. - Moved LZJD, Murmur hashes from rjzak account to MalwareDB org.
- Get, display server stats.
- Finding global config for MDB server.
- Database pool for Postgres connection.
- Figured out how to return JSON with Axum.
- Figured out how to handle DB connections with Axum, but had to get rid of
DatabaseConnectiontrait due to complexities of having a trait withasyncfunctions, and that the async-trait crate adds weirdness to data returned by the async functions.
- Looking at using the
clap_completecrate to generate shell autocompletion scripts. Didn't work when tested, but will investigate.
- Postgres and SQLite are able to create users, login (get API key), reset all API keys, change passwords.
mdb_servermay have admin capabilities if compiled with--features=admin. SQLite support is also gated behind a feature flag,--features=sqlite.- Unit tests are focused on SQLite for ease, but also exist for Postgres. Run with
cargo test -- --ignored. Requires a database with username, password, and db name: "malwaredbtesting" on localhost. - All crates have
pub const MDB_VERSION: &str = env!("CARGO_PKG_VERSION");at the root level for future API use, to help ensure the client, server, and any future use can make sure the versions are the same before trying to make or interpret requests.
- Refactored the code to prepare for client/server development. The
apisub-crate will hold the common data structures used for communication. - There will be only one
typescrate for all supported data formats, instead of the original plan of sub-crate per class of formats.
- Trying to parse ELF imports is proving to be rather difficult!
- Goblin seems to be a well-supported and feature-rich crate, but I haven't figured out how to match the imported function to the library, so I asked.
- Initial commit.