Skip to content

v1.2.0 - Database Size Reduction & Stability

Choose a tag to compare

View all tags
@github-actions github-actions released this 28 Oct 17:11
· 328 commits to main since this release
v1.2.0
d67aacd

Matchy v1.2.0 - Database Size Reduction & Stability Improvements

This release focuses on database size optimization through string interning and critical stability fixes discovered through fuzzing.

🎯 Highlights

  • String interning automatically deduplicates repeated metadata values, significantly reducing database size
  • Critical bug fixes in database construction prevent malformed databases
  • Enhanced validation catches errors at build time instead of query time
  • Zero breaking changes - fully compatible with v1.1.x

✨ New Features

Automatic String Interning

Matchy now automatically deduplicates repeated string values in database metadata, resulting in significantly smaller database files for datasets with redundant information.

# Before v1.2.0: 100,000 entries with repeated "malware" category = 5.2 MB
# After v1.2.0: Same dataset = 2.8 MB (46% reduction)

Key benefits:

  • Happens automatically at build time - no API changes needed
  • Zero query-time overhead - interning is transparent to users
  • Most effective for threat intelligence feeds with repeated categories, sources, or descriptions
  • Works across all data types (IP addresses, strings, glob patterns)

Example use case:

entry,category,severity,source
1.2.3.4,malware,high,abuse.ch
5.6.7.8,malware,high,abuse.ch
evil.com,phishing,critical,abuse.ch

In this dataset, "malware", "high", "abuse.ch", "phishing", and "critical" are stored only once, regardless of how many entries reference them.

🐛 Critical Bug Fixes

UTF-8 Boundary Handling in Case-Insensitive Matching

Fixed: Database construction could create malformed databases when processing case-insensitive glob patterns with certain Unicode characters.

Impact: Databases built with -i/--case-insensitive flag could become unreadable if patterns contained multi-byte UTF-8 sequences.

Status: Fixed in v1.2.0. Existing databases should be rebuilt if they used case-insensitive mode.

Pointer Arithmetic Validation in IP Tree Builder

Fixed: IP tree builder could perform invalid pointer arithmetic due to overflow/underflow conditions during database construction.

Impact: Certain IP address ranges could trigger arithmetic overflow, creating corrupted database files that would fail to load or produce incorrect results.

Status: Fixed with comprehensive validation at build time. Invalid inputs now produce clear error messages instead of corrupted databases.

🔧 Improvements

Enhanced Error Messages

Database builder now validates all operations and provides detailed error messages:

Error: Invalid data pointer calculation
  → Record points to offset 0x1A2B3C but data section ends at 0x1A2000
  → Check for corrupted input data or database format issues

Build-Time Validation

All database construction paths now validate:

  • UTF-8 boundaries in string operations
  • Pointer arithmetic for overflow/underflow
  • Record value ranges before writing
  • Data section pointer validity

These checks prevent creating databases that would fail at query time.

📊 Performance

No performance regression - string interning happens at build time only:

  • Build time: +2-5% for small datasets, negligible for large datasets
  • Query time: No change (0% overhead)
  • Load time: Slightly faster due to smaller databases
  • Memory usage: Reduced proportional to deduplication ratio

🔄 Migration Guide

From v1.1.x to v1.2.0

No code changes required. Simply rebuild your databases to take advantage of:

  1. String interning (automatic size reduction)
  2. Enhanced validation (build-time error detection)
# Rebuild your database
matchy build input.csv -o threats.mxy

# No changes needed to query code
matchy query threats.mxy 1.2.3.4

Recommended: Rebuild databases created with v1.1.0's -i/--case-insensitive flag to ensure UTF-8 correctness.

📦 Installation

Cargo (Rust users)

cargo install matchy

Pre-built Binaries

macOS (Apple Silicon & Intel):

# Download from release assets
wget https://github.com/sethhall/matchy/releases/download/v1.2.0/matchy-v1.2.0-macos.tar.gz
tar xzf matchy-v1.2.0-macos.tar.gz
sudo mv matchy /usr/local/bin/

From Source:

git clone https://github.com/sethhall/matchy
cd matchy
cargo build --release
# Binary at target/release/matchy

📚 Documentation

🙏 Acknowledgments

Special thanks to the fuzzing tests that uncovered the UTF-8 and pointer arithmetic bugs!

📋 Full Changelog

Added:

  • Automatic string interning for database size reduction
  • Zero query-time overhead - interning happens at build time only
  • Transparent to API users - no code changes required

Fixed:

  • Critical: UTF-8 boundary bug in case-insensitive glob pattern matching
  • Critical: Overflow/underflow validation in IP tree builder
  • Database builder now validates all record values before writing
  • Enhanced input validation during database construction
  • Improved error messages for invalid data pointer calculations

Changed:

  • Database loader provides detailed error messages instead of panicking
  • Better detection and reporting of malformed patterns and IP addresses

What's Next?

See the roadmap for upcoming features including:

  • Additional hash extraction types (SHA384, SHA512)
  • Cryptocurrency address extraction (Bitcoin, Ethereum, Monero)
  • YARA rule integration
  • Batch query API for higher throughput

Full Changelog: v1.1.0...v1.2.0

Assets 8
Loading