FlashFS v0.2.0

FlashFS v0.2.0 Release Notes

We're excited to announce the release of FlashFS v0.2.0. This release includes significant improvements to the query engine, enhanced FlatBuffers schema support, and various performance optimizations.

New Features

Improved Query Engine

• Redesigned Query Interface: The query engine now uses a more flexible interface that directly works with the storage layer, eliminating the need for adapter patterns.
• Enhanced Parallel Querying: Improved the parallel querying of hot and cold layers for better performance.
• Expanded Query Options: Added support for more granular filtering options including directory filtering.

FlatBuffers Schema Enhancements

• Updated Schema Definition: Enhanced the FlatBuffers schema to include directory information in diff entries.
• Regenerated Code: Fully regenerated the FlatBuffers code to ensure all fields are properly accessible.
• Improved Serialization: Better handling of serialization and deserialization of snapshot and diff data.

Performance Improvements

• Streaming Optimizations: Removed redundant path walking in the streaming implementation, significantly improving performance for large directories.
• Progress Reporting: Enhanced progress reporting with proper error handling for all operations.
• Memory Usage: Reduced memory footprint during snapshot and diff operations.

Bug Fixes

• Fixed Type Mismatches: Resolved several type mismatch issues between the query engine and storage layer.
• Error Handling: Improved error handling throughout the codebase, particularly in progress reporting.
• Linter Compliance: Fixed various linter warnings and unused code.

Developer Experience

• Cleaner API: Simplified the API for creating and using query engines.
• Better Testing: Enhanced test coverage for the query engine.
• Code Quality: Removed unused code and improved code organization.

Upgrading

This release includes some breaking changes to the query engine interface. If you're using the query engine directly, you'll need to update your code to use the new interface. See the updated documentation for details.

What's Next

We're continuing to work on improving FlashFS with a focus on:

• Cloud storage integration enhancements
• Further performance optimizations
• Additional query capabilities
• Improved documentation

Thank you to all contributors who helped make this release possible!

v0.1.0

FlashFS v0.1.0 Release Notes

FlashFS is a high-performance file system snapshot and diff tool designed for efficiently tracking and managing changes across file systems over time.

Overview

FlashFS v0.1.0 provides a robust foundation for creating, comparing, and managing file system snapshots with a focus on performance and efficiency. This release introduces the core functionality and architecture that will serve as the basis for future enhancements.

Key Features

Snapshot Management

• Fast Snapshot Creation: Efficiently capture the state of your file system with parallel processing
• Content-Based Deduplication: Use BLAKE3 hashing for accurate file identification
• Configurable Compression: Balance speed and size with zstd compression
• Intelligent Caching: Optimize performance for frequently accessed snapshots

Diff Computation

• Structured Diff Format: Clear representation of added, modified, and deleted files
• Bloom Filter Pre-filtering: Quickly identify potential changes between snapshots
• Parallel Processing: Distribute comparison work across multiple CPU cores
• Path Filtering: Focus comparisons on specific directories or file patterns

Snapshot Lifecycle Management

• Flexible Expiry Policies: Configure retention based on time periods and snapshot counts
• Time-Based Retention: Keep hourly, daily, weekly, monthly, and yearly snapshots
• Count-Based Retention: Limit the total number of snapshots
• Age-Based Cleanup: Automatically remove snapshots older than a specified age

Query Capabilities

• Path Pattern Matching: Find files matching specific patterns
• Metadata Filtering: Filter by modification time, size, and file type
• Multiple Output Formats: Export results as text, JSON, or CSV

Technical Highlights

• Efficient Binary Serialization: Using FlatBuffers for compact representation and fast access
• Minimal Memory Usage: Streaming processing for handling large file systems
• Thread Safety: Concurrent operations with proper synchronization
• Comprehensive CLI: Full-featured command-line interface built with Cobra

Documentation

Comprehensive documentation is available at https://TFMV.github.io/flashfs/, including:

• Component overviews
• Command references
• Usage examples
• Implementation details
• Performance considerations

Installation

go install github.com/TFMV/flashfs@latest

Example Usage

Create a snapshot:

flashfs snapshot --path /home/user/documents --output backup-20240601.snap

Compute differences:

flashfs diff --base backup-20240601.snap --target backup-20240602.snap --output changes.diff

Apply a diff:

flashfs apply --base backup-20240601.snap --diff changes.diff --output backup-20240602.snap

Set an expiry policy:

flashfs expiry set --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 12

Known Limitations

• Limited support for special file types (devices, sockets, etc.)
• No built-in encryption for snapshots and diffs
• No remote storage backend integration

Future Roadmap

• Encryption support for secure storage
• Remote storage backends (S3, GCS, etc.)
• GUI for visualization and management
• Improved handling of special file types
• Snapshot verification and integrity checking

Contributors

• Thomas F McGeehan V (Lead Developer)

License

FlashFS is released under the MIT License.

---

We welcome feedback, bug reports, and contributions! Please file issues on our GitHub repository.