Dynamic Filesystem Transformation Layer
A FUSE-based filesystem that provides dynamic filtering, on-the-fly content transformation, and virtual organizational views over existing filesystems—all without copying files or consuming additional storage.
ShadowFS creates a "shadow layer" over your existing filesystems, enabling:
- Dynamic Filtering: Show/hide files based on runtime-configurable rules
- On-the-Fly Transformation: Convert, compress, encrypt, or template files transparently during read
- Virtual Organization: Create multiple directory structures (by type, date, tags) over the same files
- Zero Storage Overhead: Files remain in place—no copying, no duplication
Think of it as "virtual filesystem middleware" that sits between applications and your files, providing programmable views and transformations.
# Install dependencies
pip install fusepy pyyaml jinja2
# Clone and install
git clone https://github.com/andronics/shadowfs.git
cd shadowfs
pip install -e .
# Run tests
pytest tests/ -v
# Mount with configuration
shadowfs --config shadowfs.yaml --mount /mnt/shadowfsHide build artifacts and clutter:
rules:
- name: "Hide build files"
type: exclude
patterns:
- "**/__pycache__/**"
- "**/node_modules/**"
- "**/dist/**"Read Markdown as HTML on-the-fly:
transforms:
- name: "Markdown to HTML"
pattern: "**/*.md"
type: convert
from: markdown
to: htmlOrganize by date, camera, tags simultaneously:
virtual_layers:
- name: by-date
type: date
date_field: ctime
- name: by-camera
type: classifier
classifier: exifCurrent Phase: Phase 3 Complete (Integration - Transforms)
- ✅ Phase 0: Development Infrastructure
- ✅ Phase 1: Foundation Layer (path utilities, file operations, validators)
- ✅ Phase 2: Infrastructure Layer (config, cache, logging, metrics)
- ✅ Phase 3: Integration - Transforms (compression, format conversion, templates)
- 🚧 Phase 4: Integration - Virtual Layers (in progress)
- ⏳ Phase 5: Application Layer (FUSE operations)
- ⏳ Phase 6: Production Readiness
Test Coverage: 187 tests passing with ~97% average coverage on completed phases.
ShadowFS follows a 4-layer architecture compliant with Meta-Architecture v1.0.0:
┌─────────────────────────────────────────┐
│ Layer 4: Application │
│ (FUSE operations, CLI, control API) │
├─────────────────────────────────────────┤
│ Layer 3: Integration │
│ (transforms, rules, virtual layers) │
├─────────────────────────────────────────┤
│ Layer 2: Infrastructure │
│ (config, cache, logging, metrics) │
├─────────────────────────────────────────┤
│ Layer 1: Foundation │
│ (path utils, file ops, validators) │
└─────────────────────────────────────────┘
Key Components:
- Transform Pipeline: Chain multiple transforms (compression → encryption → conversion)
- Rule Engine: Pattern-based file visibility control
- Virtual Layers: Multiple organizational views over same files
- Multi-Level Caching: 3-tier cache (attributes, content, transforms)
shadowfs:
version: "1.0"
sources:
- path: /data/documents
priority: 1
readonly: true
rules:
- name: "Hide hidden files"
type: exclude
pattern: "**/.*"
transforms:
- name: "Auto-decompress"
pattern: "**/*.gz"
type: decompress
algorithm: gzip
virtual_layers:
- name: by-type
type: classifier
classifier: extension
cache:
enabled: true
max_size_mb: 512
ttl_seconds: 300- Compression: gzip, bz2, lzma (compress/decompress)
- Format Conversion: Markdown→HTML, CSV↔JSON, YAML→JSON
- Templates: Jinja2 template rendering with context
- Extensible: Plugin architecture for custom transforms
- Classifier: Organize by extension, size, MIME type
- Date-based: Hierarchical date structures (YYYY/MM/DD)
- Tag-based: Organize by metadata tags
- Pattern-based: Rule-driven classification
- 3-tier caching: Attributes (60s), Content (300s), Transforms (600s)
- Async operations: Thread pool for I/O
- Lazy evaluation: Transforms only applied when accessed
- Connection pooling: Reuse file handles
- CLAUDE.md: Complete project documentation
- PLAN.md: Implementation roadmap and progress
- docs/architecture.md: Technical architecture (Meta-Architecture v1.0.0 compliant)
- docs/virtual-layers.md: Virtual layer system design
- docs/middleware-ideas.md: Future middleware extensions
- docs/typescript-type-discovery.md: Conceptual foundation
# Run all tests
pytest tests/ -v
# Run with coverage
pytest tests/ --cov=shadowfs --cov-report=html
# Run specific test suite
pytest tests/transforms/ -v
pytest tests/integration/ -v
# Run with specific coverage target
pytest tests/transforms/ --cov=shadowfs/transforms --cov-report=term-missingCurrent Test Stats:
- Foundation tests: Coming in Phase 1 completion
- Infrastructure tests: Coming in Phase 2 completion
- Transform tests: 120 tests, ~97% coverage
- Pipeline tests: 34 tests, 99.37% coverage
- Base transform tests: 33 tests, 98.82% coverage
- Python 3.11+
- FUSE (libfuse3-dev on Debian/Ubuntu)
- Optional: markdown, pyyaml, jinja2 (for specific transforms)
# Create virtual environment
python -m venv venv
source venv/bin/activate
# Install development dependencies
pip install -e .
pip install -r requirements-dev.txt
# Install pre-commit hooks
pre-commit install
# Run tests
pytest tests/ -v- Black: Code formatting
- isort: Import sorting
- flake8: Linting
- mypy: Type checking
- pytest: 100% coverage target
- pre-commit: Automated checks
The "shadow filesystem" concept was inspired by TypeScript's .d.ts type discovery system—just as TypeScript creates a type layer over JavaScript, ShadowFS creates organizational and transformation layers over filesystems.
- FUSE: Filesystem in Userspace
- fusepy: Python FUSE bindings
- UnionFS: Union filesystem (view composition inspiration)
- EncFS: Encrypted filesystem (transform inspiration)
- TagFS: Tag-based filesystem (virtual layer inspiration)
Contributions welcome! Please:
- Fork the repository
- Create a feature branch
- Write tests for new functionality
- Ensure all tests pass with good coverage
- Follow code quality standards (black, isort, flake8, mypy)
- Submit a pull request
MIT License - See LICENSE file for details.
Project Maintainer: Stephen Cox (andronics)
Repository: https://github.com/andronics/shadowfs
Status: Design & Implementation Phase Version: 0.1.0-alpha Last Updated: 2025-11-11