Content-Aware Compression Framework (CACF)

Lossless archival compression orchestrator: scan a folder, classify files, estimate compressibility, choose a reversible codec per file/chunk, optionally deduplicate, and store everything in a custom .cacf container with exact reconstruction.

This is a research MVP — an orchestration layer over zstd / brotli / lzma / raw store, not a new universal compressor.

Install

cd "content-aware compression framework"
python -m pip install -e ".[dev]"

Requires Python 3.11+.

Easiest way to compress a folder

Option A — one command (recommended)

python -m cacf compress --input "C:\Users\You\Documents\MyFolder" --output-dir "C:\Users\You\Archives"

This writes C:\Users\You\Archives\MyFolder.cacf automatically (named after your input folder).

Option B — interactive prompts

python -m cacf compress

You will be asked:

1. Input folder to compress
2. Output folder for the archive

Paste paths from File Explorer (quotes are fine).

Option C — Windows double-click

Run compress.bat in this project folder, then type/paste the two paths when prompted.
Or from a terminal:

compress.bat "C:\path\to\input" "C:\path\to\output_folder"

Restore later

Option A — restore to any folder

python -m cacf restore --archive "C:\Users\You\Archives\MyFolder.cacf" --output "C:\Users\You\Restored\MyFolder"

Option B — restore to the original folder (path saved inside the archive)

python -m cacf restore --archive "C:\Users\You\Archives\MyFolder.cacf" --to-original

Option C — interactive prompts

python -m cacf restore

You will be asked for the .cacf path, then whether to use the original folder or choose another.

Option D — Windows double-click

Run restore.bat, or:

restore.bat "C:\path\to\file.cacf" "C:\path\to\output_folder"
restore.bat "C:\path\to\file.cacf" --to-original

By default, restore also checks SHA-256 hashes after extraction (--no-verify to skip).

Verify against the original source

python -m cacf verify --archive "C:\Users\You\Archives\MyFolder.cacf" --source "C:\Users\You\Documents\MyFolder"

Quick start (sample data)

python scripts/generate_testdata.py

python -m cacf compress --input testdata/mixed --output-dir archives
python -m cacf list archives/mixed.cacf
python -m cacf restore --archive archives/mixed.cacf --output restored/
python -m cacf verify --archive archives/mixed.cacf --source testdata/mixed

Benchmark:

python -m cacf benchmark --input testdata/text_code --output report.json --codec-profile text-heavy

CLI options

Flag / command	Values	Default
compress --input	folder to compress	prompted if omitted
compress --output-dir	folder for the .cacf	prompted if omitted
restore --archive	.cacf file	prompted if omitted
restore --output	folder to extract into	prompted if omitted
restore --to-original	restore to recorded source path	off
restore --verify / --no-verify	hash-check after extract	verify on
create --input / --output	folder / archive file path	required
--archive	.cacf path	required for list/verify; optional for extract/restore
--source	original folder	required for verify
--codec-profile	auto | text-heavy | mixed | media-heavy | max-effort	auto
--chunk-size	bytes	1048576 (1 MiB)
--enable-dedupe / --no-enable-dedupe		dedupe on
--enable-benchmark / --no-enable-benchmark	create only	off
--manifest-format	json | sqlite	json
--log-level	info | debug	info

Pipeline

1. Scanner — recursive directory walk (relative paths, size, mtime).
2. Analyzer — extension, magic bytes, SHA-256, Shannon entropy sample, type class.
3. Planner — rule-based codec choice (no ML in MVP).
4. Dedupe — file-level hash reuse; fixed-size chunk table with content-hash reuse.
5. Codecs — raw, zstd, brotli, lzma (all lossless).
6. Container — .cacf with JSON or SQLite manifest + binary payload + footer hash.
7. Extract / verify — exact byte restore; SHA-256 checks against originals.

Planner heuristics (summary)

Signal	Choice
High entropy / media / archives (jpg, png, mp4, zip, …)	store raw (auto / media-heavy)
Text / code, moderate–low entropy	brotli
Documents / mixed binaries	zstd
Large low-entropy blobs	lzma
Same file SHA-256 (dedupe on)	reference existing chunks
Compression not beneficial for a chunk	fall back to raw
Profile max-effort	try zstd-19, brotli-11, lzma-9 per chunk; keep the smallest if ≥1% smaller, else raw

Profiles only bias entropy thresholds (text-heavy favors brotli; media-heavy favors raw sooner).
max-effort is slower and uses more CPU; it still cannot shrink already-compressed PNGs/JPEGs/MP4s by large factors.

Every file stores a human-readable reason string — visible in cacf list and benchmark reports.

Realistic size expectations (lossless)

CACF is bit-exact lossless. That means:

Folder type	Typical archive size vs original
Text, code, logs, CSV/JSON, duplicates	Often much smaller (sometimes 5–20×)
Mixed office + text	Moderate gains
PNG / JPEG / MP4 / MP3 / ZIP / already-compressed	Usually ~same size (± a few %)

Example: a ~39 MB folder of PNG art images will stay ~39 MB even with --codec-profile max-effort. Getting that down to 3–4 MB would require lossy image encoding, which this tool deliberately does not do.

Max-effort usage:

python -m cacf compress --input "C:\path\to\folder" --output-dir "C:\path\to\out" --codec-profile max-effort

Container format (.cacf)

[8-byte magic "CACF0001"]
[4-byte flags]              # bit0 = SQLite manifest
[8-byte manifest length]
[manifest bytes]            # JSON or SQLite DB blob
[8-byte payload length]
[payload bytes]             # concatenated compressed/raw chunks
[32-byte SHA-256(manifest||payload)]

Integrity: per-file and per-chunk SHA-256 in the manifest, plus the archive footer hash.

Sample datasets

scripts/generate_testdata.py creates:

• testdata/text_code/ — markdown, JSON, Python, logs
• testdata/documents/ — PDF-like, CSV, XML, office-like ZIP
• testdata/media/ — PNG, JPEG-like, MP4-like, MP3-like
• testdata/mixed/ — combination
• testdata/duplicates/ — identical payloads under different paths

Tests

python -m pytest -q

Layout

cacf/
  scanner/    analyzer/    planner/
  codecs/     dedupe/      container/
  benchmark/  cli.py       models.py
  ml/         # placeholder for future optional ML planner

Out of scope (MVP)

• Rolling-hash chunking
• Shared dictionaries / reversible preprocessing
• ML compressibility prediction (stub under cacf/ml/)

License

MIT