Skip to content

Releases: pomagrenate/contextsqueezer

Release list

Context Squeezer V1.1.0

Context Squeezer V1.1.0

Choose a tag to compare

View all tags
@pomagrenate pomagrenate released this 01 Mar 06:24
contextsqueezer-v1.1.0
d6af654

Stable

Loading

Context Squeezer V1.0.0

Context Squeezer V1.0.0 Latest
Latest

Choose a tag to compare

View all tags
@pomagrenate pomagrenate released this 20 Feb 15:26
contextsqueezer-v1.0.0
000a072

🚀 Context-Squeezer v1.0.0

Deterministic Context Optimization Engine (No AI-in-the-loop)

🎯 What is Context-Squeezer?

Context-Squeezer is a deterministic, CPU-only context optimization engine designed to:

  • Reduce token usage before sending data to LLMs
  • Improve latency
  • Lower API cost
  • Increase effective context window for local models
  • Preserve structure and anchors
  • Guarantee reproducible output

No embeddings.
No cloud calls.
No AI rewriting.
No hallucinations.

Just structured, deterministic information density optimization.

🔥 v1.0.0 — Production Release

This is the first production-ready release of Context-Squeezer.

It is:

  • ✅ Buildable
  • ✅ Runnable
  • ✅ Deterministic
  • ✅ Test-validated
  • ✅ Hardened
  • ✅ CI-released
  • ✅ Zero binary files in repository
  • ✅ No mock/demo behavior

🏗 Architecture Overview

Go CLI
  ↓
Pipeline Controller
  ↓
C++ Deterministic Compression Core
  ↓
Structured Result + Metadata

Key design principles:

  • Drop-only (no rewriting)
  • Preserve sentence order
  • Anchor-safe pruning
  • Budget-aware compression
  • Cross-chunk deduplication
  • Memory-controlled execution

🧠 What It Actually Does

1️⃣ Sentence segmentation (rule-based)

Fast, deterministic segmentation without heavy NLP dependencies.

2️⃣ Boilerplate & Repeated Block Removal

Removes:

  • Repeated disclaimers
  • Footer/header clones
  • Template blocks

Preserves first occurrence.

3️⃣ Near-Duplicate Sentence Removal

Cosine similarity over TF vectors
Aggressiveness-adjusted thresholds
Cross-chunk deduplication supported

4️⃣ Low-Information Pruning (TF-IDF based)

  • Drops lowest-value sentences

  • Preserves anchors:

    • URLs
    • code fences
    • numeric-heavy lines
    • headings

  • Deterministic scoring

5️⃣ Max Token Budget Controller

--max-tokens N

Guarantees:

  • tokens_out ≤ N (approximate token model)
  • bounded attempts
  • hard truncation fallback (anchor-aware)
  • deterministic output

6️⃣ Supported Formats

  • TXT
  • Markdown
  • HTML
  • DOCX
  • PDF (text-based only; no OCR)

All ingestion happens locally.

No data leaves your machine.

📊 Example

contextsqueeze paper.pdf --max-tokens 8000 --profile api

Output:

  • Reduced tokens
  • Preserved structure
  • Anchors intact
  • Deterministic behavior

📦 Release Artifacts

Artifacts are generated by CI (no binaries stored in repository):

contextsqueeze_linux_amd64
contextsqueeze_darwin_amd64
contextsqueeze_darwin_arm64
SHA256SUMS

Verify:

sha256sum -c SHA256SUMS

🛡 Reliability & Hardening

v1.0.0 includes:

  • Strict exit codes
  • Explicit error paths (no silent failures)
  • Deterministic output
  • Schema version locking
  • Go fuzz tests
  • Sanitizer builds (ASan + UBSan on Linux)
  • Acceptance test suite
  • No binary blobs in git
  • Memory guardrails
  • Bounded loops
  • Crash-proof C ABI boundary

🧾 JSON Output Schema (Stable)

{
  "schema_version": 1,
  "engine_version": "1.0.0",
  "bytes_in": 123456,
  "bytes_out": 65432,
  "tokens_in_approx": 30864,
  "tokens_out_approx": 16358,
  "reduction_pct": 46.98,
  "aggressiveness": 6,
  "profile": "api",
  "budget_applied": true,
  "truncated": false,
  "source_type": "pdf"
}

Schema stability guarantees:

  • No field removals
  • No renames
  • Additive changes only in future versions

📈 Performance

Designed for:

  • 30k–100k token documents
  • Large PDF sections
  • Repeated legal/technical docs
  • Startup cost optimization pipelines
  • Local LLM workflows

No quadratic blowups.
Chunked processing.
Controlled memory growth.

🔒 Security

  • No network calls
  • No telemetry
  • No cloud dependency
  • Pure local execution
  • No binary artifacts in repository
  • Reproducible builds

See SECURITY.md for reporting policy.

⚠️ Known Limitations

  • Token count is approximate (bytes/4 heuristic)
  • Scanned PDFs (image-only) are not supported (no OCR)
  • No semantic rewriting (drop-only model)
  • No exact OpenAI tokenizer support (by design)

🧪 Test Coverage

v1.0.0 includes:

  • C++ unit tests
  • Go unit tests
  • JSON golden tests
  • Acceptance suite
  • Determinism checks
  • Fuzz testing
  • Sanitizer builds

If any gate fails, release is blocked.

🧭 Guarantees

Context-Squeezer v1.0.0 guarantees:

  • Deterministic output for identical input + config
  • No sentence reordering
  • Anchors preserved unless forced truncation
  • MaxTokens honored
  • No silent failure
  • No mock logic

📜 Versioning

This release follows semantic versioning.

v1.0.0 = First production-stable release.

Future updates will:

  • Preserve CLI compatibility
  • Preserve JSON schema compatibility
  • Maintain deterministic guarantees

❤️ Why This Exists

Big models have big context windows.

But:

  • Tokens cost money.
  • Large inputs increase latency.
  • Local machines have memory limits.

Context-Squeezer is not another model.

It is a deterministic context optimizer.

A smart funnel before your LLM.

🧨 Built With Discipline

No hype.
No magic.
No AI-in-the-loop compression.
No black-box summarization.

Just systems engineering.

Assets 2
Loading