Skip to content

MicroLLM v0.1.0 — Calibrated System-1 Probabilistic Decision Engine

Choose a tag to compare

@jmrenouard jmrenouard released this 22 Oct 18:42
· 178 commits to main since this release

MicroLLM v0.1.0 — Calibrated, Sub-Millisecond System-1 Decision Engine for AI Agents

We are thrilled to announce the official v0.1.0 release of MicroLLM, an open-source (Apache-2.0) local probabilistic decision engine designed to eliminate the "System-2 Generative LLM tax" on structured agent routing, classification, compliance verification, and guardrail tasks.


🚀 Key Highlights & Verified Production Metrics

  • 128,497 Evaluated Items / 130,839 Decisions across 9 Benchmark Suites:
    • 100% reproducible on a single laptop CPU thread (AMD Ryzen AI 9 365, 32 GB RAM, 0% GPU / 0% NPU, < 85 MB RSS memory footprint, $0.00 API cost) via microllm benchmark --all-suites (scripts/run_all_benchmarks.py).
  • Exact 0.000000 Option-Order Drift Guarantee:
    • Implements AnyJev $L_0$ log-space circular rotation with a binary-reflected Van der Corput schedule and dynamic early stopping ($\mu(\bar{z}) \ge \tau$).
    • Evaluated on 1,867 external bias audit items (ChoiceBench 1,531 + Callo-Bias 336) and 60 core permutations: 0.000000 Order-Flip Rate and 0.000000 Max Probability Drift while saving 78.3% of rotation passes.
  • 32/32 MUST Specification Compliance:
    • Full adherence to the jevcompat 0.1 standard across all typed decision primitives (noul, choice, score, multi), RFC 8259 float safety, and HTTP 422 error contract.
  • 11,000 Structured Enterprise & Universal Test Cases:
    • 99.75% raw accuracy (10,973 / 11,000) and 100.00% selective accuracy across 18 industry domains (p50 = 0.076–0.190 ms, up to 8,601 decisions/sec on CPU). Every ambiguous edge case is caught by conformal abstention (x_abstain = True).
  • 114,018 Unsampled Rows across 16 Public External Datasets:
    • Evaluates complete official splits with zero cherry-picking: Banking77 (98.74% selective acc), Yelp Full 50k (99.97%), AG News (99.51%), Amazon Reviews (99.24%), RouteLLM (92.60%), MultiNLI, BoolQ, CLINC150, GoEmotions, Reuters, WildGuardMix, and ToxicChat. Overall selective accuracy: 87.49% (p50 = 0.369–0.938 ms).
  • Mathematical Calibration & Conformal Safety:
    • Two-Stage OpenJev-RLCD² (Stage 1 Pathwise Brier with Beta(8,2) difficulty weighting + Stage 2 Leave-One-Out REINFORCE with KL Anchor) reducing Expected Calibration Error from 15.85% down to 6.15% (3.67% on G5).
    • Split Conformal Prediction ($C_\alpha(x)$) with 98.75% empirical coverage at $1 - \alpha = 90%$.

🛠 Features & Architecture

  1. Dual-Gear StartLux Cascade:
    • Gear 1 (FastEncoderBackend): Sub-millisecond CPU NLI cross-encoder with multi-resolution BLAKE2b hashing, clause-salient max-pooling, and per-$K$ temperature scaling.
    • Gear 2 (AnyJevPrefillBackend $L_0$): Algebraic option-bias cancellation over cyclic permutations.
    • Cascade Gating: Automatic fast auto-commit on high confidence; smooth escalation ($p = 0.25 p_0 + 0.75 p_1$) or hierarchical 25-way tournament up to 255 options.
  2. Interactive Decision Studio Web UI:
    • Hosted at http://127.0.0.1:8000/ (microllm serve): visual question builder, token attribution heatmaps, live AnyJev order-flip audit, code export (Python SDK, TypeScript, cURL), 4-Rubric Guardrail shield, and RLCD² calibration curves.
  3. Developer Experience:
    • Typed Python SDK: MicroJevClient with type-safe Noul, Choice, Score, and Multi wrappers.
    • Native Model Context Protocol (MCP): JSON-RPC 2.0 server over stdio (microllm mcp) and HTTP (POST /v1/mcp).
    • LightPath Presentation Deck: 12-slide interactive widescreen deck (docs/slides.html / http://127.0.0.1:8000/slides).
  4. Complete Transparent Governance:
    • Living Action Logbook (docs/ACTION_LOGBOOK.md), RCA Register of errors, dead-ends, approximations, and ideas (docs/RCA_REGISTER.md), Algorithmic Reference (docs/ALGORITHMS.md), and Engineering Syllabus (docs/SYLLABUS.md).

📦 Getting Started

# Install via pip
pip install microllm

# Pull the lightweight ONNX INT8 CPU profile (~58 MB)
microllm pull fast-cpu-int8

# Run all 9 reproducible benchmark suites (128,497 items)
microllm benchmark --all-suites

# Launch the FastAPI server & Decision Studio UI
microllm serve

Full Documentation: https://github.com/jmrenouard/MicroLLM

What's Changed

  • chore(rules): add mandatory GitHub Issue/Branch/PR/Merge project rules (#1) by @jmrenouard in #6
  • ci(tests): add GitHub Actions workflow for automated unit tests (#2) by @jmrenouard in #8
  • fix(ci): add environment.yml and fix Conda GitHub Actions workflow (#7) by @jmrenouard in #9
  • docs(governance): inscrire le Journal de Bord et le Registre RCA alimentés à chaque demande/changement/résultat (#5) by @jmrenouard in #11
  • ci: implémenter toutes les GitHub Actions nécessaires pour un projet Python (#4) by @jmrenouard in #12
  • ci(pypi): Add GitHub Actions for PyPI publication & packaging metadata (#3) by @jmrenouard in #23
  • docs(governance): log ACT-011 and RCA-009 for Conda CI fix (#7) by @jmrenouard in #30
  • docs(governance): rename and translate Action Logbook and RCA Register to English (#29) by @jmrenouard in #31
  • fix(ci): support Python 3.10 in test_packaging_ci.py (#33) by @jmrenouard in #34
  • fix(docs): replace raw LaTeX delimiters with native HTML/CSS and Unicode math (#35) by @jmrenouard in #36
  • feat(benchmark): 540-item 5-track evaluation corpus & 1-command benchmark runner (#37) by @jmrenouard in #38
  • feat(ui): add Obsidian Craft SVG logo & hero header banner across docs and Studio (#40) by @jmrenouard in #41
  • docs(readme): add 'Why MicroLLM?' section and top-of-README 1-command benchmark table (#44) by @jmrenouard in #45
  • feat(roadmap): 20 high-impact architectural & engineering improvements + English project rules (#10) by @jmrenouard in #39
  • fix(docs): ensure strict UTF-8 encoding in ACTION_LOGBOOK.md and RCA_REGISTER.md (#10) by @jmrenouard in #46
  • docs(governance): merge docs/ROADMAP_20_IDEAS.md into docs/RCA_REGISTER.md (#43) by @jmrenouard in #47
  • test(quality): Complete Test Plan & 750–1,000 Case Performance/Quality Validation Suite (#32) by @jmrenouard in #49
  • docs(benchmark): add Section 13 to docs/index.html and sync <3ms CPU latency (#44) by @jmrenouard in #53
  • fix(ci): resolve Mypy union-attr errors and Bandit B615 revision pinning (#48) by @jmrenouard in #54
  • feat: build 500-to-1,000 structured test case entries and publish evaluation metrics (#52) by @jmrenouard in #55
  • docs(papers): 104-paper Jev, RLCD & System-1 academic literature review and /v1/papers catalog (#42) by @jmrenouard in #57
  • feat(brand): colorful System-2 herd vs MicroLLM logo header banner (#50) by @jmrenouard in #60
  • feat: add 10,000 product-independent test cases (11,000 total) and publish metrics (#58) by @jmrenouard in #62
  • chore(deps): integrate 15 Dependabot PRs for GitHub Actions and Python dependencies (#51) by @jmrenouard in #59
  • docs: add BENCHMARKS.md with published evaluation results and link from README.md (#63) by @jmrenouard in #64
  • feat(benchmarks): Comprehensive Web Dataset Survey & Adapters for MicroLLM Validation (#56) by @jmrenouard in #66
  • feat(papers): download all 104 arXiv PDF documents locally & add CLI --download-pdfs / --verify-pdfs (#67) by @jmrenouard in #68
  • feat(rca): automated sequential RCA non-regression verification runner & CLI (#65) by @jmrenouard in #69
  • feat(research): challenge code/design against research papers and enrich RCA Register with 12 hypotheses & validation tests (#61) by @jmrenouard in #71
  • docs: write comprehensive Product Documentation and Technical Specifications (#70) by @jmrenouard in #72
  • chore: enforce English-only rule for all project documentation and code (#74) by @jmrenouard in #75
  • feat(benchmarks): implement 16 external dataset adapters and benchmark suite (#73) by @jmrenouard in #76
  • docs(benchmarks): document complete benchmark procedure and hardware resource profile (#77) by @jmrenouard in #78
  • fix(ci): resolve Bandit B108 tempfile warning and Mypy assignment error (#79) by @jmrenouard in #80
  • feat(benchmarks): run full unsampled 114,018-row external datasets benchmark and document results (#81) by @jmrenouard in #82
  • fix: harden full-split external benchmark loader, adapters, CLI/API flags, and docs (#83) by @jmrenouard in #84
  • docs: add comprehensive results communication kit for executive and technical audiences (#85) by @jmrenouard in #86
  • feat: implement 4 priority roadmap developments & re-benchmark 114,018 external rows (#87) by @jmrenouard in #88
  • fix: resolve mypy/bandit CI failures and harden priority 1-4 developments (#89) by @jmrenouard in #90
  • fix: use str.maketrans two-string overload for mypy compatibility (#89) by @jmrenouard in #91
  • docs: add LightPath-branded interactive 12-slide presentation deck (#93) by @jmrenouard in #94
  • feat: add unified 9-suite benchmark automation runner and refresh all benchmarks (#92) by @jmrenouard in #95
  • docs: document worst-case benchmarks, failure modes, and structural limits (#96) by @jmrenouard in #97
  • fix: harden unified 9-suite benchmark runner and Parquet cache validation (#92) by @jmrenouard in #99
  • docs: specify core algorithms manual and comprehensive syllabus (#98) by @jmrenouard in #100
  • chore: configure github repository topics (#101) by @jmrenouard in #102
  • feat: add decision studio screencast gif to readme and prepare v0.1.0 release (#103) by @jmrenouard in #104

New Contributors

Full Changelog: https://github.com/jmrenouard/MicroLLM/commits/v0.1.0

What's Changed

  • chore(rules): add mandatory GitHub Issue/Branch/PR/Merge project rules (#1) by @jmrenouard in #6
  • ci(tests): add GitHub Actions workflow for automated unit tests (#2) by @jmrenouard in #8
  • fix(ci): add environment.yml and fix Conda GitHub Actions workflow (#7) by @jmrenouard in #9
  • docs(governance): inscrire le Journal de Bord et le Registre RCA alimentés à chaque demande/changement/résultat (#5) by @jmrenouard in #11
  • ci: implémenter toutes les GitHub Actions nécessaires pour un projet Python (#4) by @jmrenouard in #12
  • ci(pypi): Add GitHub Actions for PyPI publication & packaging metadata (#3) by @jmrenouard in #23
  • docs(governance): log ACT-011 and RCA-009 for Conda CI fix (#7) by @jmrenouard in #30
  • docs(governance): rename and translate Action Logbook and RCA Register to English (#29) by @jmrenouard in #31
  • fix(ci): support Python 3.10 in test_packaging_ci.py (#33) by @jmrenouard in #34
  • fix(docs): replace raw LaTeX delimiters with native HTML/CSS and Unicode math (#35) by @jmrenouard in #36
  • feat(benchmark): 540-item 5-track evaluation corpus & 1-command benchmark runner (#37) by @jmrenouard in #38
  • feat(ui): add Obsidian Craft SVG logo & hero header banner across docs and Studio (#40) by @jmrenouard in #41
  • docs(readme): add 'Why MicroLLM?' section and top-of-README 1-command benchmark table (#44) by @jmrenouard in #45
  • feat(roadmap): 20 high-impact architectural & engineering improvements + English project rules (#10) by @jmrenouard in #39
  • fix(docs): ensure strict UTF-8 encoding in ACTION_LOGBOOK.md and RCA_REGISTER.md (#10) by @jmrenouard in #46
  • docs(governance): merge docs/ROADMAP_20_IDEAS.md into docs/RCA_REGISTER.md (#43) by @jmrenouard in #47
  • test(quality): Complete Test Plan & 750–1,000 Case Performance/Quality Validation Suite (#32) by @jmrenouard in #49
  • docs(benchmark): add Section 13 to docs/index.html and sync <3ms CPU latency (#44) by @jmrenouard in #53
  • fix(ci): resolve Mypy union-attr errors and Bandit B615 revision pinning (#48) by @jmrenouard in #54
  • feat: build 500-to-1,000 structured test case entries and publish evaluation metrics (#52) by @jmrenouard in #55
  • docs(papers): 104-paper Jev, RLCD & System-1 academic literature review and /v1/papers catalog (#42) by @jmrenouard in #57
  • feat(brand): colorful System-2 herd vs MicroLLM logo header banner (#50) by @jmrenouard in #60
  • feat: add 10,000 product-independent test cases (11,000 total) and publish metrics (#58) by @jmrenouard in #62
  • chore(deps): integrate 15 Dependabot PRs for GitHub Actions and Python dependencies (#51) by @jmrenouard in #59
  • docs: add BENCHMARKS.md with published evaluation results and link from README.md (#63) by @jmrenouard in #64
  • feat(benchmarks): Comprehensive Web Dataset Survey & Adapters for MicroLLM Validation (#56) by @jmrenouard in #66
  • feat(papers): download all 104 arXiv PDF documents locally & add CLI --download-pdfs / --verify-pdfs (#67) by @jmrenouard in #68
  • feat(rca): automated sequential RCA non-regression verification runner & CLI (#65) by @jmrenouard in #69
  • feat(research): challenge code/design against research papers and enrich RCA Register with 12 hypotheses & validation tests (#61) by @jmrenouard in #71
  • docs: write comprehensive Product Documentation and Technical Specifications (#70) by @jmrenouard in #72
  • chore: enforce English-only rule for all project documentation and code (#74) by @jmrenouard in #75
  • feat(benchmarks): implement 16 external dataset adapters and benchmark suite (#73) by @jmrenouard in #76
  • docs(benchmarks): document complete benchmark procedure and hardware resource profile (#77) by @jmrenouard in #78
  • fix(ci): resolve Bandit B108 tempfile warning and Mypy assignment error (#79) by @jmrenouard in #80
  • feat(benchmarks): run full unsampled 114,018-row external datasets benchmark and document results (#81) by @jmrenouard in #82
  • fix: harden full-split external benchmark loader, adapters, CLI/API flags, and docs (#83) by @jmrenouard in #84
  • docs: add comprehensive results communication kit for executive and technical audiences (#85) by @jmrenouard in #86
  • feat: implement 4 priority roadmap developments & re-benchmark 114,018 external rows (#87) by @jmrenouard in #88
  • fix: resolve mypy/bandit CI failures and harden priority 1-4 developments (#89) by @jmrenouard in #90
  • fix: use str.maketrans two-string overload for mypy compatibility (#89) by @jmrenouard in #91
  • docs: add LightPath-branded interactive 12-slide presentation deck (#93) by @jmrenouard in #94
  • feat: add unified 9-suite benchmark automation runner and refresh all benchmarks (#92) by @jmrenouard in #95
  • docs: document worst-case benchmarks, failure modes, and structural limits (#96) by @jmrenouard in #97
  • fix: harden unified 9-suite benchmark runner and Parquet cache validation (#92) by @jmrenouard in #99
  • docs: specify core algorithms manual and comprehensive syllabus (#98) by @jmrenouard in #100
  • chore: configure github repository topics (#101) by @jmrenouard in #102
  • feat: add decision studio screencast gif to readme and prepare v0.1.0 release (#103) by @jmrenouard in #104
  • feat: add untruncated multi-view high-res decision studio screenshots to readme and fix css grid layout (#105) by @jmrenouard in #106
  • docs: add official Hugging Face Spaces and Benchmark Dataset badges and links (#107) by @jmrenouard in #109
  • feat: enrich external datasets and adapters from Hugging Face JEV models ecosystem (#108) by @jmrenouard in #111
  • fix: resolve CI pytest-cov RCA-012 latency SLA failure and PyPI OIDC pre-flight (#110) by @jmrenouard in #113

New Contributors

Full Changelog: https://github.com/jmrenouard/MicroLLM/commits/v0.1.0

What's Changed

  • chore(rules): add mandatory GitHub Issue/Branch/PR/Merge project rules (#1) by @jmrenouard in #6
  • ci(tests): add GitHub Actions workflow for automated unit tests (#2) by @jmrenouard in #8
  • fix(ci): add environment.yml and fix Conda GitHub Actions workflow (#7) by @jmrenouard in #9
  • docs(governance): inscrire le Journal de Bord et le Registre RCA alimentés à chaque demande/changement/résultat (#5) by @jmrenouard in #11
  • ci: implémenter toutes les GitHub Actions nécessaires pour un projet Python (#4) by @jmrenouard in #12
  • ci(pypi): Add GitHub Actions for PyPI publication & packaging metadata (#3) by @jmrenouard in #23
  • docs(governance): log ACT-011 and RCA-009 for Conda CI fix (#7) by @jmrenouard in #30
  • docs(governance): rename and translate Action Logbook and RCA Register to English (#29) by @jmrenouard in #31
  • fix(ci): support Python 3.10 in test_packaging_ci.py (#33) by @jmrenouard in #34
  • fix(docs): replace raw LaTeX delimiters with native HTML/CSS and Unicode math (#35) by @jmrenouard in #36
  • feat(benchmark): 540-item 5-track evaluation corpus & 1-command benchmark runner (#37) by @jmrenouard in #38
  • feat(ui): add Obsidian Craft SVG logo & hero header banner across docs and Studio (#40) by @jmrenouard in #41
  • docs(readme): add 'Why MicroLLM?' section and top-of-README 1-command benchmark table (#44) by @jmrenouard in #45
  • feat(roadmap): 20 high-impact architectural & engineering improvements + English project rules (#10) by @jmrenouard in #39
  • fix(docs): ensure strict UTF-8 encoding in ACTION_LOGBOOK.md and RCA_REGISTER.md (#10) by @jmrenouard in #46
  • docs(governance): merge docs/ROADMAP_20_IDEAS.md into docs/RCA_REGISTER.md (#43) by @jmrenouard in #47
  • test(quality): Complete Test Plan & 750–1,000 Case Performance/Quality Validation Suite (#32) by @jmrenouard in #49
  • docs(benchmark): add Section 13 to docs/index.html and sync <3ms CPU latency (#44) by @jmrenouard in #53
  • fix(ci): resolve Mypy union-attr errors and Bandit B615 revision pinning (#48) by @jmrenouard in #54
  • feat: build 500-to-1,000 structured test case entries and publish evaluation metrics (#52) by @jmrenouard in #55
  • docs(papers): 104-paper Jev, RLCD & System-1 academic literature review and /v1/papers catalog (#42) by @jmrenouard in #57
  • feat(brand): colorful System-2 herd vs MicroLLM logo header banner (#50) by @jmrenouard in #60
  • feat: add 10,000 product-independent test cases (11,000 total) and publish metrics (#58) by @jmrenouard in #62
  • chore(deps): integrate 15 Dependabot PRs for GitHub Actions and Python dependencies (#51) by @jmrenouard in #59
  • docs: add BENCHMARKS.md with published evaluation results and link from README.md (#63) by @jmrenouard in #64
  • feat(benchmarks): Comprehensive Web Dataset Survey & Adapters for MicroLLM Validation (#56) by @jmrenouard in #66
  • feat(papers): download all 104 arXiv PDF documents locally & add CLI --download-pdfs / --verify-pdfs (#67) by @jmrenouard in #68
  • feat(rca): automated sequential RCA non-regression verification runner & CLI (#65) by @jmrenouard in #69
  • feat(research): challenge code/design against research papers and enrich RCA Register with 12 hypotheses & validation tests (#61) by @jmrenouard in #71
  • docs: write comprehensive Product Documentation and Technical Specifications (#70) by @jmrenouard in #72
  • chore: enforce English-only rule for all project documentation and code (#74) by @jmrenouard in #75
  • feat(benchmarks): implement 16 external dataset adapters and benchmark suite (#73) by @jmrenouard in #76
  • docs(benchmarks): document complete benchmark procedure and hardware resource profile (#77) by @jmrenouard in #78
  • fix(ci): resolve Bandit B108 tempfile warning and Mypy assignment error (#79) by @jmrenouard in #80
  • feat(benchmarks): run full unsampled 114,018-row external datasets benchmark and document results (#81) by @jmrenouard in #82
  • fix: harden full-split external benchmark loader, adapters, CLI/API flags, and docs (#83) by @jmrenouard in #84
  • docs: add comprehensive results communication kit for executive and technical audiences (#85) by @jmrenouard in #86
  • feat: implement 4 priority roadmap developments & re-benchmark 114,018 external rows (#87) by @jmrenouard in #88
  • fix: resolve mypy/bandit CI failures and harden priority 1-4 developments (#89) by @jmrenouard in #90
  • fix: use str.maketrans two-string overload for mypy compatibility (#89) by @jmrenouard in #91
  • docs: add LightPath-branded interactive 12-slide presentation deck (#93) by @jmrenouard in #94
  • feat: add unified 9-suite benchmark automation runner and refresh all benchmarks (#92) by @jmrenouard in #95
  • docs: document worst-case benchmarks, failure modes, and structural limits (#96) by @jmrenouard in #97
  • fix: harden unified 9-suite benchmark runner and Parquet cache validation (#92) by @jmrenouard in #99
  • docs: specify core algorithms manual and comprehensive syllabus (#98) by @jmrenouard in #100
  • chore: configure github repository topics (#101) by @jmrenouard in #102
  • feat: add decision studio screencast gif to readme and prepare v0.1.0 release (#103) by @jmrenouard in #104
  • feat: add untruncated multi-view high-res decision studio screenshots to readme and fix css grid layout (#105) by @jmrenouard in #106
  • docs: add official Hugging Face Spaces and Benchmark Dataset badges and links (#107) by @jmrenouard in #109
  • feat: enrich external datasets and adapters from Hugging Face JEV models ecosystem (#108) by @jmrenouard in #111
  • fix: resolve CI pytest-cov RCA-012 latency SLA failure and PyPI OIDC pre-flight (#110) by @jmrenouard in #113
  • feat: integrate HF JEV model corpora into unified benchmark orchestrator and execution scripts (#114) by @jmrenouard in #115
  • feat: publish and integrate the 4 official MicroLLM Hugging Face models (#112) by @jmrenouard in #116
  • ci: support PIPY_API_TOKEN and PYPI_API_TOKEN secrets for automated PyPI publishing (#118) by @jmrenouard in #119

New Contributors

Full Changelog: https://github.com/jmrenouard/MicroLLM/commits/v0.1.0

Full Changelog: https://github.com/jmrenouard/MicroLLM/commits/v0.1.0