Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -82,29 +82,38 @@
verbose: true

msrv:
name: MSRV (1.87)
name: MSRV (1.97.1)
runs-on: ubuntu-latest
timeout-minutes: 20
# Override repo `rust-toolchain.toml` (channel=stable) so this job truly
# exercises MSRV, not latest stable.
env:
RUSTUP_TOOLCHAIN: "1.97.1"
steps:
# actions/checkout@v7.0.0
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
with:
persist-credentials: false

# dtolnay/rust-toolchain — pin to MSRV
- name: Install Rust MSRV (1.87)
# dtolnay/rust-toolchain — pin to MSRV (matches Cargo.toml rust-version)
- name: Install Rust MSRV (1.97.1)
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8
with:
toolchain: "1.87"
toolchain: "1.97.1"
components: clippy, rustfmt

# Swatinem/rust-cache@v2
- name: Cache Cargo + target
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
shared-key: "msrv-v1"
shared-key: "msrv-v1.97.1"

Check warning on line 109 in .github/workflows/ci.yml

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

.github/workflows/ci.yml#L109

A gitleaks generic-api-key was detected which attempts to identify hard-coded credentials.
cache-on-failure: true

- name: Confirm MSRV toolchain is active
run: |
rustc --version | grep -F '1.97.1'
cargo --version

- name: Check formatting
run: cargo fmt --check

Expand Down
56 changes: 45 additions & 11 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,15 +1,49 @@
# Build artifacts
/target
**/*.rs.bk
# AI Tool Local/Ephemeral (no clutter)
.kilo/worktrees/
.kilo/*.json
.devin/cache/
.mimocode/auth.json
.mimocode/plans/
.worktrees/
.swarm/
.beads/
.cline/
.claude/
.codex/
.opencode/
.beads/
docs/superpowers/

# Cargo packaging artifacts (cargo package/publish creates these)
/target/package/
# Compiled output
/target/

# Backup files
/Cargo.lock.bak
# IDE / editor
.idea/
.vscode/
*.swp
*~
.cursor/
.cursorignore
.zed/

# IDE/editor
.mimocode/
# Standard dev + your data
node_modules/
dist/
build/
*.log
lcov.info
.env*

@cubic-dev-ai cubic-dev-ai Bot Aug 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Environment templates such as .env.example and .env.template can no longer be added normally because .env* treats them as secrets. Consider explicit negations for safe template files while continuing to ignore real environment files.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At .gitignore, line 34:

<comment>Environment templates such as `.env.example` and `.env.template` can no longer be added normally because `.env*` treats them as secrets. Consider explicit negations for safe template files while continuing to ignore real environment files.</comment>

<file context>
@@ -1,15 +1,48 @@
+dist/
+build/
+*.log
+.env*
+*.env
+__pycache__/
</file context>
Suggested change
.env*
.env*
!.env.example
!.env.template
Fix with cubic

*.env
__pycache__/
.cache/
.DS_Store
neuromorphic_data/
remotes.txt
Comment thread
coderabbitai[bot] marked this conversation as resolved.

# Nested crate duplicates (from cargo package or accidental clones)
/engram-parser/
# Negations: Force-commit the good stuff
!.kilo/skills/**
!.kilo/tui.jsonc
!.devin/blueprint.yaml
!.mimocode/mimocode.jsonc
!.mimocode/AGENTS.md
!.kilocodeignore
23 changes: 20 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,32 @@ All notable changes to this project are documented in this file.

## [Unreleased]

## [0.2.0] - 2026-08-02

### Added

- GGUF tensor **wire-type** layouts (IQ/Q codes + packed `byte_len` only — **no dequant**).
- `file_type` metadata fallback for quantization label when general quant keys are absent.
- Path-gated **T1** real-GGUF pilots (`tests/real_gguf.rs`) with optional
`ENGRAM_EXPECT_MOE` / `ENGRAM_MOE_SAMPLES`.
- `examples/inspect_gguf` for human inventory of on-disk GGUF files.
- Quality-gate docs in `REVIEW.md` (T0/T1/T2; large MoE RAM budget).
- Local `rust-toolchain.toml` (`channel = "stable"`).
- **GitHub Actions CI** — `fmt`, `clippy`, `build`, and `test` on push/PR to `main`.
- **Boundary documentation** — README scope/ownership section linked to Linear LIM-9.

### Changed

- **Version:** `0.1.0` → **`0.2.0`** (canonical GGUF v3 + MoE extract ship for #7).
- **MSRV:** bumped from 1.87 to **1.97.1** (`Cargo.toml` `rust-version`, CI `msrv` job, Docker `RUST_VERSION`). CI `validate` continues to use latest **stable**.
- **License:** switched from GPL-3.0-or-later to dual MIT/Apache-2.0 for maximum adoption and ecosystem health.
- **Tensor API:** replaced unsafe `as_f32_slice` / `as_u16_bits` with safe `read_f32_values` / `read_u16_values` (allocating `Vec` instead of borrowed slices).
- **`GgufMetadata::quantization()`** returns `String` (owned) so `general.file_type` fallback is derived at call time from the live map. Callers that match on the label should use `.as_str()` or `==`.
- Wire type **31** treated as historical **Q4_0_4_4** (not IQ3_M).

### Added
### Fixed

- **GitHub Actions CI** — `fmt`, `clippy`, `build`, and `test` on push/PR to `main`.
- **Boundary documentation** — README scope/ownership section linked to Linear LIM-9.
- Wire-type 31 labeling aligned with corinth-canal’s GGUF/`ggml_type` table (metadata only).

## [0.1.0] - 2026-06-01

Expand Down
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
[package]
name = "engram-parser"
version = "0.1.0"
version = "0.2.0"
edition = "2024"
rust-version = "1.87"
rust-version = "1.97.1"
description = "Pure-Rust, zero-dependency GGUF deserializer and Mixture-of-Experts per-expert weight extractor. Returns raw byte buffers with shape metadata; performs no neural-network math."
license = "MIT OR Apache-2.0"
authors = ["Raul Montoya Cardenas <montoyaraul34@gmail.com>"]
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
#
# See .github/workflows/docker-build.yml and issue #9 for CI (GHCR on main).

ARG RUST_VERSION=1.87
ARG RUST_VERSION=1.97.1

FROM rust:${RUST_VERSION}-slim

Expand Down
65 changes: 56 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,28 @@ for the full Rust runtime/deployment boundary matrix and
[issue #4](https://github.com/Limen-Neural/engram-parser/issues/4) for
this repo's tracking issue.


## Origin / modularization (#7)

GGUF layout parsing and MoE expert **raw byte** extraction were expanded
using one-way inspiration from the experimental
[`rmems/corinth-canal`](https://github.com/rmems/corinth-canal) reference
implementation (**no** runtime dependency on corinth-canal).

- Tracking: [engram-parser#7](https://github.com/Limen-Neural/engram-parser/issues/7)
- Corinth migration companion: [corinth-canal#115](https://github.com/rmems/corinth-canal/issues/115)
- Cortex coordination: [cortex-tensor#8](https://github.com/Limen-Neural/cortex-tensor/issues/8)
- Linear: [LIM-123](https://linear.app/rpd-34/issue/LIM-123), [LIM-88](https://linear.app/rpd-34/issue/LIM-88)

**GGUF wire types vs “GGML”:** GGUF stores each tensor’s dtype as a
`ggml_type` integer. This crate only maps those codes to labels and packed
**byte sizes** so payloads and MoE slices stay in-range. It does **not**
implement GGML dequant, kernels, or the ggml runtime (that stays
downstream / corinth-canal reference). Wire-type labels follow the
corinth-canal table (e.g. type **31** is historical `Q4_0_4_4`, not the
HuggingFace “IQ3_M” preset). MoE extraction remains free functions
(`list_experts` / `extract_expert`); traits are out of scope for #7.

## Quick start

```rust
Expand All @@ -81,16 +103,28 @@ for (block, expert) in list_experts(&layout) {

## Supported dtypes

Layout-aware parsing: `F32`, `F16`, `BF16` (GGML 30), `Q8_0`, `Q4_K`,
`Q5_K`, `Q6_K`, `IQ3_S` (opaque), plus a `DType::Other(u32)` catch-all.
Layout-aware parsing (**packed byte sizes only — no dequant, no GGML
compute**) for GGUF wire types: `F32`, `F16`, `BF16` (30), `F64`,
`I8`–`I64`, `Q4_0`/`Q4_1`, `Q5_0`/`Q5_1`, `Q8_0`/`Q8_1`, K-quants
`Q2_K`/`Q3_K`/`Q4_K`/`Q5_K`/`Q6_K`/`Q8_K` (no `Q7_K`), and IQ packed
layouts `IQ2_XXS`/`IQ2_XS`/`IQ2_S`, `IQ3_XXS`/`IQ3_S`, `IQ1_S`/`IQ1_M`,
`IQ4_NL`/`IQ4_XS`. Remaining codes use `DType::Other(u32)` (including
historical **wire type 31 = `Q4_0_4_4`**, which is **not** HF “IQ3_M”
and fails closed without a modeled size).

Only `F32` and `F16` have in-crate numeric accessors; everything else
is returned as raw `Vec<u8>`.
is returned as raw `Vec<u8>`. Unknown layouts fail closed at parse time
when element count cannot be converted to a byte length.

`GgufMetadata::quantization()` prefers `general.quantization_type`, then
falls back to `general.file_type` (`0→F32`, `1→F16`, else `GGUF(n)`).

## Public API

`load_gguf`, `parse_bytes`, `GgufLayout`, `GgufMetadata`, `Tensor`,
`DType`, `extract_expert`, `list_experts`, `MoeExpertWeights`,
`RawTensor`, `ParserError`, `Result`.
`DType`, `ggml_type_label`, `extract_expert`, `list_experts`,
`MoeExpertWeights`, `RawTensor`, `ParserError`, `Result`, plus public
`GGML_TYPE_*` and `GGUF_VALUE_TYPE_*` constants.

## Ecosystem / Sibling parsers (LIM-9)

Expand Down Expand Up @@ -122,8 +156,18 @@ cargo test --all-features

# Coverage (local; requires cargo-llvm-cov: cargo install cargo-llvm-cov)
cargo llvm-cov --all-targets --all-features --locked --lcov --output-path lcov.info

# Real GGUF pilots (xai-dissect style; not CI — needs weights on disk)
# Full-file load (no mmap): one ENGRAM_GGUF per process; free RAM ≥ file size + margin
ENGRAM_GGUF=~/.models/gguf/.../model.gguf cargo test --test real_gguf -- --ignored --nocapture
# Large MoE: ENGRAM_EXPECT_MOE=1 ENGRAM_MOE_SAMPLES=3 (see REVIEW.md T1 large MoE)

@cubic-dev-ai cubic-dev-ai Bot Aug 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The documented “Large MoE” invocation never sets these variables for the test that consumes them; the following example ignores both. Showing the variables on a real_gguf_moe test command would make the advertised hard-fail and sampling configuration effective.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At README.md, line 158:

<comment>The documented “Large MoE” invocation never sets these variables for the test that consumes them; the following example ignores both. Showing the variables on a `real_gguf_moe` test command would make the advertised hard-fail and sampling configuration effective.</comment>

<file context>
@@ -122,8 +151,18 @@ cargo test --all-features
+# Real GGUF pilots (xai-dissect style; not CI — needs weights on disk)
+# Full-file load (no mmap): one ENGRAM_GGUF per process; free RAM ≥ file size + margin
+ENGRAM_GGUF=~/.models/gguf/.../model.gguf cargo test --test real_gguf -- --ignored --nocapture
+# Large MoE: ENGRAM_EXPECT_MOE=1 ENGRAM_MOE_SAMPLES=3 (see REVIEW.md T1 large MoE)
+cargo run --example inspect_gguf -- ~/.models/gguf/.../model.gguf

</file context>


</details>

```suggestion
# Large MoE (see REVIEW.md T1 large MoE)
ENGRAM_GGUF=~/.models/gguf/.../model.gguf ENGRAM_EXPECT_MOE=1 ENGRAM_MOE_SAMPLES=3 \
  cargo test --test real_gguf real_gguf_moe -- --ignored --nocapture
Fix with cubic

cargo run --example inspect_gguf -- ~/.models/gguf/.../model.gguf
```

GPU experiments on real models live in **`~/rmems/blackwell-kernel-lab`**
(and production kernels in `myelin-accelerator`) — not as deps of this crate.
See [REVIEW.md](REVIEW.md) for the T0/T1/T2 quality-gate layout.

## Docker

```bash
Expand Down Expand Up @@ -151,13 +195,16 @@ Cross-reference: #11, #8, #9, #7, #5, LIM-9.

## MSRV (Minimum Supported Rust Version)

**MSRV: 1.87**
**MSRV: 1.97.1** (current stable floor as of 2026-08)

This crate guarantees compatibility with Rust 1.87 and later. The MSRV is:
This crate guarantees compatibility with Rust 1.97.1 and later. The MSRV is:

- Declared in `Cargo.toml` via `rust-version = "1.87"`
- Declared in `Cargo.toml` via `rust-version = "1.97.1"`
- Tested in CI on every PR and push (see `msrv` job in `.github/workflows/ci.yml`)
- Verified alongside stable Rust to ensure both toolchains pass all checks
- Verified alongside **stable** (always latest) in the `validate` job so both toolchains pass

Local development defaults to the toolchain in [`rust-toolchain.toml`](rust-toolchain.toml)
(`stable` + `rustfmt` / `clippy`).

**MSRV Policy:**
- MSRV bumps will be documented in release notes
Expand Down
Loading