Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
abfab61
Enhance CI workflow with caching and stale issue actions
rmems Jul 1, 2026
86fdf17
ci: harden GitHub Actions per #11 (Codecov https://about.codecov.io/l…
rmems Jul 4, 2026
2d43222
feat: combine #9 (Docker workflow + GHCR) and #11 (harden GH Actions …
rmems Jul 4, 2026
bfece09
fix: address Devin Review bugs in Docker/CI config
devin-ai-integration[bot] Jul 7, 2026
2af3410
merge: resolve conflicts with main (keep main's Rust refactors + PR's…
devin-ai-integration[bot] Jul 7, 2026
a5fee4a
fix: add docker/setup-buildx-action for GHA cache support
devin-ai-integration[bot] Jul 7, 2026
d7674bf
fix: bump RUST_VERSION to 1.87 (matches MSRV) + lowercase Docker tags
devin-ai-integration[bot] Jul 7, 2026
53e94ad
fix: make Codecov upload non-blocking (repo may not be configured yet)
devin-ai-integration[bot] Jul 7, 2026
ce7aeab
fix: simplify Dockerfile to single-stage (library crate has no binary)
devin-ai-integration[bot] Jul 7, 2026
5a382a4
fix: address remaining bot review comments
rmems Jul 7, 2026
fec8432
fix: address remaining bot review comments
rmems Jul 7, 2026
6cf0e32
Update Dockerfile
rmems Jul 7, 2026
4e90722
fix: add --release to CMD to match build step and avoid recompilation
devin-ai-integration[bot] Jul 7, 2026
fe1bf98
Update .github/workflows/ci.yml
rmems Jul 7, 2026
2c1643e
fix: address remaining PR #17 review comments (overflow, clean-tree, …
rmems Jul 8, 2026
daa9959
fix: address remaining PR #17 bot review comments (round 2)
rmems Jul 8, 2026
87c6c1a
fix: revert cargo fetch (fails without src/ present at COPY stage)
rmems Jul 8, 2026
e2a9394
fix: remove stale gcov parser from codecov config (CI generates lcov)
rmems Jul 8, 2026
eaa90b2
fix: merge publish job back into single Docker build job (no Skipped …
rmems Jul 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Ignore junk for Docker builds (helps with the duplicated layout issue)
engram-parser/
target/
**/.git
**/*.rs.bk
17 changes: 17 additions & 0 deletions .github/codecov.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Minimal Codecov config for Rust (per https://about.codecov.io/language/rust/ and #11 AC)
# We will use codecov https://about.codecov.io/language/rust/
codecov:
require_ci_to_pass: true

coverage:
precision: 2
round: down
range: "70...100"

status:
project: true
patch: true
changes: false

# CI generates lcov format (cargo llvm-cov --lcov); no gcov parser needed.
comment: false
54 changes: 43 additions & 11 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ on:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: '0 0 * * *' # Run daily at midnight

concurrency:
group: ci-cpu-${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
Comment thread
rmems marked this conversation as resolved.

permissions:
contents: read
Expand All @@ -13,6 +19,9 @@ jobs:
validate:
name: Build & Test
runs-on: ubuntu-latest
timeout-minutes: 20
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
steps:
# actions/checkout@v4.2.2
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
Expand All @@ -23,19 +32,14 @@ jobs:
- name: Install Rust stable
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8
with:
components: clippy, rustfmt
components: clippy, rustfmt, llvm-tools-preview

# actions/cache@v4
- name: Cache cargo registry
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830
# Swatinem/rust-cache@v2 (adopted from corinth-canal for better caching)
- name: Cache Cargo + target
uses: Swatinem/rust-cache@42dc69e1aa15d09112580998cf2ef0119e2e91ae # v2
with:
path: |
~/.cargo/registry
~/.cargo/git
target
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-
shared-key: "cpu-ci-v1"
cache-on-failure: true

- name: Check formatting
run: cargo fmt --check
Expand All @@ -48,3 +52,31 @@ jobs:

- name: Test
run: cargo test --all-features
Comment thread
qodo-code-review[bot] marked this conversation as resolved.

# Verify build/test leaves no unexpected artifacts (clean-tree guard per qodo review)
- name: Check working tree is clean
run: |
if [ -n "$(git status --porcelain)" ]; then
echo "❌ Working tree has unexpected changes after build+test:"
git status --short
exit 1
fi
echo "✅ Working tree is clean"

# Coverage using Codecov (per review: "We will use codecov https://about.codecov.io/language/rust/")
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@16b05812d776ae1dfaabc8277e421fb6d2506419 # v2
with:
tool: cargo-llvm-cov

- name: Generate coverage report
run: cargo llvm-cov --all-targets --all-features --locked --lcov --output-path lcov.info

- name: Upload coverage to Codecov
if: ${{ env.CODECOV_TOKEN != '' }}
uses: codecov/codecov-action@04b047e8bb82a0c002c8312c1c880fbc6a999d45 # v5
with:
token: ${{ env.CODECOV_TOKEN }}
files: lcov.info
fail_ci_if_error: false
verbose: true
70 changes: 70 additions & 0 deletions .github/workflows/docker-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
name: Docker Build

on:
pull_request:
branches:
- '**'
push:
branches:
- main

concurrency:
group: docker-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

permissions:
contents: read

jobs:
build:
name: Build Docker Image (CPU-only)
permissions:
contents: read
packages: write
runs-on: ubuntu-latest

steps:
# actions/checkout@v4.2.2
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
persist-credentials: false

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0

- name: Set Docker tags
id: tags
env:
EVENT_NAME: ${{ github.event_name }}
REF: ${{ github.ref }}
PR_NUMBER: ${{ github.event.number }}
COMMIT_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
REPO_RAW: ${{ github.repository }}
run: |
REPO="ghcr.io/$(echo "$REPO_RAW" | tr '[:upper:]' '[:lower:]')"
TAGS="${REPO}:${COMMIT_SHA}"
if [ "$EVENT_NAME" = "push" ] && [ "$REF" = "refs/heads/main" ]; then
TAGS="$TAGS,${REPO}:main"
fi
if [ "$EVENT_NAME" = "pull_request" ]; then
TAGS="$TAGS,${REPO}:pr-${PR_NUMBER}"
fi
echo "tags=$TAGS" >> $GITHUB_OUTPUT

- name: Login to GitHub Container Registry
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build Docker Image
uses: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25 # v5
with:
context: .
push: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
tags: ${{ steps.tags.outputs.tags }}
# No build-args needed (CPU-only, stable Rust in Dockerfile)
cache-from: type=gha
cache-to: type=gha,mode=max
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
/target
/Cargo.lock.bak
**/*.rs.bk
.mimocode/
engram-parser/
40 changes: 40 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# syntax=docker/dockerfile:1.4
#
# Dockerfile for engram-parser (pure-Rust, zero-dep GGUF/MoE parser).
# Single-stage build for CI verification and reproducible builds.
#
# engram-parser is a library crate with no binary target, so there is no
# standalone artifact to deploy to a runtime image. This image is used for:
# - CI build/test verification
# - Reproducible build environment
# - Base image for downstream crates that depend on engram-parser
#
# Usage:
# docker build -t engram-parser .
# docker run --rm engram-parser cargo test --all-features
#
# See .github/workflows/docker-build.yml and issue #9 for CI (GHCR on main).

ARG RUST_VERSION=1.87

FROM rust:${RUST_VERSION}-slim

RUN useradd -m -u 10001 appuser

WORKDIR /app

# Copy manifests and lock file for reproducibility
COPY Cargo.toml Cargo.lock ./

# Copy source
COPY . .
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
Comment on lines +27 to +30

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Dockerfile COPY pattern defeats dependency layer caching

The Dockerfile copies manifests first (COPY Cargo.toml Cargo.lock ./ at line 27), then immediately copies everything (COPY . . at line 30) before any cargo build. The typical Docker caching pattern is: copy manifests → run dependency build → copy source → run full build. Since there's no build step between the two COPYs, the first COPY provides no caching benefit — any source change invalidates the COPY . . layer and triggers a full rebuild anyway. For a zero-dependency crate this is moot today, but the comment "Copy manifests and lock file for reproducibility" is misleading about the actual purpose. If dependencies are ever added, this pattern would need restructuring to actually cache them.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged — the separate COPY Cargo.toml/Cargo.lock before COPY . . is kept for organizational clarity and forward compatibility. This crate has zero deps today so there is no caching benefit either way. Adding RUN cargo fetch between the COPYs was attempted but failed because cargo requires a target (src/lib.rs) to parse Cargo.toml. The standard Docker dummy-source pattern was rejected for simplicity in the single-stage design (ce7aeab).

— Cline agent: DeepSeek-v4-pro


# Build and test the crate (zero external deps, no system packages needed)
RUN cargo build --release --all-features && \
cargo test --release --all-features

RUN chown -R appuser:appuser /app

USER appuser
Comment thread
rmems marked this conversation as resolved.

CMD ["cargo", "test", "--release", "--all-features"]
48 changes: 48 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,14 +92,62 @@ is returned as raw `Vec<u8>`.
`DType`, `extract_expert`, `list_experts`, `MoeExpertWeights`,
`RawTensor`, `ParserError`, `Result`.

## Ecosystem / Sibling parsers (LIM-9)

- **engram-parser** (this crate): canonical zero-dep GGUF v3 deserializer + per-expert MoE raw weight ripper.
- Safetensors extraction (header inspection, deterministic manifest, MoE router/expert candidate discovery via classify + groups + layout families) from `rmems/corinth-canal` (experimental source of inspiration) is tracked as a **separate issue** in this repo: #10 (parallel to the GGUF work in #7).
- Source-side bootstrap/supporting: rmems/corinth-canal#116.
- Coordination for consumers (e.g. future multi-format in cortex): Limen-Neural/cortex-tensor#9.
- The reusable implementation will target a dedicated Limen-Neural crate (per org boundary matrix LIM-9); engram-parser charter remains GGUF-only.
- **Clarification**: one-way extraction/copy of code from inspiration. We are not adding any dependency from corinth-canal. corinth-canal keeps an unmodified reference copy (per its PROMOTION_RULES "frozen" status). See #10, #7, and the plan for full cross-links and "no dep on corinth-canal" language.

Cross-links and updates performed when #10 was created.

## Development

This is a pure-Rust, zero-dependency crate. Build, lint, and test commands use `--all-features`.

```bash
# Format
cargo fmt --check

# Lint (fail on warnings)
cargo clippy --all-targets --all-features -- -D warnings

# Build
cargo build --all-features

# Test
cargo test --all-features

# Coverage (local; requires cargo-llvm-cov: cargo install cargo-llvm-cov)
cargo llvm-cov --all-targets --all-features --locked --lcov --output-path lcov.info
```

## Docker

```bash
# Build the image locally (includes build + test verification)
docker build -t engram-parser .

# Run tests in the container
docker run --rm engram-parser

# Pull from GHCR (published on merges to main)
docker pull ghcr.io/limen-neural/engram-parser:main
```
Comment thread
coderabbitai[bot] marked this conversation as resolved.

## CI

- GitHub Actions: `.github/workflows/ci.yml` (hardened via #11; uses Codecov per <https://about.codecov.io/language/rust/>)
- Azure Pipelines: `azure-pipelines.yml` (tracked in #8 for cross-platform ubuntu/mac/windows)
- Docker: `Dockerfile` + `.github/workflows/docker-build.yml` (tracked in #9 for GHCR reproducible builds; use user's Docker CLI for local verification)
Comment thread
rmems marked this conversation as resolved.
- Other CI/DX issues: #12 (security), #13 (releases on tags w/ sentry option), #14 (MSRV), #15 (Dependabot no auto-merge), #16 (layout clean)
Comment thread
qodo-code-review[bot] marked this conversation as resolved.

See the issue bodies for full ACs and corinth-canal inspiration patterns (one-way copy only; no dep on corinth-canal).

Cross-reference: #11, #8, #9, #7, #5, LIM-9.
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

## License

Licensed under either of
Expand Down
7 changes: 6 additions & 1 deletion src/moe/extract.rs
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,12 @@ fn stacked_slice_range(
path: layout.path.clone(),
reason: format!("stacked stride overflow for tensor '{}'", tensor.name),
})?;
let end = start + stride;
let end = start
.checked_add(stride)
.ok_or_else(|| ParserError::InvalidLayout {
path: layout.path.clone(),
reason: format!("stacked end overflow for tensor '{}'", tensor.name),
})?;
Comment thread
rmems marked this conversation as resolved.
if end > buffer_len {
return Err(ParserError::InvalidLayout {
path: layout.path.clone(),
Expand Down
Loading