[engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal)

[rmems]

Source

Synced / linked to GitHub for 1:1 mirror (2026-07-26).

• GitHub: [engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal) #10 — OPEN
• Repo: Limen-Neural/engram-parser GH#10
• Match confidence: 1.15

---

Summary

Track the separate extraction (one-way copy of implementation as inspiration) of the reusable Safetensors header inspection, deterministic manifest generation, and MoE router/expert candidate discovery logic from the experimental reference implementation in rmems/corinth-canal (parallel to the GGUF work in this repo's #7).

This is the separate dedicated issue in engram-parser (per user request) for the Safetensors case. Companion/parallel to GGUF #7 / corinth#115. See corinth-canal#116 (the supporting/bootstrap issue in the source experimental repo) and rmems/cortex-tensor#9 (coord).

Important clarification (per modularization intent and prior reviews): This is one-way extraction/copying. We are not making corinth-canal depend on any new or target crate. corinth-canal remains self-contained as the experimental end-to-end reference (per its own PROMOTION_RULES.md: after "frozen", it keeps an unmodified reference copy of the logic). "we are not making corinth-canal dep. We are simply copying and implementing code from inspiration." "corinth-canal repo more of my experimental repo. Limen-Neural organization repos are more about modular reusable libraries. Ideally I am trying to extract useful code from corinth-canal repo."

Context

• The Safetensors support in corinth-canal (src/moe/safetensors.rs + safetensors/discovery.rs) is a self-contained pure-Rust implementation (header parse without the official safetensors crate for the header/inspect path, sharded/HF index/dir support, inspect_safetensors_checkpoint / write_safetensors_manifest, SafetensorsManifest with tensor records (name/dtype/shape/byte_size/offsets/shard) + metadata + checkpoint source, MoE candidate labeling via name/shape heuristics for routers/experts/groups (classify_tensor, discover_candidates, SafetensorsRouterCandidate, SafetensorsExpertGroup, layout family detection e.g. "generic_moe", "deepseek_v3_family", "phimoe" etc.), plus MappedSafetensorsCheckpoint for loading tensor bytes on demand).
• It is used for local model onboarding, examples/safetensors_manifest.rs, verification checklist, and dual-backend (with GGUF) in the Router/Checkpoint for experiments.
• Currently marked "reference" (implied under moe/checkpoint in MODULE_STATUS.md targeting rmems-moe).
• No Limen-Neural crate yet owns the reusable Safetensors parser + MoE discovery surface (unlike GGUF → engram-parser). Enram-parser remains GGUF-only per its charter (this issue is for ecosystem/sibling coordination of the parallel extraction).
• Phase 1 (GGUF) issues already created: engram#7 (primary in this repo), corinth#115. corinth#115 non-goals noted "Do not move Safetensors handling (separate module) unless trivial" — this work (tracked via the separate corinth#116 + this engram issue) makes it non-trivial.
• Parallel extractions (cortex-tensor from corinth) and the overall LIM-9 boundary matrix favor focused reusable libs under Limen-Neural. corinth-canal#116 exists as the source-side supporting issue (with note that primary work for a new crate will follow); this is the separate tracking issue here in engram-parser (the GGUF home) for the Safetensors extraction story.
• Supporting corinth-canal#116 and cortex#9 exist; this creates the requested separate issue in engram-parser.

Related

• safetensors: extract reusable inspection/manifest/MoE-candidate logic from experimental ref impl (copy to dedicated Limen-Neural/safetensors-parser; no dep on corinth-canal) corinth-canal#116 (supporting extraction issue in source repo; cross-link + update its comment)
• Phase 1 GGUF siblings: parser: adopt/enhance as canonical GGUF parser by extracting from corinth-canal #7 (this repo's GGUF primary; this issue is the Safetensors separate parallel), moe: migrate GGUF parser layer to engram-parser dep (extract useful code from experimental ref impl) corinth-canal#115 (migration side), moe/gguf: coordinate with engram-parser extraction from corinth-canal; avoid overlapping parser work cortex-tensor#8 (GGUF coord) + Add Docker workflow for reproducible / containerized builds (publish to GHCR on main) #9 (Safetensors coord), Linear LIM-88 (extend), beads
• corinth docs/PROMOTION_RULES.md (frozen = reference copy kept unmodified in corinth), MODULE_STATUS.md (update safetensors layer), MODEL_SOURCE_VERIFICATION_CHECKLIST.md, configs/safetensors, examples/safetensors_manifest.rs
• LIM-9 (org boundary matrix)
• engram-parser README / scope (add ecosystem note for sibling parser extraction from corinth-canal)

Goal

Ensure the Safetensors extraction from corinth-canal (as inspiration, one-way copy, no dep) is tracked with a dedicated separate GitHub issue in this repo (engram-parser), parallel to #7. This provides visibility in the parser-related issues here, cross-links the full story (corinth source #116, cortex coord #9, future reusable crate work, LIM-9), and follows the precedent of explicit GitHub issues for such modularization.

(The actual reusable crate implementation can be a new Limen-Neural/safetensors-parser or placed per future decision; this issue tracks the extraction effort and ecosystem from engram's perspective.)

Non-goals

• Do not add a runtime or build dependency from corinth-canal's Cargo.toml or code paths.
• Do not change corinth-canal's Router/CheckpointBackend/Safetensors experiment paths, outputs, or examples (they continue using the local reference copy).
• Do not move corinth-specific extensions.
• This issue does not change engram-parser's GGUF-only implementation surface (engram charter preserved; this is coordination/ecosystem).
• No full matmul/dequant/CUDA (pure parser/extractor like the GGUF side).
• Do not create new repo in this plan (per user revision).

Acceptance criteria

• This separate GitHub issue created in Limen-Neural/engram-parser with full cross-links, "no dep on corinth-canal" language, and details.
• Core logic extraction tracked: the inspect, manifest, discovery/classify from corinth src/moe/safetensors* + discovery.rs (header-only, single/index/dir support, SafetensorsManifest, MoE router/expert groups/candidates, layout families).
• Cross-links and comments added/updated on corinth#116, cortex#9, this repo's parser: adopt/enhance as canonical GGUF parser by extracting from corinth-canal #7.
• engram README updated with ecosystem/sibling note for the Safetensors parser extraction from corinth-canal (parallel to GGUF/engram).
• corinth-canal docs updated (MODULE_STATUS for safetensors layer toward frozen, cross-refs, explicit "no dep" in comments on #115/#116).
• Linear LIM-88 extended or noted with link to this new engram issue + corinth#116.
• Beads created/updated in corinth-canal for the Safetensors (separate).
• Coordination acknowledged (e.g. comment in cortex parser: adopt/enhance as canonical GGUF parser by extracting from corinth-canal #7 or Add Docker workflow for reproducible / containerized builds (publish to GHCR on main) #9).
• All places carry the clarifications: one-way copy from inspiration, Limen-Neural org for reusable libs, no dep on corinth-canal, engram stays GGUF-only.
• Plan.md updated with the new issue number.

Validation

• Manual review of new issue, all cross-comments, README, docs updates.
• Links render; language matches prior reviews ("no dep...", "copying and implementing code from inspiration").
• (Later, when port happens) New reusable surface produces equivalent manifests/candidates to corinth on test cases from corinth configs.

Suggested branch

docs/safetensors-sibling-extraction (or feat/safetensors-ecosystem if any engram updates).

References

• Primary plan: this engram-parser session plan.md
• corinth-canal#116 (bootstrap/supporting in source; the "extract ... from corinth-canal")
• Phase 1: engram-parser#7, corinth#115, cortex#8/Add Docker workflow for reproducible / containerized builds (publish to GHCR on main) #9, Linear LIM-88 + beads
• corinth src/moe/safetensors.rs + discovery.rs + examples + configs + docs/PROMOTION_RULES.md + MODULE_STATUS.md
• LIM-9

Created as the separate GitHub issue in engram-parser for the Safetensors extraction from corinth-canal (parallel to #7; per user request for the issue here, one-way copy, no dep on corinth-canal, following modularization pattern from LIM-9 and GGUF precedent).

---

Siblings (cross-linked/updated together):

• safetensors: extract reusable inspection/manifest/MoE-candidate logic from experimental ref impl (copy to dedicated Limen-Neural/safetensors-parser; no dep on corinth-canal) corinth-canal#116 (supporting; already existed, link + comment updated)
• safetensors: coordinate support for new safetensors-parser (extracted/copied from corinth-canal as inspiration) cortex-tensor#9 (coord; link + comment updated)
• engram-parser#7 (GGUF primary in this repo; comment updated)
• Linear LIM-88, beads in corinth