safetensors: coordinate support for new safetensors-parser (extracted/copied from corinth-canal as inspiration)

[rmems]

Summary

Coordination/planning issue: align on the extraction of reusable Safetensors header inspection, manifest generation, and MoE candidate discovery logic from the experimental rmems/corinth-canal (as source of inspiration / one-way copy) into a new dedicated Limen-Neural safetensors-parser crate (parallel to how engram-parser was extracted for GGUF).

Clarification: No dependency from corinth-canal on the new crate. corinth-canal keeps reference copy per its PROMOTION_RULES. The new crate is for reusable use by cortex-tensor (future multi-format MoE support) and others.

See primary:

• safetensors: extract reusable inspection/manifest/MoE-candidate logic from experimental ref impl (copy to dedicated Limen-Neural/safetensors-parser; no dep on corinth-canal) corinth-canal#116 (supporting extraction issue in source repo)
• (Future) new Limen-Neural/safetensors-parser repo + its port/API issues
• Related to Phase 1: engram-parser#7, corinth#115, cortex#8, Linear LIM-88

Context

• cortex-tensor is currently GGUF-only in src/moe/checkpoint.rs / adapter (see its README and open Define tensor + MoE traits inside this crate #7 traits issue, which already calls out reusability with engram-parser).
• Safetensors support in corinth-canal enables the same Router/CheckpointBackend for many local models in experiments (see corinth #40, safetensors_lineup configs, etc.).
• The logic is general and reusable (header parse + deterministic manifest + name/shape-based MoE router/expert labeling + family detection), exactly analogous to the GGUF parser surface moved to engram-parser.
• No Limen-Neural crate currently provides it.

Related

• corinth-canal#116 (this extraction)
• Phase 1 GGUF siblings (engram#7 etc.)
• cortex-tensor#7 (traits, already mentions engram reusability), docs: produce cortex-tensor role in runtime/ownership boundary matrix (LIM-9) #5 (boundary), moe/gguf: coordinate with engram-parser extraction from corinth-canal; avoid overlapping parser work #8 (GGUF coord)
• LIM-9

Goal

Document the relationship so that when cortex-tensor wants multi-checkpoint (Safetensors + GGUF) support for its MoE router, it can depend on the new safetensors-parser (plus engram-parser) rather than duplicating parser code. Planning/alignment only for now.

Non-goals

• Do not implement the dep or full Safetensors backend in cortex in this issue.
• Do not change corinth-canal.

Acceptance criteria

• This issue (or update to moe/gguf: coordinate with engram-parser extraction from corinth-canal; avoid overlapping parser work #8) exists with cross-links to corinth#116, Phase 1 issues, and the future safetensors-parser primary issues.
• Discussion in Define tensor + MoE traits inside this crate #7 (traits) or here acknowledges the new external parser crate as the reusable home for Safetensors (parallel to engram for GGUF).
• README (or Scope/Boundaries) notes potential future multi-format support via the extracted parser crates.
• No duplication of Safetensors header/manifest/candidate logic started in cortex.

Validation

• Manual review of links and discussion in Define tensor + MoE traits inside this crate #7/moe/gguf: coordinate with engram-parser extraction from corinth-canal; avoid overlapping parser work #8.
• (Later) Revisit under a follow-up when ready to integrate in cortex.

Suggested branch

docs/safetensors-parser-coordination

References

• Detailed plan in engram-parser session plan.md.
• corinth-canal#116 for the extraction tracking.

Coordination issue for Safetensors extraction (copy from corinth-canal inspiration) to keep Limen-Neural reusable libs aligned.

[rmems]

Siblings: corinth-canal#116, updated LIM-88, beads raulmc-yqj.

See engram-parser#7 for GGUF sibling and the session plan.md for details. Primary work will live in new safetensors-parser repo once created.

[rmems]

New separate tracking issue in engram-parser (for the Safetensors extraction from corinth-canal, separate from GGUF #7): rmems/engram-parser#10

See the engram issue + plan (and corinth#116) for the full separate tracking. No dep from corinth-canal. Cross-links updated.

[rmems]

Grok Build agent: Started work on #9 (per user + plan).

• Went and looked at /home/raulmc/corinth-canal (the source of the Safetensors reference impl).
• Analyzed reusable surface: inspect_safetensors_checkpoint / write_safetensors_manifest, SafetensorsManifest + records + discover_candidates + layout family / router/expert grouping (in src/moe/safetensors.rs + safetensors/discovery.rs).
• Integration notes (for future): resolve_safetensors_adapter, CheckpointBackend::Safetensors, MappedSafetensorsCheckpoint, safetensors_gate_scores.
• Mirrored against engram-parser pattern (GGUF extraction precedent).
• Added coordination notes + cross links (corinth#116, engram#10, cortex Define tensor + MoE traits inside this crate #7/moe/gguf: coordinate with engram-parser extraction from corinth-canal; avoid overlapping parser work #8) to:
  • README.md (moe section + ecosystem callout)
  • src/moe/mod.rs (module docs)
• No implementation, no dep, no changes to corinth (per non-goals + "one-way copy").
• Gates: fmt + cargo check/test --no-default-features clean.
• Committed + pushed.

See updated docs and session plan. Ready for next step (e.g. actual parser crate bootstrap when aligned).

Refs: cortex#9, corinth-canal safetensors code, engram-parser.