safetensors.mojo v0.7.0

Summary

Version 0.7.0 expands native support from Linux x86-64 to Linux x86-64, Linux ARM64, and Apple silicon macOS while preserving the existing Safetensors API and filesystem security boundaries.

Highlights

• Added native linux-aarch64 and osx-arm64 Conda packages alongside linux-64.
• Made memory-mapped readers and atomic file replacement portable across the three supported targets.
• Added a Darwin shard resolver using descriptor-relative lookup, nonblocking opens, symlink rejection, regular-file checks, and file-identity validation.
• Preserved the descriptor-pinned Linux resolver and added architecture-correct open flags for both x86-64 and ARM64.
• Added platform-specific unit, integration, compile-contract, fuzz, and clean-installed package coverage.
• Expanded CI to validate and fuzz on native Linux x86-64, Linux ARM64, and macOS ARM64 runners without duplicate feature-branch runs.
• Reworked the release pipeline to build and smoke-test every package natively, validate the complete artifact set before publication, safely reuse immutable GitHub assets on reruns, and publish only after all platforms pass.
• Moved procedural release policy into independently tested tooling while keeping GitHub Actions focused on permissions, runners, orchestration, and artifact transfer.
• Documented the supported host baselines and portability design in ADR-008.

Installation

[workspace]
channels = [
  "https://prefix.dev/modular-community",
  "https://conda.modular.com/max",
  "conda-forge",
]
platforms = ["linux-64", "linux-aarch64", "osx-arm64"]

[dependencies]
safetensors-mojo = "==0.7.0"
mojo-compiler = "==1.0.0"

The installed Mojo package is imported as safetensors; the project does not create a mojo.safetensors namespace. The project release is published first to the verified egor-fedorov/safetensors-mojo channel; availability of version 0.7.0 from modular-community follows its separate recipe update.

Compatibility

The Conda package requires Mojo exactly 1.0.0 and is built natively for linux-64, linux-aarch64, and osx-arm64. The public format, reader, mapped-view, writer, and sharding APIs are unchanged from v0.6.0.

Linux packages require glibc 2.34 or later and a C compiler usable as the linker. The x86-64 build requires an x86-64-v3 CPU, the ARM64 build requires a Neoverse N1-class or newer CPU, and the macOS build requires Apple silicon with macOS 15 and Xcode or Xcode Command Line Tools 16 or later.

Deliberate limitations

Version 0.7.0 supports the complete native platform set of Mojo 1.0.0. Release packages are built and verified on their native targets rather than cross-built. Remote Hub downloads, index writing, automatic shard planning, slicing, MAX or tensor-runtime adapters, checksums, authentication, immutable file snapshots, and fsync crash durability remain outside the current scope.

Index-controlled resolution still does not distinguish hard links from ordinary regular files. Backing files must remain stable for the lifetime of mapped views; same-length mutation may be visible, and external truncation can cause SIGBUS.

Full Changelog: v0.6.0...v0.7.0

safetensors.mojo v0.6.0

Summary

Version 0.6.0 adds secure local sharded Safetensors reading through standard index files or trusted explicit shard lists, with exact cross-shard validation, bounded hostile-input handling, buffered access, and zero-copy mapped views.

Highlights

• Added open_safetensors_index() and map_safetensors_index() for validated *.safetensors.index.json archives, plus open_sharded_safetensors() and map_sharded_safetensors() for application-trusted shard lists.
• Added deterministic ShardedSafeTensorMetadata and ShardedTensorInfo APIs with global lexicographic and shard-grouped iteration.
• Index construction validates every referenced shard and requires exact routing: declared tensors must exist in the selected shard, every physical tensor must be mapped exactly once, and optional metadata.total_size must match the checked aggregate payload size.
• Index-controlled shard names are treated as untrusted basenames and cannot traverse directories, select symlinks, or resolve to non-regular files; explicit-list paths remain caller-trusted and may follow symlinks for layouts such as Hugging Face cache snapshots.
• Added configurable limits for index bytes, weight_map entries, unique shards, and nested ignored JSON values while retaining the existing per-shard header limit.
• Buffered readers retain at most one active shard reader and revalidate identity, length, and metadata when switching shards; mapped archives eagerly retain one origin-safe mapping per unique shard so zero-copy views from different shards can coexist.
• Restored PathTraversal at its reserved ordinal and appended six sharding-specific error kinds without renumbering existing categories.
• Added valid, malformed, routing, path-security, file-identity, ownership, and integration coverage, plus deterministic shard-index survival fuzzing in CI.
• Added current architecture documentation and ADR-007, moved installation guidance to the shared modular-community channel, surfaced CodeQL status, and guarded release notes against duplicate GitHub titles.

Installation

[workspace]
channels = [
  "https://prefix.dev/modular-community",
  "https://conda.modular.com/max",
  "conda-forge",
]
platforms = ["linux-64"]

[dependencies]
safetensors-mojo = "==0.6.0"
mojo-compiler = "==1.0.0"

The installed Mojo package is imported as safetensors; the project does not create a mojo.safetensors namespace. The project release is published first to the verified egor-fedorov/safetensors-mojo channel; availability of version 0.6.0 from modular-community follows its separate recipe update.

Compatibility

The Conda package targets linux-64 and requires Mojo exactly 1.0.0. The sharded APIs are additive, and existing single-file parser, reader, mapped-view, and writer contracts remain unchanged. strict=True applies to each Safetensors shard header; index JSON always follows its fixed compatibility and security policy.

SafeTensorErrorKind.PATH_TRAVERSAL, removed in v0.5.0 while its ordinal remained reserved, is restored at ordinal 22. New sharding-specific error kinds occupy ordinals 26–31, so existing numeric categories are not renumbered.

Deliberate limitations

Descriptor-relative index resolution and mapped access remain Linux-only. The project does not download or resolve remote Hub artifacts and does not provide an index writer, automatic shard planner, slicing, MAX or tensor-runtime adapters, checksums, authentication, or immutable file snapshots.

Index path defenses do not distinguish hard links from ordinary regular files, so an attacker-writable archive directory is not a trusted boundary. Buffered readers are not concurrent or reentrant, while mapped archives consume one descriptor and whole-file mapping per unique shard. Backing files must remain stable; same-length in-place mutation may go undetected, and truncating a mapped shard can still cause SIGBUS.

The shard-index fuzz harness checks process survival rather than whether every generated input should be accepted or rejected.

Full Changelog: v0.5.0...v0.6.0

safetensors.mojo v0.5.0

Summary

Version 0.5.0 improves compatibility with the Safetensors 0.8.0 reference implementation, strengthens differential and fuzz coverage, aligns error diagnostics, and adds reproducible performance measurements.

Highlights

• Readers now accept leading and trailing JSON whitespace supported by the reference implementation, including space, tab, line feed, and carriage return.
• Unknown tensor descriptor fields are ignored by default after bounded validation of their complete JSON values, improving compatibility with additive format extensions.
• Added strict=True to parse_raw_header(), parse_metadata_from_header(), parse_metadata(), open_safetensors(), and map_safetensors() for applications that require the previous closed-schema and canonical-boundary policy.
• Preserved exact unsigned-integer parsing, decoded duplicate detection for schema-significant keys, checked arithmetic, and complete dtype, shape, size, offset, and data-coverage validation in both reader modes.
• Added a 79-tensor byte-exact matrix generated by the pinned Safetensors 0.8.0 serializer for its 20 supported dtypes across valid byte-addressable scalar, vector, multidimensional, and zero-element cases.
• Added manually constructed cases for both recognized F6 encodings and verified their dtype, shape, and payload semantics with the pinned reference deserializer.
• Added byte-exact Mojo writer parity against the reference-generated matrix.
• Non-byte-addressable complete sub-byte tensors now report MisalignedSlice, matching the reference implementation; the unused PathTraversal error category was removed while its numeric ordinal remains reserved.
• Added deterministic survival fuzzing for the parser, buffered reader, mapped reader, raw spans, and representative typed views in a separate CI job.
• Added a reproducible manual benchmark for a sparse 967 MiB archive with 193 F32 tensors, separating warmed open/map latency from fresh-process startup cost and recording raw samples and environment metadata.
• Replaced the completed repository roadmap with GitHub Issues for planned work and GitHub Releases for curated history, and added project status badges.

Installation

[workspace]
channels = [
  "https://prefix.dev/egor-fedorov/safetensors-mojo",
  "https://conda.modular.com/max",
  "conda-forge",
]
platforms = ["linux-64"]

[dependencies]
safetensors-mojo = "==0.5.0"

The installed Mojo package is imported as safetensors; the project does not create a mojo.safetensors namespace.

Compatibility

The Conda package targets linux-64 and requires Mojo exactly 1.0.0. Reader compatibility is intentionally broader than in v0.4.0; applications that depended on rejecting leading JSON whitespace, non-space JSON padding, or unknown tensor descriptor fields should pass strict=True. Writer output remains canonical and compatible with Safetensors 0.8.0.

SafeTensorErrorKind.PATH_TRAVERSAL was removed because it was never produced by the library. Code that referenced that public constant must be updated. Non-byte-addressable sub-byte tensors now produce MisalignedSlice instead of InvalidTensorSize.

Deliberate limitations

Mapped access remains Linux-only and requires the backing file to remain stable for the lifetime of the mapping and all borrowed views. The project does not provide slicing, sharding, MAX or tensor-runtime adapters, decoded or byte-swapped fallback views, typed-value encoding, incremental or in-memory archive writing, append or update-in-place operations, mmap writes, or fsync crash durability.

The fuzz harness checks process survival rather than whether each generated file should be accepted or rejected. Benchmark results are machine- and workload-specific and do not claim a Mojo parsing-speed advantage over the reference implementation.

Full Changelog: v0.4.0...v0.5.0

safetensors.mojo v0.4.0

Summary

Version 0.4.0 adds the first public Safetensors writer for Mojo: deterministic format planning from owned raw tensor entries followed by atomic local-file replacement on Linux.

Highlights

• Added the root-level SafeTensorData model and save_safetensors() one-shot writer.
• Added canonical compact JSON serialization with stable metadata ordering, dtype/name tensor ordering, little-endian header-length encoding, and ASCII-space padding to an 8-byte boundary.
• Added checked shape, bit-length, payload-length, cumulative-offset, header-length, and complete-file layout validation before any filesystem mutation.
• Added exclusive mode-0600 sibling temporary files using Linux OS randomness, complete writes through FileHandle.write_all(), and atomic destination replacement through rename(2).
• Preserved existing readers and mappings across path replacement, and defined symlink replacement, cleanup, missing-parent, and concurrent last-successful-rename semantics in ADR-005.
• Added an independently generated canonical fixture with exact-byte and SHA-256 checks plus semantic loading through Python safetensors==0.8.0.
• Extended root API contracts and clean-installed Conda package smoke coverage to exercise the writer.

Installation

[workspace]
channels = [
  "https://prefix.dev/egor-fedorov/safetensors-mojo",
  "https://conda.modular.com/max",
  "conda-forge",
]
platforms = ["linux-64"]

[dependencies]
safetensors-mojo = "==0.4.0"

The installed Mojo package is imported as safetensors; the project does not create a mojo.safetensors namespace.

Compatibility

The Conda package targets linux-64 and requires Mojo exactly 1.0.0. Writer output is semantically compatible with Safetensors 0.8.0; the project defines deterministic bytes for its own writer but does not require byte identity with reference implementations whose metadata-map iteration order may differ.

Deliberate limitations

Writer inputs are already packed C-order little-endian wire bytes. Version 0.4.0 does not add typed-value encoding, byte swapping, serialization to a complete in-memory archive, a stateful or incremental writer, append/update-in-place behavior, mmap writes, fsync crash durability, cross-platform replacement, slicing, sharding, MAX adapters, or tensor-runtime adapters.

Full Changelog: v0.3.0...v0.4.0

safetensors.mojo v0.3.0

safetensors.mojo v0.3.0 adds zero-copy, read-only access to Safetensors payloads through Linux memory mappings, including raw byte spans and exact native scalar views whose Mojo lifetimes remain tied to the mapping owner.

Highlights

• Adds map_safetensors() and the movable, non-copyable MappedSafeTensorFile owner for Linux whole-file PROT_READ | MAP_PRIVATE mappings.
• Adds tensor_bytes() for immutable zero-copy Span[UInt8] access to any validated tensor payload.
• Adds tensor_view[DType.*]() for flat immutable native scalar spans with exact runtime dtype matching, checked native sizes, and, when applicable, little-endian and actual-address alignment checks before the pointer cast.
• Supports exact native views for signed and unsigned 8-, 16-, 32-, and 64-bit integers; F16, BF16, F32, and F64; and all five corresponding Mojo 1.0 float8 encodings.
• Preserves raw byte access for valid unaligned tensors and for BOOL, packed F4 and F6, and C64, which intentionally have no native typed view in this release.
• Returns typed SafeTensorError failures for unsupported representations, incompatible endianness, misaligned tensor addresses, and the new DTypeMismatch case.
• Uses Mojo origins to prevent mapped spans from being mutated, escaping their owner, or remaining usable after the owner is consumed; these properties are pinned by positive and negative compiler contracts.
• Retains the opened file descriptor so renaming, unlinking, or replacing the path does not redirect an existing mapping, and rejects observed file growth or truncation before returning a view.
• Reorganizes production source, tests, and tooling by responsibility while preserving the supported root import package, safetensors, and using absolute imports throughout the project.

The strict format core from v0.1.0 and the buffered local random-access reader from v0.2.0 remain available through the root safetensors API.

Install

[workspace]
channels = [
  "https://prefix.dev/egor-fedorov/safetensors-mojo",
  "https://conda.modular.com/max",
  "conda-forge",
]
platforms = ["linux-64"]

[dependencies]
safetensors-mojo = "==0.3.0"

Import the installed Mojo package as safetensors:

from safetensors import map_safetensors


def main() raises:
    var mapped = map_safetensors("model.safetensors")
    var bytes = mapped.tensor_bytes("weights")
    var values = mapped.tensor_view[DType.float32]("weights")

    print("mapped bytes:", len(bytes))
    print("native values:", len(values), values[0])

Compatibility

• Mojo 1.0.0 exactly.
• Linux x86-64 (linux-64).
• Pixi/Conda distribution: safetensors-mojo.
• Mojo import package: safetensors; no mojo.safetensors namespace is created.
• The generated .mojoc package is compiler-version-specific.
• Safetensors wire data remains packed C-order and little-endian.
• Root-package exports are the supported API; nested module paths are internal and may change between releases.
• This is a pre-1.0 API; later minor releases may refine public interfaces.

Scope and limitations

Mapped views are read-only and one-dimensional; logical tensor shape remains in validated metadata. This release does not include writers, slicing, sharding, MAX adapters, tensor-runtime adapters, decoded fallback views, or byte-swapped fallback views.

The backing inode must remain unchanged from before map_safetensors() begins until the mapping owner and all borrowed spans are dead. A read-only private mapping is not an immutable snapshot: same-length changes may become visible, and dereferencing mapped pages after external truncation can terminate the process with SIGBUS.

Safetensors does not provide authenticity, integrity, signatures, or encryption. Use a trusted stable source or an independent owned snapshot when those properties are required.

The verified .conda artifact is published on the egor-fedorov/safetensors-mojo channel and attached to this release.

Full Changelog: v0.2.0...v0.3.0

safetensors.mojo is an independent implementation of the Safetensors file format for Mojo and is not affiliated with or endorsed by Hugging Face.

safetensors.mojo v0.2.0

safetensors.mojo v0.2.0 adds safe local random-access file reading on top of the strict format core introduced in v0.1.0.

Highlights

• Metadata-only opening through open_safetensors() without loading tensor payloads.
• An owned, non-copyable SafeTensorReader that retains one read-only file handle.
• Exact named-tensor reads into caller-owned byte buffers with read_tensor_into().
• Explicit owned payload loads with load_tensor().
• Checked absolute-offset arithmetic and native-size conversions for every file-controlled value.
• Detection of ordinary file-length changes around reads.
• A retained-handle design that prevents path replacement from redirecting later reads.
• End-to-end tests covering clean Conda installation and the public root API.
• A verified release pipeline that publishes the same tested artifact to Prefix.dev and GitHub Releases.

The runtime-independent parsing and validation API from v0.1.0 remains available unchanged.

Install

[workspace]
channels = [
  "https://prefix.dev/egor-fedorov/safetensors-mojo",
  "https://conda.modular.com/max",
  "conda-forge",
]
platforms = ["linux-64"]

[dependencies]
safetensors-mojo = "==0.2.0"

Import the installed Mojo package as safetensors:

from safetensors import open_safetensors

Compatibility

• Mojo 1.0.0
• Linux x86-64 (linux-64)
• Pixi/Conda distribution: safetensors-mojo
• Mojo import package: safetensors
• The generated .mojoc package is compiler-version-specific.

Scope and limitations

Reader results are owned raw wire bytes. This release intentionally does not include memory mapping, borrowed or typed tensor views, writers, slicing, sharding, MAX adapters, or tensor-runtime adapters. It detects observed length changes but cannot detect every same-length in-place modification. Calls on one reader share its seek cursor and must not execute concurrently.

The verified .conda artifact is published on the egor-fedorov/safetensors-mojo channel and attached to this release.

Full Changelog: v0.1.0...v0.2.0

safetensors.mojo is an independent implementation of the Safetensors file format for Mojo and is not affiliated with or endorsed by Hugging Face.

safetensors.mojo v0.1.0

The first public release of safetensors.mojo establishes a strict, runtime-independent Safetensors format core for Mojo 1.0.

Highlights

• Complete SafeDType model for every recognized Safetensors wire dtype.
• Strict UTF-8 and JSON header parsing, including duplicate decoded-key detection.
• Exact unsigned-integer parsing without a floating-point intermediate.
• Checked shape, bit-length, byte-length, offset, and full-data-coverage validation.
• Separate raw and validated metadata structures.
• Deterministic valid and malformed fixtures, including compatibility coverage generated with the Python reference implementation.
• Reproducible Pixi environment and Conda package build.

Install

[workspace]
channels = [
  "https://prefix.dev/egor-fedorov/safetensors-mojo",
  "https://conda.modular.com/max",
  "conda-forge",
]
platforms = ["linux-64"]

[dependencies]
safetensors-mojo = "==0.1.0"

Import the installed Mojo package as safetensors:

from safetensors import parse_metadata

Compatibility

• Mojo 1.0.0
• Linux x86-64 (linux-64)
• Pixi/Conda distribution: safetensors-mojo
• Mojo import package: safetensors
• The generated .mojoc package is compiler-version-specific.

Scope and limitations

This release parses and validates complete caller-owned .safetensors byte buffers. It intentionally does not include file readers, memory mapping, tensor-data views, writers, slicing, sharding, MAX adapters, or tensor-runtime adapters.

The verified .conda artifact is published on the egor-fedorov/safetensors-mojo channel and attached to this release.

Full Changelog: 054f7fe...v0.1.0

safetensors.mojo is an independent implementation of the Safetensors file format for Mojo and is not affiliated with or endorsed by Hugging Face.