Skip to content

v0.56.0: legacy-format PyTorch files and .nemo checkpoints are read to the last pickle

Choose a tag to compare

@munimv2 munimv2 released this 09 Nov 20:07
· 40 commits to main since this release

model-file-scan 0.13.1, pickle_model_scan 0.5.0, nemo_container_scan 0.3.0. Made with AI agents working in the name of MV2 of Munim, Inc.

  • pickle_model_scan 0.5.0 walks every pickle the loader reads, not just the first. torch.save(..., _use_new_zipfile_serialization=False) (the legacy format, the default before PyTorch 1.6, still read by torch.load) writes five pickles back to back and then raw tensor bytes. Up to 0.4.0 the scanner read only the 15-byte magic-number pickle and reported such a file clean without seeing any import. Files written with several pickle.dump calls are now walked one stream after another. New keys streams and torch_legacy in pickle_walk(). No new rule and no severity change.
  • nemo_container_scan 0.3.0 does the same for a legacy-format model_weights.ckpt inside a .nemo tar. 0.2.0 reported nothing at all for one, not even the NEMO010 note a zip-format checkpoint gets. It now reports NEMO010 with "legacy torch.save, 5 of 5 pickles".
  • model-file-scan 0.13.1: version bump only. Routing and every other scanner are unchanged.

Pre-registered checks, graded against torch.load / pickle.load on honest files:

  • legacy-torch-v1, 19 of 19 held (REALFILE_CHECK_LEGACY_TORCH_2026-11.md). 0.5.0 ends its walk exactly where the loader's last pickle ends and resolves every import the loader asked for on all 8 files with ground truth. Among the 500 pytorch_model.bin headers read from the 1,000 most downloaded pytorch models on the Hugging Face Hub, 79 (15.8%) are legacy format, among them gpt2, bert-base-uncased and t5-small. On four small ones 0.4.0 read 15 bytes and 0.5.0 read the whole pickle.
  • nemo-legacy-v1, all 12 release rows held (REALFILE_CHECK_NEMO_LEGACY_2026-11.md). All 280 weights members in 55 published .nemo files are zip format, none legacy, so this closes a silent case for private or older checkpoints.

Each fix's full suite ran on its own tree (1,396 and 1,397 tests, 0 failures). Before commit, the changed modules' tests ran together on this release's files.

A clean scan is not a guarantee of safety: pickle-based formats can run code when loaded, so only load files from sources you trust. Apache-2.0.