Repository navigation
v0.10.0
Optional register tokens for eligible single-CLS ViTs, with register-aware checkpoint layouts and patch consumers.
Added
--register-tokens none|0|4(defaultnone) onpretrain:4adds four trainable registers and migrates the one-CLS pretrained backbone (the register token keeps its native initialization while every CLS/patch/block weight and position is preserved);nonekeeps the backbone's own architecture, i.e. no registers on a plain ViT or a native-register model's own count.- A shared read-only checkpoint loader with a register-layout decision table, rejection of non-canonical
reg_tokenkeys, and RNG-isolatedpretrained=Falseprobes;extract,camandexportall load through it. - Registered pretraining resume and registered fine-tune sources, including student/teacher synchronization of the newly initialized registers and explicit layout validation before any optimizer state is restored.
- An attention-pool patch policy (
num_prefix_tokens,register_tokens,prefix_excluded,attention_pooling_type) recorded in new pool checkpoints and saved to a policy-tagged sibling whose contents are validated even when the filename matches. - CPU FP32 PyTorch versus CPU ONNX Runtime numeric validation of exports (
atol=1e-4,rtol=1e-3), recorded asvalidation_status,max_abs_diff,atolandrtol. - A design document and implementation plan under
docs/superpowers/.
Changed
- Every patch consumer (patch-topk, attention-query training, attention-pool extraction and the ViT CAM reshape) now drops all prefix tokens via
backbone.num_prefix_tokens, so a CLS-free backbone keeps its first patch. - New checkpoints record
config.register_tokens(the actual count) andargs.register_tokens(the invocation value); fine-tuning re-saves the inherited layout and includes it in the v0.9.0 pseudo-label experiment identity. vit_reshape_transformgained an optionalprefix_tokensargument; the default is unchanged for direct callers.- README and README.cn.md document the option, the patch policy and the export validation.
Fixed
- A failed or numerically mismatched export no longer publishes a model and can no longer leave a stale
validated: truereport behind; only this attempt's temporary files are cleaned up. - An incompatible attention-pool checkpoint at the policy-tagged path now fails naming its path instead of being masked by another fallback artifact.
config.register_tokensis validated as a non-negative integer at every checkpoint entry point, so a malformed value is rejected rather than treated as a legacy zero.
Upgrade notes
noneis the default and preserves the previous construction, so existing zero-register workflows are unchanged; the 272 existing checkpoints were verified end to end (extract, CAM, ONNX export) with the corrected patch boundaries.- ONNX export now fails, instead of publishing a shape-only validated model, when ONNX Runtime is installed but the graph fails to run or disagrees numerically beyond the approved tolerances.
- Patch-consumer outputs change for CLS-free/no-register checkpoints (the first patch is now included) and for CLS-bearing native-register checkpoints (registers are no longer treated as patches): regenerate affected patch-topk embeddings and attention pools.
- Fine-tuning has no new switch; it inherits the source architecture and its register layout.