Skip to content

v0.113.0: CLS classification pooling

Choose a tag to compare

@mrothroc mrothroc released this 16 Sep 22:09
· 102 commits to main since this release

CLS classification pooling

Opt into a learned class token with training.classification.pooling: "cls".
A trainable cls_token of shape [1, model_dim] is prepended after the input
adapter and before positional embeddings, and the classifier reads position
zero after final normalization.

  • seq_len and batch_tokens keep their loader meanings. The backbone runs at
    seq_len + 1; learned positional capacity defaults to that longer length, and
    an explicit max_positions must cover it.
  • Supported for token embeddings, linear_frames, and discrete_codebooks
    inputs over bidirectional plain attention, optionally composed with mlp,
    swiglu, geglu, or moe. Causal/recurrent mixers, FFN-only stacks,
    BatchNorm, RC equivariance, and embedding side channels are rejected with a
    clear error until they have dedicated coverage.
  • Padding masks gain a valid CLS entry inside the graph, so loader and runtime
    input shapes are unchanged and padded batches agree with unpadded ones.
  • The class token initializes from Normal(0, 0.02); training.weight_init: "normal" uses training.weight_init_std instead.
  • Hugging Face export carries the token, the extra learned-position row, and the
    padding-aware mask. Native vs. Transformers max logit error is 7.15e-7
    (linear_frames) and 2.38e-7 (tokens). Codebook adapters remain native-only.

Existing mean and last pooling layouts are untouched, so current
checkpoints load unchanged. Switching pooling mode is not checkpoint-compatible.

New example: examples/cls_frames_classifier.json

Other Changes

  • feat: publish Docker Hub overviews and short descriptions for the app and
    CUDA layers directly from the repo, correcting understated GPU support
  • fix: the Docker Hub overview sync reuses the push step's secret binding
  • refactor: class-token initialization dispatches on a weight-meta init mode
    rather than a literal tensor name, matching every other special initializer

Install

brew install mrothroc/tap/mixlab

Or build from source:

CGO_ENABLED=1 go build -tags mlx -o mixlab ./cmd/mixlab/