Skip to content

Add a natural-image DINOv2 ViT-B/14 control tile encoder#213

Merged
clemsgrs merged 2 commits into
mainfrom
agent/232-natimage-encoder
Jul 4, 2026
Merged

Add a natural-image DINOv2 ViT-B/14 control tile encoder#213
clemsgrs merged 2 commits into
mainfrom
agent/232-natimage-encoder

Conversation

@clemsgrs

@clemsgrs clemsgrs commented Jul 3, 2026

Copy link
Copy Markdown
Owner

What

Registers dinov2-vitb14: the original DINOv2 ViT-B/14 self-supervised on LVD-142M natural images (Oquab et al., 2024), shipped by timm as vit_base_patch14_dinov2.lvd142m. Weights are hosted publicly on Hugging Face under timm/vit_base_patch14_dinov2.lvd142mno gated access, no HF token.

Implements clemsgrs/soma#232.

Why

soma's detection benchmark (Paper 1) needs a non-pathology ViT control to answer "does pathology-pretraining actually pay off for detection?" (and Paper 2 the "catches up late" curve). Almost every pathology tile encoder here (UNI, Virchow, GigaPath, H-optimus, Midnight, …) is a DINOv2-family ViT, so pairing them with a DINOv2 ViT trained on natural images holds architecture and the SSL objective fixed and varies only the pretraining domain — exactly the control the benchmark wants.

How

Structurally a plain TimmTileEncoder, mirroring lunit / prost40m / uni: the dense (encode_tiles_dense) and attention (encode_tiles_attention) paths are inherited unchanged from the timm ViT base, so the control is dense-extraction- and attention-capable exactly like the pathology encoders. dynamic_img_size=True lets the natively-518px backbone run at the 224px detection tile geometry via positional-embedding interpolation (a no-op at native size, verified in the shared dense-extraction suite).

Metadata

field value note
level tile
output dim 768 ViT-B, single default variant
input_size 224 matches every pathology tile encoder → same tile geometry
patch_size 14 224 / 14 = 16 → 16×16 dense grid
precision fp16
supported_spacing_um None spacing-agnostic — no validation constraint (see below)
default_spacing_um 0.5 tiling default when selected by name
source timm/vit_base_patch14_dinov2.lvd142m public

Spacing model — split constraint from default

A natural-image model has no intrinsic micron-per-pixel spacing, so advertising 0.5 as a supported/recommended spacing (the way the pathology encoders do) would be a lie: it would make validate_encoder_config reject any other spacing as "non-recommended" when in fact no spacing is more correct than another. This PR splits the previously overloaded supported_spacing_um into its two real roles:

  • supported_spacing_um — the validation constraint: the spacing(s) a model was trained/validated for. Off-list requests raise unless allow_non_recommended_settings=True. Set to None here to mark the encoder spacing-agnostic — the spacing check is skipped entirely.
  • default_spacing_um — the single spacing a slide is tiled at when the encoder is selected by name with no explicit requested_spacing_um. Set to 0.5 here so the control lands on the pathology encoders' tile geometry and drops in with zero config.

Consequences:

  • Selecting dinov2-vitb14 by name still tiles at 0.5 µm/px → matched geometry, zero config (acceptance criterion satisfied).
  • Sweeping other task-spacings (e.g. the 0.25 used in the OCELOT campaign) now needs no escape hatch — any requested spacing is accepted as-is, because the model has no spacing to violate.

default_spacing_um is optional and derived from supported_spacing_um when that is a single value, so the ~13 single-spacing pathology encoders need no edits. The multi-spacing guardrail is preserved: virchow2 / musk / midnight (a list of spacings) still require an explicit requested_spacing_um unless they opt in with their own default_spacing_um. Default resolution is kept lazy so tile-size validation never trips over a list/agnostic encoder.

Other judgment calls

  • Plain DINOv2 (no registers). Picked vit_base_patch14_dinov2 over the reg4 variant for the cleanest "vanilla ViT" control: a single CLS prefix token, which is the simplest dense/attention layout (like uni/midnight/phikon). The reg4 variant is a trivial swap if you'd prefer to match the register-carrying pathology models (UNI2, H-optimus, Virchow2).
  • ViT-B/14 as the size: the mid-size that most directly parallels the ViT-L/g pathology backbones while staying cheap.
  • Aliases dinov2 / dinov2-base / dinov2-vitbdinov2-vitb14.

Tests

New tests/test_dinov2_natimage.py mirrors the existing encoder suite:

  • registration/metadata contract (supported_spacing_um is None, default_spacing_um == 0.5) + alias resolution,
  • spacing model: name-only selection resolves the 0.5 tiling default, and validate_encoder_config accepts any requested spacing (0.25 / 2.0) with no escape hatch,
  • dense grid parity vs timm's get_intermediate_layers oracle at the 224px detection geometry (offline, pretrained=False) — pins spatial registration, not just shape,
  • CLS-only per-head attention shape.

The registry split (supported_spacing_um accepts None; new default_spacing_um) lives in slide2vec/encoders/registry.py; the existing multi-spacing guard tests (virchow2/musk/midnight in test_regression_core / test_regression_models) are unchanged and still pass. Also updated the shared exact-membership lists (test_encoder_registry, test_patch_size_metadata, test_regression_core), docs (docs/models.rst), and the README count.

Full suite (-o addopts=""): 483 passed, 4 skipped (the 4 skips are the weight-dependent heavy tests, which run on the nightly heavy workflow). Separately verified end-to-end with the real downloaded weights on CPU: dense (1, 768, 16, 16) · attn (1, 12, 16, 16) · pooled (1, 768) at 224px.

Coverage note: this repo's --cov addopts hit a sqlite lock on my (network) filesystem, so I ran pytest with -o addopts="". Heavy weight-parity test (test_static_patch_size_matches_runtime) is excluded from the PR suite by design and runs on the nightly heavy workflow.

Register `dinov2-vitb14`: the original DINOv2 ViT-B/14 self-supervised on
LVD-142M natural images, shipped by timm as
`vit_base_patch14_dinov2.lvd142m` (public weights on Hugging Face, no gated
access). It mirrors the plain timm ViT tile encoders (lunit / prost40m / uni),
so the dense (`encode_tiles_dense`) and attention (`encode_tiles_attention`)
paths are inherited unchanged and it is dense-extraction-capable exactly like
the pathology encoders.

Almost every pathology tile encoder here (UNI, Virchow, GigaPath, H-optimus,
Midnight, ...) is a DINOv2-family ViT, so a DINOv2 ViT trained on natural
images holds architecture and objective fixed and varies only the pretraining
domain -- the control needed to ask "does pathology-pretraining actually pay
off?" for a downstream task.

Metadata: dim 768, input_size 224, patch_size 14, precision fp16. A
natural-image model has no intrinsic micron spacing; supported_spacing_um=0.5
is a convention that matches the pathology tile encoders' default task-spacing
so selecting this encoder by name lands on identical tile geometry. Sweep other
task-spacings via allow_non_recommended_settings=True (it is spacing-agnostic).
`dynamic_img_size=True` lets the natively-518px backbone run at the 224px tile
geometry (pos-embed interpolation, a no-op at native size).

Add `dinov2` / `dinov2-base` / `dinov2-vitb` -> `dinov2-vitb14` aliases, docs,
and registration + dense/attention parity tests mirroring the encoder suite.
Split the overloaded supported_spacing_um into its two real roles: a
validation constraint and a tiling default. supported_spacing_um now accepts
None to mark a spacing-agnostic encoder (validate_encoder_config skips the
spacing check entirely), and default_spacing_um declares the single spacing a
slide is tiled at when the encoder is selected by name without an explicit
requested_spacing_um.

default_spacing_um is optional and derived from supported_spacing_um when that
is a single value, so single-spacing encoders need no edits. The multi-spacing
guardrail is preserved: encoders advertising a list of spacings still require an
explicit choice unless they declare default_spacing_um. Default resolution stays
lazy so tile-size validation never trips over a list/agnostic encoder.

The natural-image DINOv2 control declares supported_spacing_um=None,
default_spacing_um=0.5: it tiles at 0.5 to match the pathology encoders' geometry
but accepts any requested spacing without the allow_non_recommended escape hatch,
since it has no intrinsic micron-per-pixel spacing to violate.
@clemsgrs clemsgrs merged commit f05fec1 into main Jul 4, 2026
3 checks passed
@clemsgrs clemsgrs deleted the agent/232-natimage-encoder branch July 4, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant