Skip to content

Payload Validation

Jump to bottom
Sietse edited this page Sep 29, 2026 · 3 revisions

Payload Validation

Refuse a corrupted block at write time, so a bad one never reaches the model.
Status ✅ Works
Verified 28 August 2026: 0 false alarms in 400 scans of healthy data
Default every block is checked when it is written; its size is checked again when it is read
Error code MERLIN_ERR_CORRUPT_PAYLOAD

In plain words

Saved memory is a large block of numbers. If some of those numbers are damaged (a bad disk, a half-finished write, a memory fault) the model will still use them and still produce a fluent answer. It will simply be wrong, with nothing in the logs.

This checks blocks for impossible values (NaN, infinity) and impossible shapes, and refuses the bad ones.

The everyday example

A server loses power mid-write. The block on disk is half old data, half new. On the next request, the check notices the size does not add up, treats it as a miss, and recomputes. The user waits a moment instead of receiving nonsense.

The basics

How it works

  1. Before a block is saved, Galahad checks it for impossible values (NaN, infinity) and a wrong size.
  2. A damaged block is refused and never saved.
  3. When a block is read back, its size is checked again. A mismatch is a miss, and the model recomputes.

How to use it

  1. The checks are always on. There is nothing to switch on.
  2. Tell Galahad your number format (dtype, for example bf16), so it can check the values. See below.
  3. Watch the counters, or handle MERLIN_ERR_CORRUPT_PAYLOAD in your code.

What runs, and when

Path Default What it does
write ✅ on a fast value scan plus shape rules, before the block is written
load, size check ✅ on checks the block's size matches its token count

⭐ A failed check on load is a miss, not an error. Your host already handles a miss by recomputing.

Check What it finds
value scan NaN and infinity values
shape rules a block that is all zeros
size check a block that is cut short, or the wrong size for its token count

What it cannot catch

⚠ A flipped bit that still gives a valid number, and a block from a different model of the same size, are invisible to any value scan.

Those are covered by other checks:

  • every read from disk is checked against what was written, see Collision Safety
  • encryption rejects any modified file, see Encryption at Rest
  • memory is only shared between identical model_fingerprint values, see Install

⭐ Use this for impossible values. The read check and encryption cover arbitrary damage.

Counters

In merlin_get_stats:
payload_poison_deposits bad blocks refused at write
payload_poison_loads bad blocks caught at load
payload_structural_fails size mismatches
payload_quarantined blocks set aside because a later check failed
payload_scans_total · payload_scan_ns_total how much scanning is costing you

⭐ Scrape payload_poison_deposits. Above zero means your hardware or your producer is generating bad data, and you are learning it before it was stored.

⚠ payload_scans_total tells you the checks are running. If it stays at 0, no scans ran, whatever the other counters say. Most often this means dtype is not set (see below).

The error code

merlin_deposit_bytes returns MERLIN_ERR_CORRUPT_PAYLOAD when a block is refused.

⚠ Handle it separately from an I/O error. An I/O error means retry; a corrupt payload means do not.

Telling Galahad your data type

merlin_paged_layout layout = {0};
layout.struct_size = sizeof layout;
/* ... dimensions and strides ... */
layout.dtype       = MERLIN_DTYPE_BF16;   /* or F16, FP8_E4M3, F32 */
merlin_set_paged_layout(&layout);

⚠ Without dtype, the value scan cannot read the numbers and only the size checks apply. Always set struct_size too; a caller that does not is refused with MERLIN_ERR_ABI_MISMATCH.

Troubleshooting

Symptom Cause Fix
MERLIN_ERR_CORRUPT_PAYLOAD on deposit the block really is bad do not retry; find the producer
payload_poison_deposits climbing failing hardware or a bad producer investigate the source, not the scanner
payload_scans_total stays at 0 dtype not set set dtype in merlin_set_paged_layout
nothing detected but data is wrong a flipped bit is invisible to a value scan rely on the read check and encryption

Related

Clone this wiki locally