Is eval_loss computed after response-only masking? (train_on_completions=true, alpaca template)

[NightvoyagerLin]

Environment: Unsloth Studio 2026.10.8 (bundled unsloth 2026.10.7), Qwen3.5-2B, QLoRA 4-bit,
max_seq_length 4096, train_on_completions: true, dataset formatted with the alpaca
(instruction / input / output) template, 195k train rows / 5k eval rows.

While training, the log prints:

Configuring train on responses only...
Train on responses only configured with dataset template markers (alpaca)

Reading studio/backend/core/training/*, apply_completion_masking -> train_on_responses_only is a
dataset.map(...) applied to both the train and the eval split, so my reading is that the reported
eval_loss is also computed on response tokens only. I would like to confirm this, because it changes
how the number may be used:

1. Is the eval_loss shown in the UI / stored in training_metrics computed on response tokens only,
   or on the full sequence? If it is response-only, it is not comparable with a plain LM eval loss,
   and any cross-version or cross-config comparison has to say so explicitly.

2. What happens to eval rows whose response is empty after truncation to max_seq_length? On the
   train side we have seen the guard message "... samples were dropped after applying
   'Train on completions'" (we hit it with packing: true -- 96.4% dropped). For eval I could not find
   the equivalent counter -- are those rows skipped, counted as 0 tokens, or do they push a NaN into
   the average?

3. Is there a supported switch to pick the eval metric's convention (response-only vs
   full-sequence), so a run can be compared against an external baseline without ambiguity?

We are not reporting a crash -- training itself runs fine (we have a separate, unrelated resume/UI
issue: #15405). We only need to state in our thesis exactly what the reported eval number is, and
right now we can only infer it from the source.

[nicholas-greenwood]

your reading agrees with the published unsloth and unsloth_zoo 2026.10.7/2026.10.8 wheels. the studio masking call processes both splits

on that path, eval_loss uses response labels, including any retained eos

for a non-streaming eval dataset, rows with every label set to -100 are filtered out. if the whole eval split is masked, that code raises rather than averaging an empty split. a row with an unmasked eos label stays in

i couldn't find an independent eval-only convention switch in that studio release's training settings. i'd evaluate the saved checkpoint separately for a full-sequence baseline

how losses are averaged depends on the installed versions. for your thesis, could you share the unsloth_zoo, trl and transformers versions from studio's training environment?

[NightvoyagerLin]

Thanks, this is exactly the kind of answer we needed. Recording it so the thread is complete:

• eval_loss is computed on response labels only, including any retained EOS.
• rows whose labels are entirely -100 after truncation are dropped.
• there is no eval-only convention switch in 2026.10.8; a full-sequence baseline has to be
  produced separately from a saved checkpoint.

To answer your question directly - the versions come from the environments themselves, not from the
docs:

• the 2026.10.8 Studio environment (the run in question):
  unsloth 2026.10.7, unsloth_zoo 2026.10.7, transformers 5.5.0, trl 0.23.1, torch 2.11.0+cu130.
• the previous environment ("old stack", used for the comparison below):
  unsloth 2026.7.5, unsloth_zoo 2026.7.6, torch 2.11.0+cu130
  (transformers 5.5.0 and trl 0.23.1 - recovered later from the upgrade log that was still on disk; see my latest comment).

What we observe, on comparable runs (same dataset, same recipe):

• old stack (2026.7.x, log: "Training on full sequences (including prompts)"):
  eval_loss starts around 0.6 and comes down gradually over training.
• new stack (2026.10.7/10.8, response-only): eval_loss is already ~0.0xxx at the very first
  evaluation - far earlier than such a value should be possible - and then decreases slightly
  further.

Two orders of magnitude lower, or close to it, and the starting point is already near zero. We are
not asking you to justify our numbers; we need to know what this curve still means.

1. How is eval_loss computed in each version? Beyond "response-only", we would like the mechanics:
   is the loss averaged over supervised (response) tokens only, or over all tokens in the batch?
   Does the retained EOS carry the same per-token weight as ordinary answer tokens? Which of these
   changed between 2026.7.x and 2026.10.x?

2. Is the "starts at ~0.0xxx" itself a symptom of how much text is scored? Our own guess is that for
   Alpaca-style rows that Studio renders into a single "text" column (our log shows
   Unsloth: Tokenizing ["text"]), the detected response span may collapse to the trailing token(s)
   (the EOS/newline the template appends), so only a handful of already-saturated tokens contribute.
   If so, is that intended, or should the span cover the whole assistant answer? Is there a supported
   way to inspect what is actually being scored (e.g. the Preview tokens view) so we can confirm the
   span on our own data?

3. All-masked behaviour, per version. You describe a fully masked split raising instead of averaging
   an empty set. We had noted that fast_cross_entropy_loss: all-masked labels give loss 0 instead of NaN #13325 ("return 0 instead of NaN when everything is masked") was
   not in 2026.10.7. Which behaviour do 2026.10.7 and 2026.10.8 have in practice - raise, NaN, or 0?
   This decides whether a mis-detected span shows up as a silent ~0.0xxx or as a visible failure.

4. How should we read this metric from now on? Concretely:

   • Is eval_loss still usable to detect overfitting within one run (train/val divergence)?
   • Is it comparable between checkpoints of the same run, and between two runs on the same version
     and data?
   • Is the curve still meaningful as a convergence signal at all, given it starts so low?
   • If we want one number that can be compared with published work, is the intended recipe
     "reload the checkpoint with plain transformers, same rows, same chat template, no masking"?
     Any padding/truncation detail we should replicate?

One more, unrelated to the above: on the old stack (2026.7.x) a 2B run went NaN around step 30k
(caught from the logged metric, not a crash); the current run has only reached about 1k steps, so we cannot yet say whether it recurs. Was that
addressed by the masking work, or is it an independent numerical issue that could still appear?

[nicholas-greenwood]

thanks for pulling the versions from both environments. i'd check the scored text before using that near-zero curve as evidence of convergence

in the published 2026.10.7 wheels, the fast loss and zoo's own fused ce sum token cross-entropies and divide by the non--100 target count, or the supplied num_items_in_batch. an unmasked eos has the same weight as any other supervised token

transformers 5.5.0 then gathers batch loss scalars and averages them. that isn't necessarily a single token-weighted mean over the entire eval set. keep eval batch size and grouping fixed for comparisons

fewer scored tokens alone don't make that token mean smaller. scoring only easy eos/newline tokens could, but Tokenizing ["text"] doesn't establish that. the span should include the answer after the response marker. the preview implementation recomputes sample masks; it isn't a dump of the run's actual eval labels

if you can inspect trainer in the training worker after masking, this reads a collated eval batch without running the model:

batch = next(iter(trainer.get_eval_dataloader()))
tok = getattr(trainer.processing_class, "tokenizer", trainer.processing_class)
targets = batch.get("shift_labels", batch["labels"][..., 1:])
for ids, labels in zip(batch["input_ids"], targets):
    scored = labels[labels != -100]
    print("scored tokens:", scored.numel())
    print("input:", tok.decode(ids.tolist(), skip_special_tokens=False))
    print("targets:", tok.decode(scored.tolist(), skip_special_tokens=False))

check several batches, including long examples

#13325's zero-count guard is already in the published unsloth 2026.10.7 and .8 wheels. their corresponding zoo fused ce has it too. these guarded paths return 0 for an all-masked batch. the non-streaming dataset filter separately drops all-masked rows and raises if none survive. a row retaining only eos survives that filter and gets scored, so the guard doesn't detect that shortened span

the old unsloth 2026.7.5 / zoo 2026.7.6 fast/fused loss code lacks that guard and can divide by zero. that doesn't identify the cause of your step-30k nan, or establish that masking fixed it

once the targets check out, validation loss can compare checkpoints and show worsening held-out prediction within a run. compare runs with the same eval rows, tokenisation, labels, truncation, packing and batching

for a published baseline, match that work's evaluation protocol. plain transformers with no response mask gives a different objective if the paper scores responses only. match special tokens too, and mask padding labels to -100

yes, the old image's transformers and trl versions would help pin down its exact averaging

[NightvoyagerLin]

We did the scored-token check, and it confirms your expectation rather than overturning it.

Method: for 5 rows of the eval split (shortest two, median, longest two) we rebuilt the exact training text (alpaca markers), tokenized it with the run's own tokenizer, and located the span train_on_responses_only keeps. We also read the masking path in the installed tree - we know the preview view recomputes sample masks rather than dumping the run's labels, so we did not treat it as the run's labels either.

row	total tokens	scored	answer alone	coverage
shortest	47	35	35	100%
2nd shortest	47	35	35	100%
median	65	46	46	100%
2nd longest	443	388	388	100%
longest	502	450	450	100%

Decoded, the scored span equals the answer on all five rows, and it starts right after ### Response:\n and runs to the next user turn / EOS. So the near-zero curve is not produced by scoring only trailing tokens.

On the version question we asked earlier, we can now answer it from the wheels:

• The change is in studio's masking decision, not in train_on_responses_only. In 2026.7.5 both masking call sites skip alpaca datasets outright - trainer.py: and not (is_deepseek_ocr or dataset_final_format == "alpaca"), and worker.py: and format_type != "alpaca" and dataset_final_format not in ("alpaca", "raw_text"), with the comment "Alpaca-rendered text lacks the chat markers". That is why that run printed "Training on full sequences (including prompts)".
• In 2026.10.x those exclusions are gone; the same call passes dataset_template="alpaca", and completion_masking.py resolves it through TEMPLATE_TO_RESPONSES_MAPPER to ### Instruction:\n / ### Response:\n. That is the log line we see now.
• One correction: the environment we re-measured today reports unsloth / unsloth_zoo 2026.10.8 - we rebuilt our environment package in between. It does not change the run in question: studio/backend/utils/datasets/completion_masking.py is byte-identical between 2026.10.7 and 2026.10.8 (sha256 e01b64cf...), and 2026.10.7 already passes dataset_template="alpaca".

Two points we are taking from your reply:

• The batch-scalar averaging: agreed, we will keep eval batch size and grouping fixed for any comparison and will not compare numbers across different groupings.
• The old image's transformers / trl: recovered since. The upgrade log that survived on disk records transformers 5.5.0 and trl 0.23.1 (with torch 2.11.0+cu130), and the upgrade to 2026.10.7 changed only unsloth and unsloth_zoo. Full list in my next comment.

Correction - the paragraph that used to sit here is wrong. I argued from the token counts (12-55 prompt vs 35-450 answer) that the two metrics could only differ by ~1.1-1.4x and that supervising the prompt should push the loss down. Both were wrong: measured on one checkpoint, the prompt segment is harder than the answer, and the ratio between the two metrics is ~36x at step 500 and ~66x at step 1000. The conclusion survives (the two runs are not comparable, the near-zero curve is only a direction check) but for the measured reason, not the estimated one.

If you would like the runtime labels rather than a reconstruction, we can dump one collated eval batch with your snippet the next time we start that environment.

[NightvoyagerLin]

A correction, the measurements behind it, and where we still are.

In my previous comment I argued that the masking change could not explain the size of the difference, because the prompt is only 12-55 tokens against 35-450 answer tokens. That argument was wrong. I would rather correct it with numbers than leave it standing.

How we measured it

Take a checkpoint, load its base weights, apply the adapter's LoRA delta, and push the same fixed batch of eval rows through it twice. The text is built exactly the way the trainer builds it for the alpaca template, and the two runs differ in one thing only - the labels:

• answer-only: everything after the response marker is labelled (EOS included), the rest is masked;
• full sequence: every token is labelled.

For each row we compute per-token cross-entropy, sum it, and divide by the number of labelled tokens. The prompt segment is obtained by subtracting the two runs, so it is measured, not estimated. We repeated this on the base model and on two checkpoints of the same run, over 400 rows sampled evenly from the eval split.

Results

weights	answer-only	full sequence	prompt segment
base	2.7193	2.9553	3.3556
step 500	0.0406	1.4715	3.8988
step 1000	0.0301	1.9866	5.3055

Two checks. Our answer-only numbers land within 8% of the eval_loss the trainer logged at the same steps (0.0439 and 0.0316 over the full 5k split). And the three columns close arithmetically to rounding error, so the decomposition is not a fit.

What it says

• The prompt segment is not cheap. At every checkpoint it is harder than the answer, and it rises with training, because with alpaca masking on the prompt is never supervised.
• So the full-sequence metric is not the same quantity at a different scale. It is dominated by a segment the new objective does not optimise, and it can move in the opposite direction to the answer-only loss - it does here, between steps 500 and 1000.
• My earlier "12-55 tokens, so the denominator moves by ~1.1-1.4x" was wrong for a second reason: it was computed before we matched the text the trainer actually builds. Once matched, the ratio between the two metrics is about 36x at step 500 and 66x at step 1000.

What this means for the two runs

The 2026.7.x run and the 2026.10.x run differ in two ways, and both come from the same switch: what is scored (full sequence vs answer only), and what is optimised (in 7.5 the alpaca branch skips masking entirely, so the prompt is trained; in 10.x only the answer is trained). Because of the second, the old run's prompt segment fell during training, which is why its full-sequence number came down; with masking on, the prompt segment here rises instead. These are not the same quantity, and we will keep them in separate tables and say why.

What we now consider resolved

• eval_loss is computed on response labels only when masking is on - we reproduce it to within 8% from outside.
• Fully masked rows are dropped on a non-streaming split, and a fully masked split raises rather than averaging an empty set.
• There is no eval-only convention switch in this release.
• The scored span is not collapsed; it is the whole answer (checked on real rows, and against the preview endpoint).
• The reason the number is so low: it scores only the answer, and an answer of this kind is largely recoverable from the prompt - while the prompt segment itself is an order of magnitude harder.

What we are still unsure about

1. The prompt segment rises under answer-only training (3.36 to 3.90 to 5.31). Is that expected, does it plateau, and is it considered acceptable for alpaca-style data? It decides whether a full-sequence number from a masked run means anything at all.
2. Whether dropping prompt supervision for alpaca was a deliberate trade-off in 10.x. We can see the switch; we cannot see the reasoning.
3. The old image's versions, from the upgrade log we kept: unsloth 2026.7.5, unsloth_zoo 2026.7.6, transformers 5.5.0, trl 0.23.1, torch 2.11.0+cu130, peft 0.18.1. The same log records that upgrading to 2026.10.7 changed exactly two packages - unsloth and unsloth_zoo; transformers, trl and torch were untouched. So the averaging path cannot explain the difference; whatever changed lives in unsloth. (One caveat: that Studio build can also pin a per-model transformers venv - we see a 5.3.0 one injected for a specific model - so 'the transformers version' is not a single global number in that setup.)
4. How to produce one number that is comparable with published work. Our plan: reload the checkpoint with plain transformers, same rows, same template, no masking, padding labels at -100. If any detail there is wrong we would like to know.
5. Unrelated: the step-30k NaN we saw on the old stack is still unexplained. The current run has only reached about 1k steps, so we cannot yet say whether it recurs - we will report if it does.

Thanks again for the pointers - "check the scored tokens first" is exactly what led us to the text mismatch and to the correction above.

[NightvoyagerLin]

A correction, and the whole picture in one place.

I owe you an apology for something in my earlier comments. Our environment descriptions were written before we had gone through all of the old notebooks, and part of what I said about the 2026.7.x environment was simply wrong - I claimed those versions were no longer recoverable and gave a bound instead of the real numbers. That was our mistake, not a limitation of the software. I have corrected the earlier comments in place; since nobody should have to scroll back through several edits, everything is restated in full below.

First, thank you. The masking code is well documented, and your suggestion to check the scored text before trusting the near-zero curve is exactly what led us to the real problem - which turned out to be our own text reconstruction, not the trainer.

How our two environments were actually built

Both came from the same base image, and the operations we applied only ever changed two packages:

• The old environment was produced by downgrading unsloth to 2026.7.5 and unsloth_zoo to 2026.7.6, with --no-deps. Everything else was left as it was: transformers 5.5.0, trl 0.23.1, torch 2.11.0+cu130, peft 0.18.1.
• That environment was later upgraded to unsloth / unsloth_zoo 2026.10.7, again with --no-deps. The upgrade log we kept records the entire diff: 274 packages, two changed - unsloth and unsloth_zoo. transformers, trl and torch were identical before and after.
• The environment we re-measured for this thread is a later rebuild of that package, reporting unsloth / unsloth_zoo 2026.10.8 with the same transformers 5.5.0. completion_masking.py is byte-identical between 2026.10.7 and 2026.10.8, so the masking path we measured is the one your code runs.

One caveat found while digging: that Studio build can also pin a per-model transformers venv - we see a 5.3.0 one injected for a specific model - so "the transformers version" is not a single global number in that setup.

How we measured

We take a checkpoint, load its base weights, apply the adapter's LoRA delta, and push the same fixed batch of eval rows through it twice. The text is built exactly the way the trainer builds it for the alpaca template, and the two runs differ in one thing only - the labels: answer-only labels everything after the response marker (EOS included) and masks the rest; full-sequence labels every token. For each row we sum per-token cross-entropy and divide by the number of labelled tokens. The prompt segment is obtained by subtracting the two runs, so it is measured, not estimated.

weights	answer-only	full sequence	prompt segment
base	2.7193	2.9553	3.3556
step 500	0.0406	1.4715	3.8988
step 1000	0.0301	1.9866	5.3055

Two checks: our answer-only numbers land within 8% of the eval_loss the trainer logged at the same steps (0.0439 and 0.0316 over the full 5k split), and the three columns close arithmetically to rounding error. The ratio between the two metrics is about 36x at step 500 and 66x at step 1000 - which is why the old and new numbers cannot be placed side by side.

What we now consider resolved

• eval_loss is computed on response labels only when masking is on; we reproduce it to within 8% from outside.
• Fully masked rows are dropped on a non-streaming split, and a fully masked split raises rather than averaging an empty set.
• There is no eval-only convention switch in this release.
• The scored span is not collapsed; it is the whole answer, checked both on real rows and against the preview endpoint.
• The reason the number is so low: it scores only the answer, and an answer of this kind is largely recoverable from the prompt - while the prompt segment itself is an order of magnitude harder. To be precise: "scored on the answer only" and "the prompt segment is an order of magnitude harder" are measured; "the answer is largely recoverable from the prompt" is our interpretation of why the answer is easy, and we have not measured how much of the answer is copied from the prompt.

What we are still unsure about

1. The prompt segment rises under answer-only training (3.36 to 3.90 to 5.31). Is that expected, does it plateau, and is it considered acceptable for alpaca-style data?
2. Whether dropping prompt supervision for alpaca was a deliberate trade-off in 10.x. We can see the switch; we cannot see the reasoning.
3. How to produce one number that is comparable with published work. Our plan: reload the checkpoint with plain transformers, same rows, same template, no masking, padding labels at -100. If any detail there is wrong we would like to know.
4. Unrelated: the step-30k NaN we saw on the old stack is still unexplained, and the current run has only reached about 1k steps, so we cannot yet say whether it recurs - we will report if it does.

Thanks again for the patience and for the pointers - and sorry for the noise the earlier version of that environment description caused.

[nicholas-greenwood]

thanks for doing the 400-row check and tracking down the --no-deps history

the prompt loss can rise under answer-only training. masking the prompt labels removes their prediction errors from the objective while leaving the prompt as context for the answer. it doesn't freeze predictions on prompt text. the answer updates change adapter weights shared across token positions, so prompt predictions can improve or worsen without a prompt-loss penalty

the full-sequence number still measures how well the model predicts that whole rendered text. it mixes prompt prediction with answer prediction, so it needn't track answer quality

your table fits that pattern on those 400 rows. it doesn't tell us whether prompt loss will plateau, or whether generated answers get better. i'd compare generated answers from steps 500 and 1000 on held-out prompts using your task's correctness criteria before deciding whether the extra training helps

[NightvoyagerLin]

Quick background, since the numbers below only make sense with it: we're building a small Chinese-language task parser. Input is one sentence of Chinese describing a to-do ("remind me to hand in the lab report at 3pm tomorrow"); output is a single JSON object — an intent (create / update / query / mark-done) plus a fixed schema of about a dozen fields (title, due time, priority, subtasks, a completion flag, a list name, and so on). We fine-tune a 2B model for this; it's a student project, not a benchmark submission.

Following your suggestion, we stopped ranking checkpoints by loss and instead scored the generated answers.

Setup. Qwen3.5-2B, QLoRA (4-bit base + LoRA) trained in Studio; 7 intermediate checkpoints from 500 to 14,000 steps, evaluated by us. 500 rows sampled uniformly from a 5,000-row validation split (not the official test set), the same rows for every checkpoint. Greedy decoding, batch 8, max_new 1024, stop as soon as the JSON braces balance. Base weights in fp16 with the LoRA merged manually — PEFT silently no-ops in this environment, so we verified the merge numerically. The scoring code was copied verbatim from our scorer, which we cross-checked bit-for-bit against the official script on 5,000 previously generated answers.

checkpoint	exact match (whole JSON right: structure + every field)	all 6 field groups right	unparseable
500	68.00%	73.80%	1
3,000	70.80%	73.20%	0
5,000	76.40%	80.60%	0
8,000	57.60%	62.20%	0
10,000	71.80%	73.40%	13
11,000	52.80%	60.80%	0
14,000	57.80%	60.60%	0

Paired McNemar on the same rows: 5,000 is significantly better than every other checkpoint (smallest margin is against 10,000: −4.60 points, 95% paired half-width ±4.02). After 5,000 it doesn't decline monotonically — it drops into a 53–72 band and oscillates inside it (8,000 and 14,000 are statistically indistinguishable, +0.20 ±4.91; so are 3,000 and 10,000, +1.00 ±3.66). We read that as instability rather than a clean over-fitting curve.

On the loss side — same rows, same checkpoints — both scoring modes point the wrong way. Answer-only eval loss (the number train_on_responses_only produces): 0.04472 → 0.02975 → 0.02798 while exact match goes 76.40 → 52.80. Full-sequence loss (prompt included): 1.4381 → 4.2013 → 5.8771, monotone in the wrong direction too, and non-monotone relative to the answers. The prompt-segment loss alone climbs from 3.88 to 11.50 to 16.11 nats/token: the model loses the ability to reproduce the templated prompt, which looks like a direct consequence of masking the prompt out of the objective. Sanity check: our recomputed answer-only loss sits within 1.02–1.17× of what the trainer logged for those steps (4-bit training vs fp16 merge, plus our 500-row subset of the 5,000-row eval split).

What actually goes wrong when the answers degrade: the intent is still right ~99% of the time and the JSON always closes, so it isn't a formatting failure. The field values collapse — the completion flag gets filled with a time string, a list field that should be empty gets a default bucket, every relative-time answer becomes the same phrase. Loss cannot see that by construction.

Three things we haven't resolved, and they're the reason I'm posting rather than just reporting a score:

1. We don't know why it oscillates after 5k. Candidate explanations are the learning-rate tail, the prompt-side distribution drifting once the prompt is masked out of the objective, or the answer mode collapsing into a few high-frequency templates. We have no experiment that separates them.
2. We don't know whether "train longer" is inherently harmful here. Our run kept a single schedule; we never compared a shorter schedule that plateaus at 5k against simply stopping at 5k. "Peak at 5k" may be a property of this schedule, not of the data.
3. We don't know which selection criterion to trust long-term. Exact match is all-or-nothing; a partial-credit score might rank checkpoints differently. In our case the six-field-group column and the slot-level accuracy agree with exact match (both peak at 5,000), but that is one observation, not a rule.

What we're doing next, and why the numbers above are deliberately provisional: the winning checkpoint (5,000 steps) still has to be run on our actual test set — one full pass over its 5,000 rows, roughly 9–10 GPU hours on a single T4. Everything in this thread is a 500-row subset of the validation split, so we're not putting an absolute number next to anyone else's results yet. The only figure we'd treat as comparable is that full test-set run; we'll post it when it's done, including whether the post-5k oscillation reproduces there. If it does, that's the more interesting finding than the peak itself.

Caveats: 500-row validation subset (so these numbers are not comparable to anyone's test-set results), one training run, and the paired half-widths are only valid within this batch.

One question, since this is a Studio UX matter as much as a training one: would it make sense for Studio to surface a generation-based metric after eval, or at least label eval loss as answer-only? Right now that number silently changed meaning between versions, and it's easy for people to compare across that boundary.