Controlled function-calling corpus ablations and mechanism study

[ReinforcedKnowledge]

Parent: #220

Goal

Determine whether the candidate corpus-processing intervention causes the observed function-calling gains, which parts of the intervention matter, and which mechanisms explain any transfer to multi-turn agent behavior.

This issue deliberately describes the controlled methodology without publishing the exact processing rule yet. The complete transformation code, manifests, and arm definitions will be linked when the analysis is ready.

Study design

The study starts with Qwen/Qwen3.5-0.8B-Base as a low-cost causal screen. Larger Qwen3.5 checkpoints and an OLMo-family transfer are promotion stages, not substitutes for the first controlled assay.

The experiment family freezes the following within each comparison:

• source revisions and eligible training population;
• tokenizer, chat and tool serialization, loss masking, and sequence length;
• per-source exposure, packed-token budget, optimizer schedule, and checkpoint selection;
• serving and parser configuration;
• BFCL and tau2 evaluation revisions, task sets, trial counts, and reporting rules.

The core comparisons include an unprocessed baseline, the candidate processing intervention, source-matched and token-matched controls, structure-preserving variants, and controls that separate sample removal from message removal or target masking. Additional mechanism arms test whether changes are explained by loss of useful function-calling demonstrations, shifts between single-turn call emission and multi-turn interaction, confidence strata, or simple changes in corpus mass. Repletion and placebo controls are used where support can be defined without leaking examples across arms.

The first screen uses paired training order and multiple seeds for the key arms. Promotion requires a stable direction across seeds, no unresolved parser or data confound, and agreement between aggregate scores, per-category or per-domain results, reliability curves, and qualitative trace audits.

Outcomes

• BFCL overall and category-level changes, including abstention and multi-turn behavior.
• tau2 retail, airline, and telecom pass^k curves rather than pass^1 alone.
• Tool-call parseability, call emission, recovery after tool errors, and long-context behavior.
• General capability regressions and contamination checks.
• Mechanism contrasts that distinguish useful-demonstration depletion from semantic or structural effects.

Problems encountered and resolutions

Problem	Resolution
The shared training stack did not natively support the Qwen3.5 architecture and structured tool template.	Built an isolated, pinned Qwen-native path and validated model loading, rendering, packing, and checkpoint identity before any scientific training.
Early trainer review found incorrect selected-row loss behavior, local rather than global assistant-target normalization, insufficient optimizer-state precision, repeated schedules, seed coupling, and an incomplete resume comparison.	Implemented a target-selected objective, global normalization, FP32 optimization state, hashed no-repeat schedules, seed-specific paired order, exact resume checks, and independent validators.
The intended fused loss path failed a preregistered numerical comparison against the reference.	Rejected that accelerated path and moved to a correctness-first non-fused, checkpointed, chunked selected-row objective.
Full 32K qualification exposed issues in parser flags, hardware assertions, sparse tensor probes, allocator-trace capacity, callback compatibility, and report validation.	Fixed each issue under frozen tests, reran the bounded qualification, retained failed attempts, and independently validated the replacement evidence.
One add-back control lacks a clean donor reserve that is excluded from every canonical arm.	Kept the affected mechanism arms unfrozen until a preregistered reserve can be defined without withholding or duplicating the core population.

Current status

• The core model-neutral arm design and data construction are accepted.
• Two independent complete builds reproduce byte for byte, including membership, operations, rendering, packing, and matching checks.
• The corrected Qwen-native trainer and C00 baseline are staged and independently validated locally.
• Hardware qualification has passed identity, reference parity, distributed correctness, and one-GPU real-32K memory and kernel gates through H4.
• H5 four-GPU production-path preparation is active. Resume, topology, reporting closure, scientific training, BFCL, and tau2 remain blocked until the remaining gates pass.
• No causal training result is being claimed yet.

Deliverables

• Complete the remaining Qwen3.5-0.8B hardware and trainer qualification gates.
• Run the preregistered 0.8B causal pilot with paired seeds.
• Freeze and run the remaining mechanism controls once their support is valid.
• Promote only supported effects to larger Qwen3.5 checkpoints.
• Test cross-family transfer on an OLMo checkpoint.
• Publish arm manifests, cleaned construction and training scripts, exact revisions, and BFCL and tau2 reports.

[ReinforcedKnowledge]

Final status: evaluation invalidated; no causal result

This issue is being closed after the final evidence audit. The controlled training campaign was executed, but its original evaluation matrix was not a valid measurement of model capability. Consequently, the study does not support a causal or mechanistic conclusion about the corpus interventions.

Training and infrastructure completed

• Qualified the Qwen3.5 training path, including model and checkpoint identity, tool serialization, selected assistant-token loss, distributed normalization, deterministic paired ordering, long-context execution, optimizer state, checkpointing, exact restart, serving compatibility, and result reporting.
• Completed the 0.8B controlled campaign: 54 checkpoint trees covering the controlled and mechanism arms across three paired training seeds.
• Qualified the larger-model training and serving paths.
• Preserved the training manifests, arm definitions, construction logic, and audit evidence needed to reconstruct the experimental design.

Original BFCL evaluation

The original BFCL no-Web matrix contained:

• 162 completed evaluation components;
• 3,726 result files;
• 812,754 generation rows.

The post-hoc audit found:

• 184,828 generation rows with tracebacks;
• 153,724 mapping or tool-argument failures;
• 31,104 client-closed failures;
• all 129,600 scored multi-turn rows incorrect or zero;
• all 25,110 scored vector-memory rows failed;
• systematic saturation of scalar non-live and live generations at the 4,096-token output limit.

The resulting aggregate scores are not valid capability measurements.

No conclusion is supported about:

• the benefit of the proposed semantic or structural filtering;
• any individual mechanism arm;
• the causal effect of sample removal, message removal, or target masking;
• cross-seed stability of a capability effect;
• transfer to a larger model.

Generic removal controls performed as well as or better than the proposed intervention in several comparisons, but the evaluation defects prevent interpreting those differences as model behavior.

Corrected evaluation

The corrected BFCL path fixed the known termination, history-mapping, vector-memory, scoring, and serving problems.

One complete all-category canary was run for each of the 0.8B and 9B model sizes. Each covered all 5,017 BFCL rows, including 800 multi-turn and 192 vector-memory rows, without row-level evaluation errors.

The final server-log audit found parser-side tracebacks that were absent from the evaluation receipts. The canaries were therefore retained as infrastructure diagnostics only. Their scores were not promoted, and they did not justify launching the full corrected matrix.

No corrected multi-seed capability matrix was run.

No Qwen tau2 or BFCL Web result was obtained.

Resource constraint and possible resumption

After the audit, the available compute was insufficient to regenerate and run the complete multi-seed training and corrected evaluation matrix, including tau2. The corrected canaries were not scaled into another large campaign because the required compute, runtime, and evaluation capacity were not available.

The study may be resumed if sufficient compute becomes available. A valid resumption would require regenerating the training artifacts, passing an all-category canary that includes server-log validation, and then running a preregistered corrected matrix. The previous checkpoints are no longer retained.

Outcome

The intended causal study is non-confirmatory because the measurement was invalid, not because the interventions were demonstrated to have no effect.

The reusable outputs are:

• the evaluation failure audit;
• all-category canary and server-log acceptance requirements;
• checks for multi-turn and memory degeneracy;
• output-length saturation checks;
• explicit separation of evaluation repetitions from training seeds;
• reproducible dataset construction and analysis tooling.

Follow-up work on the data and evaluation methodology moves to #220. No filtering, capability, or mechanistic claim is made from this campaign.