Encoder-Only Chinese Mini MLM

SCENIC 8-GPU Sparsity Sweep

Run the full 21-row sparsity comparison from a base model path:

./scripts/run_scenic_sparsity_revision_from_base.sh /path/to/base-encoder-checkpoint

Replace /path/to/base-encoder-checkpoint with the model path you want to start from. The script defaults to 5-epoch SFT, 8-GPU torchrun for SFT and one-shot pruning, GPU-split progressive pruning, and a base encoder-only dense baseline.

H20 SFT, NVIDIA 2:4 pruning with rebuilt classifier, ONNX dynamic INT8 comparison, and TensorRT benchmarking entrypoint: scripts/run_h20_encoder_only_sft_prune_trt24.sh; native TensorRT INT8 rows still require trtexec plus valid Q/DQ ONNX or calibration caches.

This project mirrors the corpus and systems workflow from /Users/luke/Documents/Decoder Only, but trains a BERT-style encoder-only masked language model instead of a decoder-only causal LM.

The full H20 recipe keeps the same public Chinese data blend and tokenizer-first preparation:

• webText2019zh and wiki-like Firefly data
• Baike QA
• Chinese medical dialogue
• Zhihu-KOL
• BELLE 1M, 2M, and 3.5M Chinese instruction/chat data

See REFERENCES.md for the model, systems, and dataset sources cited for this setup. See ENVIRONMENT.md for the dedicated H20 environment recipe and preflight checks. See ARCHITECTURE.md for the exact architecture classification and citations. See PRETRAINING_AUDIT.md for the BERT/RoBERTa tokenization and MLM training audit. See BENCHMARKS.md for C-Eval output notes after training.

How To Cite

GitHub can read CITATION.cff and show a "Cite this repository" button. For an IEEE-style references section, you can paste this entry:

[1] L. Z. Hu, "Encoder-Chinese-SLM: An Encoder-Only Chinese Small Language Model Training Codebase," GitHub repository, version 0.1.0, 2026. [Online]. Available: https://github.com/huluk98/Encoder-Chinese-SLM

Use this BibTeX entry for this codebase:

@software{encoder_chinese_slm_2026,
  author = {Hu, Luke Ztz},
  title = {Encoder-Chinese-SLM: An Encoder-Only Chinese Small Language Model Training Codebase},
  year = {2026},
  version = {0.1.0},
  url = {https://github.com/huluk98/Encoder-Chinese-SLM}
}

When describing the model architecture or training recipe, also cite the upstream datasets, benchmarks, and software listed in REFERENCES.md.

License

This repository's code and documentation are released under the MIT License. External datasets and model resources referenced by this project remain governed by their respective source licenses and terms.

Setup

conda env create -f environment.yml
conda activate chatlm-encoder
pip install -e ".[deepspeed]"

Full 8-GPU Workflow

One command on the 8-GPU host:

./scripts/run_full_pipeline_8gpu.sh

The launcher defaults to the DeepSpeed CLI and uses all visible GPUs:

LAUNCHER=deepspeed CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 ./scripts/launch_h20_8gpu.sh

Use LAUNCHER=torchrun only for debugging the same DeepSpeed-initialized training script under torchrun.

Equivalent manual steps:

HF_HUB_ENABLE_HF_TRANSFER=1 python scripts/download_data.py --config configs/h20_8gpu_bert_0p2b_deepspeed.yaml
python scripts/prepare_data.py --config configs/h20_8gpu_bert_0p2b_deepspeed.yaml
python scripts/train_tokenizer.py --config configs/h20_8gpu_bert_0p2b_deepspeed.yaml
python scripts/pack_tokens.py --config configs/h20_8gpu_bert_0p2b_deepspeed.yaml
./scripts/launch_h20_8gpu.sh

Resume after stopping or crashing:

./scripts/launch_h20_8gpu.sh --resume runs/h20-8gpu-bert-0p2b-mlm-deepspeed/latest

Monitor training:

tail -f runs/h20-8gpu-bert-0p2b-mlm-deepspeed/metrics/training_metrics.csv
python scripts/summarize_training_run.py runs/h20-8gpu-bert-0p2b-mlm-deepspeed

Plot loss:

python scripts/plot_loss.py --metrics runs/h20-8gpu-bert-0p2b-mlm-deepspeed/metrics/training_metrics.csv

The full pipeline also runs encoder-style C-Eval after training and writes results to eval_results/ceval/latest:

python scripts/eval_ceval.py --checkpoint runs/h20-8gpu-bert-0p2b-mlm-deepspeed/latest --split val --n-shot 5

Recorded C-Eval result for step-100000 (val, 5-shot, MLM cloze scoring):

Scope	Accuracy	Correct / Total
Overall	26.52%	357 / 1346
Humanities	21.40%	55 / 257
Other	27.60%	106 / 384
STEM	27.21%	117 / 430
Social Science	28.73%	79 / 275

Set RUN_CEVAL_AFTER_TRAIN=0 if you want to train first and run C-Eval manually later.

The pipeline skips an already-created tokenizer and packed token file by default. Use FORCE_TOKENIZER=1 or FORCE_PACK=1 only when you intentionally want to rebuild those artifacts.

Fast 0.194B Check

Run this before launching a fresh H20 training job to confirm the config creates the intended 0.194B encoder:

git pull
PYTHONPATH=src python - <<'PY'
from chatlm_encoder.config import load_config
from chatlm_encoder.model import count_parameters, create_model
from chatlm_encoder.tokenizer import load_tokenizer

cfg = load_config("configs/h20_8gpu_bert_0p2b_deepspeed.yaml")
tok = load_tokenizer(cfg["tokenizer"]["path"])
model = create_model(cfg["model"], tok)

print("tokenizer_size:", len(tok))
print("layers:", model.config.num_hidden_layers)
print("hidden_size:", model.config.hidden_size)
print("heads:", model.config.num_attention_heads)
print("intermediate:", model.config.intermediate_size)
print("vocab_size:", model.config.vocab_size)
print("parameters:", f"{count_parameters(model):,}")
PY

Expected:

tokenizer_size: 29298
layers: 24
hidden_size: 768
heads: 12
intermediate: 3072
vocab_size: 29298
parameters: 193,700,466

If an older-size run already exists, move it aside and start a fresh run without --resume:

mv runs/h20-8gpu-bert-0p2b-mlm-deepspeed "runs/old-wrong-size-run-$(date +%Y%m%d-%H%M%S)"
PACK_TOKENS=0 ./scripts/launch_h20_8gpu.sh

The startup log should print Parameters: 193,700,466. After the first checkpoint, confirm the saved model shape:

python - <<'PY'
import json
p = "runs/h20-8gpu-bert-0p2b-mlm-deepspeed/latest/config.json"
cfg = json.load(open(p, encoding="utf-8"))
print(cfg["num_hidden_layers"], cfg["hidden_size"], cfg["num_attention_heads"], cfg["intermediate_size"], cfg["vocab_size"])
PY

Expected:

24 768 12 3072 29298

SCENIC Encoder SFT

This is encoder-only supervised fine-tuning, so it trains prompt-to-response classifiers rather than generative decoders. The two SCENIC datasets are trained into separate checkpoints for comparison tables:

• data/scenic/SCENIC_full_training_dataset.json -> runs/scenic-sft-training-dataset/latest
• data/scenic/SCENIC_full_anchor_positive_negative.json -> runs/scenic-sft-contrastive-dataset/latest

The contrastive-dataset model uses anchor -> response as the supervised task and also uses the positive/negative fields as an auxiliary contrastive loss.

New SCENIC SFT configs use attention-mask mean pooling for the classification embedding, cosine-style logits, and initialize each classifier row from the encoder embedding of that response label. The response classifier also uses a separate higher learning rate than the encoder. Earlier checkpoints that do not record these settings still load with their original CLS/random-head behavior.

These SCENIC JSON files are tracked in this repo, so git pull brings them down with the SFT scripts. Train both models sequentially on the 8-GPU H20 box:

./scripts/launch_scenic_sft_separate_8gpu.sh

Or train one model at a time:

CONFIG=configs/scenic_sft_training_dataset_8gpu.yaml ./scripts/launch_scenic_sft_8gpu.sh
CONFIG=configs/scenic_sft_contrastive_dataset_8gpu.yaml ./scripts/launch_scenic_sft_8gpu.sh

Both configs expect the MLM-pretrained encoder at:

runs/h20-8gpu-bert-0p2b-mlm-deepspeed/latest

Set model.base_model in each SCENIC SFT config if your checkpoint path is different. Change each model output path with run.output_dir.

Evaluate a local SCENIC-style JSON file:

python scripts/eval_scenic_sft_local.py

By default this evaluates on the 200-row SCENIC IoT benchmark:

data/scenic/iot_instruction_benchmark_200.json

Run the full comparison suite:

python scripts/eval_scenic_sft_comparison.py

For a direct base-SFT EM@1/EM@5 sanity check on training retention and benchmark rows, run:

python scripts/check_scenic_sft_em1_em5.py

This writes:

eval_results/scenic_sft/em1_em5_check/em1_em5_summary.json
eval_results/scenic_sft/em1_em5_check/em1_em5_summary.csv

To evaluate the MLM-pretrained base encoder before SFT, use the base-model cloze scorer:

python scripts/check_scenic_base_model_em1_em5.py

This ranks candidate responses from SCENIC_full_training_dataset.json with masked-LM scores and writes:

eval_results/scenic_base_model/em1_em5_check/base_model_em1_em5_summary.json
eval_results/scenic_base_model/em1_em5_check/base_model_em1_em5_summary.csv

If you prefer not to pass CLI flags, edit the path block at the top of:

scripts/run_scenic_sft_eval_local.py

Then run:

python scripts/run_scenic_sft_eval_local.py

This runs four evaluations:

• training-dataset model on the 200-row benchmark
• training-dataset model on SCENIC_full_training_dataset.json for retention
• contrastive-dataset model on the 200-row benchmark
• contrastive-dataset model on SCENIC_full_anchor_positive_negative.json for retention, using only anchor as the input prompt

The suite writes:

eval_results/scenic_sft/comparison/comparison_summary.json
eval_results/scenic_sft/comparison/comparison_summary.csv
eval_results/scenic_sft/comparison/comparison_groups.csv

50% SCENIC SFT pruning and outcomes

After both SCENIC SFT checkpoints exist, run the pruning-and-evaluation launcher:

./scripts/prune_scenic_sft_50_eval.sh

This creates two separate 50% sparse pruned checkpoints:

runs/scenic-sft-training-dataset-pruned50
runs/scenic-sft-contrastive-dataset-pruned50

By default, pruning is unstructured magnitude pruning over encoder transformer linear weights. It leaves token embeddings, LayerNorm/bias tensors, and the response classifier unchanged. The checkpoint files are not expected to become 50% smaller unless you use sparse storage or sparse inference kernels later; the script reports the real zero-rate in each prune_summary.json.

The launcher evaluates the pruned models on:

• the 200-row SCENIC benchmark: data/scenic/iot_instruction_benchmark_200.json
• the original training source for the training-dataset model: data/scenic/SCENIC_full_training_dataset.json
• the original anchor source for the contrastive model: data/scenic/SCENIC_full_anchor_positive_negative.json

Outcome files:

runs/scenic-sft-training-dataset-pruned50/prune_summary.json
runs/scenic-sft-contrastive-dataset-pruned50/prune_summary.json
eval_results/scenic_sft/pruned50_comparison/comparison_summary.csv
eval_results/scenic_sft/pruned50_comparison/comparison_groups.csv

To independently verify active/nonzero parameters from the saved pruned checkpoint tensors, run:

python scripts/audit_scenic_active_parameters.py

The audit reloads each checkpoint and recomputes tensor != 0 counts directly, then cross-checks targeted_* and model_* sparsity fields against prune_summary.json when present. If no checkpoint paths are provided, it auto-discovers generated pruned SCENIC checkpoints under runs/ and writes eval_results/scenic_sft/active_parameters_audit.json.

Use comparison_summary.csv for the main table. The benchmark_200 rows are the SCENIC benchmark outcomes. The training_dataset_retention and contrastive_anchor_retention rows are the original-data retention outcomes.

Useful overrides:

SPARSITY=0.5 PRUNE_SCOPE=encoder-linear ./scripts/prune_scenic_sft_50_eval.sh
EVAL_DTYPE=fp32 BATCH_SIZE=64 ./scripts/prune_scenic_sft_50_eval.sh
BENCHMARK_JSON=/path/to/benchmark.json ./scripts/prune_scenic_sft_50_eval.sh

If you want 50% sparsity over every matrix weight, including embeddings, use:

PRUNE_SCOPE=all-matrix ./scripts/prune_scenic_sft_50_eval.sh

Run all pruning variants and aggregate the benchmark/original-data outcomes:

./scripts/prune_scenic_sft_all_methods_eval.sh

This runs:

• encoder-linear: encoder transformer linear weights only, classifier kept dense
• all-linear-classifier: all non-embedding 2D weights, including the response classifier
• all-matrix-classifier: all matrix weights, including embeddings and the response classifier

The combined table is written to:

eval_results/scenic_sft/pruned50_all_methods/all_methods_summary.csv
eval_results/scenic_sft/pruned50_all_methods/all_methods_summary.json

To run only selected methods:

METHODS=encoder-linear,all-matrix-classifier ./scripts/prune_scenic_sft_all_methods_eval.sh

Reference-style 50% pruning for T5 comparison

For an apples-to-apples comparison with the reference T5 pruning scripts, first train the encoder SFT checkpoints:

./scripts/launch_scenic_sft_separate_8gpu.sh

Then run the reference-style 50% pruning suite:

./scripts/prune_scenic_sft_reference_methods_50_eval.sh

If you prefer not to pass CLI flags, edit the path block at the top of:

scripts/run_scenic_reference_pruning_local.py

Then run:

python scripts/run_scenic_reference_pruning_local.py

This runs four post-SFT pruning methods on both SCENIC SFT checkpoints:

• magnitude: per-nn.Linear unstructured magnitude pruning
• nvidia_2_4: NVIDIA-style 2:4 structured pruning, two zeros in every four input weights
• wanda: WANDA pruning with calibration activations from the matching SCENIC JSON
• gradient: Taylor-style gradient pruning using (weight * grad).abs() over matching SCENIC calibration rows

By default, the reference-method launcher now prunes encoder transformer nn.Linear.weight tensors and leaves the response classifier dense. This is the safer default for the encoder-only SFT classifier, because the classifier is the full response-label head. The pruned checkpoints are written under:

runs/scenic-pruned50-reference-methods/

The combined comparison table is written to:

eval_results/scenic_sft/pruned50_reference_methods/reference_methods_summary.csv
eval_results/scenic_sft/pruned50_reference_methods/reference_methods_summary.json
eval_results/scenic_sft/pruned50_reference_methods/reference_methods_debug_report.json

Audit active/nonzero parameters in the generated reference-method checkpoints:

python scripts/audit_scenic_active_parameters.py

The final summary is validated across all requested methods. With the default four methods it should contain 16 outcome rows: four evaluation rows per pruning method, including two rows for each contrastive-anchor pruned checkpoint. Each row records the exact pruned checkpoint and prune_summary.json used for that evaluation. The launcher also evaluates the unpruned SFT checkpoints first and embeds those baseline rows in reference_methods_debug_report.json, so the same file shows whether a low score came from the source checkpoint or from pruning.

Use reference_methods_debug_report.json as the single file to share for debugging low pruning outcomes. It includes all method/model/dataset metrics, unpruned baseline metrics, compact prune summaries, collapse diagnostics, and wrong-prediction samples.

Useful overrides:

METHODS=magnitude,nvidia,wanda,gradient ./scripts/prune_scenic_sft_reference_methods_50_eval.sh
PRUNE_SCOPE=encoder-linear ./scripts/prune_scenic_sft_reference_methods_50_eval.sh
INCLUDE_CLASSIFIER=0 ./scripts/prune_scenic_sft_reference_methods_50_eval.sh
CALIBRATION_BATCHES=128 CALIBRATION_BATCH_SIZE=8 ./scripts/prune_scenic_sft_reference_methods_50_eval.sh
REINIT_CLASSIFIER=0 ./scripts/prune_scenic_sft_reference_methods_50_eval.sh

Because SCENIC encoder SFT is response selection, not generation, low accuracy after pruning cannot come from late stopping or missing EOS generation. The reference pruning launcher now reinitializes the dense response classifier from response texts after pruning by default, which tests whether pruning mostly broke the encoder/classifier alignment. The normal command is:

./scripts/prune_scenic_sft_reference_methods_50_eval.sh

If this recovers accuracy, the main failure mode is classifier alignment after encoder pruning. If it does not, the pruned encoder representation itself has lost too much task information at that sparsity. To reproduce the old no-reinit control, run with REINIT_CLASSIFIER=0.

For the older apples-to-apples all-linear T5-style surface, use:

PRUNE_SCOPE=all-linear INCLUDE_CLASSIFIER=1 REINIT_CLASSIFIER=0 ./scripts/prune_scenic_sft_reference_methods_50_eval.sh

NVIDIA-only ONNX edge inference water test

To test the NVIDIA 2:4 path separately through ONNX, run:

./scripts/run_scenic_onnx_nvidia_eval.sh

For the simplest end-to-end encoder-only run from a base encoder checkpoint, use:

./scripts/run_encoder_only_base_nvidia_fp16_int8.sh /path/to/encoder-base-checkpoint

This writes a generated 5-epoch SFT config, trains the encoder-only SCENIC SFT model, applies NVIDIA 2:4 pruning, exports ONNX FP16 and ONNX INT8 dense and pruned variants, exports a dense ONNX FP32 baseline for precision benchmarking, evaluates benchmark/training EM@1 and EM@5 on CUDA, runs ONNX batch-1 latency checks by default, runs the paper-facing FP32/FP16/INT8 ONNX precision benchmark, and writes:

eval_results/scenic_sft/encoder_only_nvidia_fp16_int8/encoder_only_fp16_int8_table.json
eval_results/scenic_sft/encoder_only_nvidia_fp16_int8/encoder_only_fp16_int8_table.csv
eval_results/scenic_sft/encoder_only_nvidia_fp16_int8/encoder_only_fp16_int8_table.tex
eval_results/scenic_sft/encoder_only_nvidia_fp16_int8/onnx_precision_benchmark/onnx_precision_benchmark.md
eval_results/scenic_sft/encoder_only_nvidia_fp16_int8/onnx_precision_benchmark/onnx_precision_benchmark.csv
eval_results/scenic_sft/encoder_only_nvidia_fp16_int8/onnx_precision_benchmark/onnx_precision_benchmark.tex

Useful overrides for the simple run:

TOKENIZER_PATH=/path/to/tokenizer ./scripts/run_encoder_only_base_nvidia_fp16_int8.sh /path/to/base
TRAIN_WITH_TORCHRUN=0 ./scripts/run_encoder_only_base_nvidia_fp16_int8.sh /path/to/base
RETRAIN=0 OVERWRITE=1 ./scripts/run_encoder_only_base_nvidia_fp16_int8.sh /path/to/base
GPU_IDS=0,1,2,3,4,5,6,7 PROVIDERS=cuda EDGE_PROVIDERS=cuda ./scripts/run_encoder_only_base_nvidia_fp16_int8.sh /path/to/base
RUN_EDGE_BENCHMARK=0 ./scripts/run_encoder_only_base_nvidia_fp16_int8.sh /path/to/base
RUN_ONNX_PRECISION_BENCHMARK=0 ./scripts/run_encoder_only_base_nvidia_fp16_int8.sh /path/to/base
PROVIDERS=auto EDGE_PROVIDERS=auto FP16_EXPORT_DEVICE=auto PARALLEL_GPU_EVAL=0 PARALLEL_GPU_BENCHMARK=0 ./scripts/run_encoder_only_base_nvidia_fp16_int8.sh /path/to/base
PRECISION_BENCHMARK_DEVICE_NAME="Jetson Orin Nano" ./scripts/run_encoder_only_base_nvidia_fp16_int8.sh /path/to/base
PRECISION_BENCHMARK_POWER_LOG=/path/to/power.csv ./scripts/run_encoder_only_base_nvidia_fp16_int8.sh /path/to/base

The real evaluation defaults require onnxruntime-gpu, exports FP16 on CUDA, and fans out the eight accuracy jobs across GPU_IDS: dense/pruned FP16/INT8 times benchmark/training retention.

The existing-checkpoint ONNX helper uses the training-dataset SFT checkpoint by default, builds or reuses one encoder-linear NVIDIA 2:4 checkpoint with the response classifier kept dense, then exports and evaluates four ONNX variants:

• fp16_dense
• fp16_nvidia_2_4
• int8_dense
• int8_nvidia_2_4

Each variant is evaluated on:

• benchmark: data/scenic/iot_instruction_benchmark_200.json
• training retention: data/scenic/SCENIC_full_training_dataset.json

The combined EM1/EM5 table is written to:

eval_results/scenic_sft/onnx_nvidia/onnx_nvidia_summary.json
eval_results/scenic_sft/onnx_nvidia/onnx_nvidia_summary.csv

The default runner only exports/evaluates ONNX FP16 and ONNX INT8 variants. The extra PyTorch/TensorRT edge report is opt-in. If enabled with RUN_EDGE_BENCHMARK=1, it is written to:

eval_results/scenic_sft/onnx_nvidia/edge_fp16_report.json
eval_results/scenic_sft/onnx_nvidia/edge_fp16_report.csv
eval_results/scenic_sft/onnx_nvidia/fp16_deployment_table.tex

That report is batch-size 1 by default and includes:

• runtime path: PyTorch FP16, ONNX Runtime FP16, and TensorRT FP16 when available
• mean latency in ms/query
• p95 latency in ms/query
• throughput in queries/second
• process GPU memory when nvidia-smi can report it
• PyTorch CUDA allocated/reserved memory for PyTorch rows
• CPU RSS peak
• source model/ONNX size, plus TensorRT engine-cache size when available
• benchmark EM@1 and EM@5 on the 200-row benchmark
• fixed input length, defaulting to both 64 and 128

The LaTeX table is rendered in the final paper format:

\begin{table}[t]
\centering
\caption{FP16 Deployment Feasibility Across Compact Architectures}
\label{tab:fp16_deployment}
\begin{tabular}{lcccccc}
\hline
Architecture & Runtime & Seq. Len. & Latency & P95 Lat. & Memory & EM@1/EM@5 \\
\hline
Encoder-only & PyTorch & 64  & -- & -- & -- & -- \\
Encoder-only & TensorRT & 64  & -- & -- & -- & -- \\
Decoder-only & PyTorch & 64  & -- & -- & -- & -- \\
Decoder-only & TensorRT & 64  & -- & -- & -- & -- \\
Encoder--decoder & PyTorch & 64  & -- & -- & -- & -- \\
Encoder--decoder & TensorRT & 64  & -- & -- & -- & -- \\
\hline
\end{tabular}
\end{table}

Run the renderer directly if you need to rebuild only the table:

python scripts/render_fp16_deployment_table.py \
  --encoder-report eval_results/scenic_sft/onnx_nvidia/edge_fp16_report.json \
  --seq-len 64 \
  --output eval_results/scenic_sft/onnx_nvidia/fp16_deployment_table.tex

The encoder-only rows are filled from the encoder FP16 edge report when present. Decoder-only and encoder-decoder rows remain -- unless you pass matching reports with --decoder-report and --encoder-decoder-report.

The exported ONNX files are written under:

runs/scenic-onnx-nvidia/onnx/

Useful overrides:

PROVIDERS=cuda ./scripts/run_scenic_onnx_nvidia_eval.sh
REBUILD_PRUNED=1 ./scripts/run_scenic_onnx_nvidia_eval.sh
DENSE_CHECKPOINT=/path/to/checkpoint ./scripts/run_scenic_onnx_nvidia_eval.sh
RUN_INT8=0 ./scripts/run_scenic_onnx_nvidia_eval.sh
RUN_EDGE_BENCHMARK=1 EDGE_INPUT_LENGTHS=128 EDGE_MEASURE_QUERIES=1000 ./scripts/run_scenic_onnx_nvidia_eval.sh
RUN_EDGE_BENCHMARK=1 RUN_TENSORRT=1 ./scripts/run_scenic_onnx_nvidia_eval.sh

The INT8 variants use ONNX Runtime dynamic weight quantization. The NVIDIA 2:4 variants preserve the pruned zero pattern in the exported weights, but actual speedup still requires an inference engine/kernel that exploits 2:4 sparsity such as a suitable NVIDIA/TensorRT sparse path. TensorRT rows require an onnxruntime-gpu build that exposes TensorrtExecutionProvider; otherwise the runner skips them when RUN_TENSORRT=auto.

Evaluate both checkpoints on the same local JSON file and write exact-match summaries:

python scripts/eval_scenic_sft_local.py \
  --json data/scenic/iot_instruction_benchmark_200.json \
  --checkpoint runs/scenic-sft-training-dataset/latest \
  --output eval_results/scenic_sft/training_dataset_predictions.jsonl \
  --summary-output eval_results/scenic_sft/training_dataset_summary.json \
  --dtype auto

python scripts/eval_scenic_sft_local.py \
  --json data/scenic/iot_instruction_benchmark_200.json \
  --checkpoint runs/scenic-sft-contrastive-dataset/latest \
  --output eval_results/scenic_sft/contrastive_dataset_predictions.jsonl \
  --summary-output eval_results/scenic_sft/contrastive_dataset_summary.json \
  --dtype auto

Evaluate a different local JSON file:

python scripts/eval_scenic_sft_local.py \
  --json /path/to/eval.json \
  --checkpoint runs/scenic-sft-training-dataset/latest \
  --output eval_results/scenic_sft/eval_predictions.jsonl \
  --summary-output eval_results/scenic_sft/eval_summary.json

For no-flag usage, edit these constants at the top of scripts/eval_scenic_sft_local.py:

LOCAL_JSON_PATH = "data/scenic/iot_instruction_benchmark_200.json"
CHECKPOINT_DIR = "runs/scenic-sft-training-dataset/latest"
OUTPUT_PATH = "eval_results/scenic_sft/benchmark_200_predictions.jsonl"
SUMMARY_OUTPUT_PATH = "eval_results/scenic_sft/benchmark_200_summary.json"
EVAL_DTYPE = "auto"

The evaluator accepts JSON lists with either prompt + response rows or anchor + response rows. It prints exact-match accuracy, top-5 accuracy, label-space coverage, writes per-row predictions, and saves a summary JSON containing exact_match_accuracy, prediction-collapse diagnostics such as prediction_unique_count and top_prediction_share, plus grouped accuracy by difficulty, task_type, and source when those fields exist.

Smoke Run

python scripts/train.py --config configs/smoke.yaml

The H20 config uses 8 GPUs, BF16, DeepSpeed ZeRO-1 with FusedAdam, SDPA attention, a 29,298-token Chinese BPE tokenizer with <|mask|>, 512-token bidirectional blocks with a CLS-style prefix, and 15% dynamic masked language modeling. The model is approximately 0.194B parameters, close to the intended 0.2B class. The per-GPU microbatch is set to 128 with gradient accumulation 2, matching the decoder recipe's 1,048,576 input tokens per optimizer step while giving the encoder shorter, high-throughput sequences.