Integration report: fine-tuned Laya as a LangGraph router, vs XGBoost, BERT and Gemini

[AdityaThakare72]

Hi! I built a small, reproducible benchmark that uses Laya as the routing step of an agent workflow, and I'm sharing
the results here since CONTRIBUTING.md treats benchmarks and integration reports as first class.

Repo (Apache-2.0, Colab notebooks): https://github.com/AdityaThakare72/typed-decision-models

Setup

• Task: route a support ticket. Input is text plus tabular metadata (account tier, channel, urgency score,
  past failures). Output is a typed decision mapped onto Laya's three question types:
  • route_target: choice, 6 teams
  • priority_level: score, low / medium / high / critical
  • requires_human: noul
• Data: 3,000 synthetic tickets with rule-based labels and 5% label noise. Splits are 2,099 train / 451 val / 450 test.
  The ticket is passed as a dict state.
• Fine-tuning: laya.train.finetune from convaiinnovations/laya with the defaults (RLCD loss), 2 epochs,
  micro_batch=8, grad_accum=4, on a Colab T4.
• Laya version: 0.4.11, with transformers / torch as installed on Colab.

Results (450-ticket test set; exact match = all three fields right)

model	exact match	route F1	priority F1	requires-human F1	p50 / p95 latency
convaiinnovations/laya, zero-shot	0.173	0.615	0.190	0.530	46 / 49 ms (T4)
laya-typed-decisions, zero-shot	0.296	0.674	0.302	0.574	46 / 52 ms (T4)
Laya fine-tuned (2 epochs)	0.720	0.958	0.679	0.916	47 / 54 ms (T4)
XGBoost, TF-IDF + tabular	0.736	0.951	0.641	0.924	17 / 19 ms (CPU)
bert-base-uncased, fine-tuned (3 epochs)	0.667	0.955	0.555	0.921	11 / 15 ms (T4)

As the router node in a LangGraph workflow (same graph, same 100 test tickets, only the router changes):

router	exact match	correct destination	router p50 / p95	cost per 1M tickets
Laya fine-tuned (T4)	0.76	0.90	45 / 66 ms	~$5 (GPU time)
Gemini gemini-3.1-flash-lite, structured JSON, policy + 12 examples	0.51	0.88	972 / 1,438 ms	~$381
Hybrid: Laya first, Gemini when Laya's least confident field < 0.8	0.71	0.89	68 / 1,148 ms	~$96

Observations

1. Fine-tuning is essential for a custom schema, and cheap. Zero-shot was near chance on our labels. Two epochs on
   about 2k tickets took held-out per-question accuracy from 0.545 to 0.911, as reported by finetune's own
   before/after evaluation.
2. Calibration after fine-tuning was excellent. Held-out ECE went from 0.173 to 0.015. On the test set, mean route
   confidence was 0.956 against 0.958 accuracy, so the refit temperatures work as advertised.
3. Best at the field that needs text and metadata together. Fine-tuned Laya had the best priority macro-F1 of all
   models (0.679). The dict state with numeric fields worked well without any feature engineering.
4. Versus an LLM router: about 25 points better on exact match, about 21x lower median latency, and about 80x cheaper.
   Gemini picked the team well but couldn't apply a numeric priority policy described in words: 57 of 100 priorities
   right, against 84 for Laya.
5. The single-request hot path is fast. agent.predict (one state) runs at about 45 ms p50 on a T4, and
   predict_batch is used for evaluation.

A small thing I noticed

Loading convaiinnovations/laya (and our fine-tune, which inherits it) emits:

RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5];
using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.

It only affects choice questions with 11 or more options, so it didn't touch our results (6 options). It may be
worth refitting or removing that bucket in the published rl_agent_config.json.

Caveats

• Synthetic data with rule-based labels. Tree models are well suited to learning the thresholds, which flatters XGBoost.
• One seed and one run.
• The LangGraph comparison uses 100 tickets, so the margin of error is about ±5 points.
• Latency compares CPU (trees) with GPU (transformers), and Gemini latency includes the network round trip from Colab.

Happy to contribute

If useful, I could open a focused PR, for example:

• a runnable example under examples/ showing how to fine-tune on a custom multi-question routing schema and plug
  the agent into a LangGraph router node, or
• a link from the docs to this benchmark as a community integration.

Let me know which, if any, would fit, and I'll follow CONTRIBUTING.md (conventional commits, ruff, one runnable check).