Native typed-decision model training (task: decision_model) — worth pursuing, or out of scope?

[Ru0k3]

Hi — I've been using Soup for RL fine-tuning and wanted to float something before building more of it than makes sense.

I opened #1187 adding training.laya_reward, which lets a Laya/Jev-style typed-decision model (score/noul answers, not generated text) be used as a GRPO/PPO reward source. That's a small addition using Soup's existing reward_fn seam.

The bigger question is whether Soup should go further and let people train a typed-decision model natively — task: decision_model — instead of needing Laya's separate Kaggle notebook. I've done a source-level feasibility pass on both repos and wanted to check in before writing more code, because this is a real architectural departure, not a small extension:

• The model returns option logits from a bidirectional encoder (AutoModel), not autoregressive token logits — closer to Soup's embedding task than to grpo/ppo.
• The reward is a proper-scoring-rule objective (log + spherical, plus ranked-probability-score for ordinal questions), computed over sampled logit perturbations — not trl.GRPOTrainer's completion-based reward loop.
• It needs its own data schema (state + typed questions + gold distributions), its own collator, and a post-training calibration-fitting step.

I'd also plan to preserve Laya's native checkpoint layout exactly, so anything trained this way would drop straight into #1187's laya_reward adapter with no bridging code needed.

Given the scope, this isn't one PR — it'd need to land as 3–4 smaller ones (checkpoint/data loading → training loop → calibration → CLI dispatch), each independently reviewable.

Before I put more time in: is this in scope for Soup at all, or do you see non-generative/decision-model training as outside what this project wants to own? And if it's in scope, would you rather see it as a first-class task:, or is some kind of plugin/extension point a better fit long-term?

Happy to share the full feasibility writeup if useful.

[MakazhanAlpamys]

Thanks for asking before building, @Ru0k3; for something this size that is exactly the right order.

A new, non-generative training task (task: decision_model: an encoder returning option logits, a proper-scoring-rule objective, its own data schema and a calibration step) changes what the project owns, so it is a scope decision for the project owner rather than a review call. I have raised it. Please don't start on it until there is an answer here.

The same applies to #1187: the review there stands, and it needs a tracking issue with the scope agreed before the laya_reward field and the new extra can be considered.

[MakazhanAlpamys]

@Ru0k3, the decision on the scope question: not now. A native task: decision_model would add a new, non-generative task area to Soup (an encoder returning option logits, a proper-scoring-rule objective, its own data schema and a calibration step), plus a dependency that is only a week old on PyPI. That is more than the project wants to take on before v0.76.0, so it will be revisited after that release. #1187 stays blocked on the same decision; I've said so there.

Thank you for writing up the feasibility pass before building it; it made this an easy question to answer, and it will be the starting point when this comes back.