Remarkable results after targetted training

[Zei33]

This project caught my eye after the recent release of Jev, and it piqued my interest immediately. I already intended to integrate Jev into a couple of my projects, but I was curious to see how an open source project built in a week could perform.

My first tests were not impressive. On the provided checkpoints, Jev significantly outperforms with tasks that are more novel or diverse (e.g. not basic customer service). What I realised as I experimented was that Jev's biggest advantage is that it's a 'one-shot' model. You don't need to bring anything to the party, you ask your questions and get great answers.

I was curious though. How would Laya compare if you actually trained it specifically for the tasks you require it to do in a specific project? So I did just that, I used Qwen 3.8 27B FP8 to train it up for a specific project's needs. The outcome was interesting. I quizzed my version and Jev on the same exact 2495 questions, and then compared the results (specifically NOT using Jev to refine my version). What I found is that my version practically tied or outperformed Jev in the tasks I trained it to perform.

The one area of advantage that Jev had and I was unable to match with Qwen 3.8 was out-of-the-box calibration. Qwen's confidence levels were all over the place, but primarily extremely overconfident. Jev's confidence levels are spot on and I think would be difficult to match, I can see why they advertise that as one of the most important features.

Family	Jev	Qwen Teacher	Verdict
S1 naturalness (Spearman)	0.446	0.452, 0.489 with two-pass averaging	Tie
S2 damaged (AUROC)	0.900	0.920	About level; Qwen slightly ahead
S3 translation faithful (AUROC)	0.983	0.972	Level
P1 repair picker (set accuracy)	0.750	0.874	Qwen better, significant
P2 any acceptable (AUROC)	0.849	0.878	Level
G1 grammar, human JCoLA labels (AUROC)	0.766	0.782	Level
Calibration (ECE, lower is better)	0.03-0.08	0.05-0.25	Jev much better

I won't go into the meaning of all this, it would be a long explanation. But the short of it is, I'm using it to judge the quality of Japanese sentences, and then having it make decisions about the best alternate versions. The big area of weakness in these two is that neither model can judge the naturalness of the Japanese very well. I would be fascinated to see if a much more powerful model would end up with significantly better results here. The big surprise was the Qwen/Laya decision making. It makes much better choices from options than Jev does, at least on this particular task.

A couple of other notes. Laya is very cheap to run. It can fit in nicely to an enterprise-level stack, and even theoretically sit inside a monorepo with a version specifically trained for that project. The other advantage is response time. As an Australian, I have to deal with latency from the US with Jev, and also the reality that anything I send to them is theoretically a security risk. Using Laya means cutting the latency from 300ms to 30ms or less.

I will say though, Jev is astonishingly cheap. I could fire off 6000 questions to do performance comparisons, and it didn't even cost 3 cents. Quite incredible.

My verdict on the laya project (as it is right now) breaks down into a few points:

• Do you have the time, skill and resources to train your own checkpoint?
• Is your use-case novel, requires multilingual usage, or requires very specific knowledge?
• Is user privacy a concern for you and you're outside the US?

If you answer the first and either of the second, then I think it's worth the trouble. However, I really think in 95% of cases, you're better off going with Jev. The price is so ridiculously cheap, you wouldn't even notice it. And it's so easy and reliable in general, it's going to be hard to justify deploying your own. I also expect they'll keep improving their model, so it's only going to get better over time.