Skip to content

v0.3.1: evaluation on public data

Choose a tag to compare

@HayateV30 HayateV30 released this 02 Oct 19:45
· 20 commits to main since this release

LUAR was tested on 300 texts from each of two public datasets labeled by people. The manual now uses these numbers instead of figures from our short hand-written examples.

Laya LM Studio (Qwen 3.5 4B)
News topic (English, AG News) 93% 89%
Would recommend? (Portuguese reviews, B2W) 70% not run
Sentiment pos/neutral/neg (B2W) 68% (neutral: 5%) not run
Star rating 1–5, score (B2W) 26% (60% within one star) not run
Time per row ~1 s ~8 s per question

Method, script and full results: bench/.

Changes

  • Score questions: low-confidence score answers no longer mark rows for review. On real reviews they marked 97% of rows; a missing answer still counts.
  • Manual:
    • Speeds and accuracy: real figures, with Laya recommended as the default engine.
    • New tip: on "middle" options such as neutral.
    • New note: what the confidence is worth on different kinds of data.
  • Evaluation script: saves results after each dataset.