Skip to content

fmev v0.1.5

Latest

Choose a tag to compare

@mrubens mrubens released this 21 Oct 19:00
· 2 commits to main since this release
  • New opt-in FMEV_RUNNER_UP=1: Choice questions with 3+ options get a second question to find the model's runner-up, which gets a larger share of the remaining probability instead of an even split. On jevbench's public items the runner-up was the right answer on 21 of 45 misses (47%, against 34% by chance). The chosen option and its probability are unchanged. It costs 1.5–2× latency, so it's off by default.
  • Score questions now give the remaining probability mostly to the levels next to the chosen one, so expected scores are no longer pulled toward the middle of the scale.
  • The chosen option is always kept strictly above the others, so it never ties.
  • Install: brew install mrubens/tap/fmev

SHA-256 (fmev-0.1.5-macos-arm64.zip): 73eee4df9379d97d66e659e1f05bba463893f5254c3235cffb566ca0d68356f2