Skip to content

jevos-v5

Latest
Latest

Choose a tag to compare

View all tags
@LosaLosSantos LosaLosSantos released this 16 Oct 23:53
jevos-v5
67c7846
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Yes/no decisions on a laptop CPU in 28–130 ms: jev, one binary for Windows, Linux and macOS, now with jevos-v5 at 8-bit weights. 78.9% on the benchmark's headline yes/no sets and 74.0% on agent instructions (DynaBench); choices 73.5%, scores 48.3%. On the 999 hand-written yes/no questions it gets 79.6%, on held-out rules 88.8%.

Questions asked in the negative (#44), such as "Is this not about X?", are now read as negated: on 1,398 plain and negated questions written for that issue, jevos-v5 gets 96.7% of the negated ones right and 99.0% of the plain ones. It was trained on negated questions written the same way, so this checks the fix. Public negation sets test something else, a negation inside the text: there it gets 61.7% on CondaQA and 74.1% on the negation NLI sets of Hossain et al., behind larger models.

Where it is weaker: the support cases of typed_decisions (yes/no 64.7%) and PubMedQA (70.4%); it is also fairly confident when a needed fact is missing from a rated policy. Check it on your own questions before relying on it.

jevos-v5 is a training run of the 17-layer model on a wide mix: policies, records and contract excerpts written as natural text, and yes/no questions asked in the negative; a short second pass on the natural-text part was then averaged with the first run. Measured through jev itself (OpenVINO INT8).

Download the archive for your system and the model, and unpack the model into the jev folder:

tar -xzf jev-linux-x64.tar.gz                 # Windows: unzip jev-windows-x64.zip
cd jev
unzip ../jevos-v5-openvino-int8.zip           # creates model/
./jev serve                                   # Windows: jev.exe serve
File What it is
jev-windows-x64.zip, jev-linux-x64.tar.gz (glibc 2.35+), jev-macos-arm64.tar.gz the jev binary with OpenVINO's libraries and the licenses
jevos-v5-openvino-int8.zip the model jev runs, as model/
jevos-v5-q4_k_m.gguf, jevos-v5-q8_0.gguf the same model as GGUF, for llama.cpp and the tools built on it
SHA256SUMS.txt the SHA-256 of every file here

The API, options and numbers are in the README.

Assets 9
Loading