Skip to content

BaseRT engine 0.2.5

Choose a tag to compare

View all tags
@prabod prabod released this 17 Sep 05:21
· 4 commits to main since this release
v0.2.5
3b31090
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

BaseRT 0.2.5

Tool calls stream as they are written, and convert-on-pull no longer fails
after a complete download. An agent UI used to sit blank while a tool call
was generated, then receive the whole thing at once; the call now arrives the
way OpenAI clients reassemble it. And a convert-on-pull of a repo whose
weights are 32 MB or larger — which is all of them — failed at the end with
no .safetensors shards despite having downloaded every byte. (basert pull
still accepts only supported architectures backed by safetensors.)

Scope. 0.2.5 is a patch release cut from main: streaming tool calls,
the convert-on-pull fix, and benchmark-harness telemetry.

Added

Streaming tool calls arrive in parts

/v1/chat/completions with stream: true now delivers a tool call as an
opening delta.tool_calls entry carrying index, id and function.name,
then further entries on the same index carrying only function.arguments
fragments, then finish_reason: "tool_calls".

{"index":0,"id":"call_…","type":"function","function":{"name":"get_weather","arguments":""}}
{"index":0,"function":{"arguments":"{\"city\":\"Paris\""}}
{"index":0,"function":{"arguments":",\"unit\":\"celsius\""}}
{"index":0,"function":{"arguments":"}"}}

The function name comes first and cannot be streamed before it is known — the
server learns it by parsing the call body — so the opening entry waits for it
and everything after that streams. How much streams depends on how the model
writes calls: a JSON body streams its raw arguments bytes, and an XML-dialect
body (Qwen3.5/3.6, Qwen3-Coder) streams one argument per parameter, as each
parameter's value is settled. Gemma's native call:fn{k:v} form and the ATEM
dialect are not JSON and still arrive whole when the call closes.

A call cut short by the token budget keeps finish_reason: "length" — including
when earlier calls in the same response completed, so a client never sees
"tool_calls" alongside a half-written call it cannot run. The
fragments already sent stand, but nothing is invented to close them.

Benchmark harness: telemetry from the measured run

basert-benchmark-harness now reports power, energy, temperature and memory
observed during the timed repetitions themselves rather than from a separate
pass, and --headline-first runs PP512 and TG128 first in one freshly loaded
model with a fixed 4,096-token reservation, before the larger prefill sweep —
so the headline numbers are not read off a machine already heated by them. The
report records the order and capacities it actually used. basert-bench is
unchanged, as are ordinary harness invocations.

Also new: --isolated-workloads (each workload in its own process at its own
minimum context capacity), --workload {all|prefill|decode|headline} and
--idle-baseline. -c/--ctx is rejected with --headline-first and with
--isolated-workloads, since those profiles choose the capacity themselves.

Fixed

  • basert pull <org/model> from a Hugging Face repo failed with
    no .safetensors shards after downloading the whole model. The resumable
    downloader used for files of 32 MB or more left the weights out of the
    snapshot the converter reads. A pull that already downloaded the model
    resumes and converts without fetching it again.
  • Energy figures from the benchmark harness counted the probes around a
    workload, not just the workload: on CUDA and ROCm the vendor telemetry
    commands landed inside the measured interval while average power divided by
    the shorter one. Both counters are now read next to the timed repetitions.

Changed

  • Benchmark report schema (breaking for report consumers). Same-run
    telemetry is basert-telemetry/4; the 0.2.4 /2 and /3 sections are
    gone, and the harness now fails a run whose workload does not carry /4
    rather than degrading. The new throughput profiles add
    basert-throughput-protocol/1, /2 and basert-bench-capacity/1. Anything
    parsing a 0.2.4 harness report needs updating; numbers from the new
    --headline-first / --isolated-workloads profiles are not comparable to
    shared-context 0.2.4 numbers, and --telemetry now samples inside the timed
    window, so its throughput is not comparable to 0.2.4 --telemetry
    throughput either.
  • The benchmark harness executable is rebuilt whenever it is missing, rather
    than only when its sources change. A packaged build could otherwise omit it.
Assets 6
Loading