Skip to content

v0.80.2

Choose a tag to compare

View all tags
@github-actions github-actions released this 09 Sep 08:12
· 22 commits to main since this release
v0.80.2
68a457e

openglacier Core v0.80.2

Native tool calling, GPU-aware local inference, and a much more deliberate Agent execution loop.

openglacier Core v0.80.2 deepens the local AI path introduced in v0.80.0.

The LLM runtime now integrates more directly with llama.cpp common/chat for tool-aware prompt rendering and response parsing, adds explicit CPU/CUDA device selection, exposes thinking policy, and can reuse prompt-prefix state across multi-step Agent runs.

The Agent also moves away from an ad-hoc model-output protocol toward structured tool definitions and native tool calls.

Database-oriented runs gain collection description, bounded evidence gathering, multiple read-only analysis queries, separate synthesis budgets, and more careful context management.

Highlights

Tool calling becomes a first-class LLM contract

llm.generate can now receive structured tool definitions and tool-choice policy.

The request model supports:

tools
toolChoice
messages[].toolCalls
messages[].toolCallId
cacheKey

and generation can emit structured:

toolCall

events alongside normal text.

The Agent no longer needs to instruct a model to manufacture a private JSON envelope for every tool invocation.

llama.cpp common/chat integration

v0.80.2 introduces a native bridge around the llama.cpp common/chat implementation.

The bridge is used to:

  • apply model chat templates;
  • pass structured tool definitions;
  • apply tool-choice policy;
  • preserve generation prompts;
  • expose stop conditions and parser metadata;
  • parse generated content;
  • extract native tool calls;
  • separate reasoning content when supported.

Conceptually:

OpenGlacier messages + tools
          │
          ▼
llama.cpp common/chat
          │
          ▼
model-specific prompt
          │
          ▼
generation
          │
          ▼
common/chat parser
          │
          ├── assistant content
          └── structured tool calls

This delegates model-specific chat behavior to the same machinery used by llama.cpp instead of rebuilding template conventions independently in Rust.

Structured tool calling

The LLM request contract now supports tool definitions containing:

name
description
parameters

Tool choice can be:

auto
none
required

Assistant history can also carry structured tool calls, while tool results are linked back through a tool-call id.

This makes multi-turn tool use representable as normal chat history rather than as opaque text.

Process-safe tool parsing

The current runtime uses common/chat for prompt rendering and native tool-call parsing.

For the pinned llama.cpp generation path in this release, token-level tool grammar installation is intentionally not relied upon when it can threaten process stability.

Tool-aware templates and the common/chat parser remain active.

The practical tradeoff is:

native tool format + parser

instead of:

unsafe in-process grammar enforcement

until the underlying grammar path is safe for the daemon.

Thinking policy

Local generation now has an explicit thinking policy:

auto
on
off

configured through:

OGD_LLM_THINKING

auto keeps the runtime policy tool-safe:

  • native/template behavior for ordinary turns;
  • thinking disabled for tool-oriented turns.

on and off can force the behavior when a model and template support it.

The selected policy is also exposed through LLM status.

CPU and CUDA device selection

The local LLM runtime can now choose its execution device explicitly.

Configuration is available through:

OGD_LLM_DEVICE

with:

auto
cpu
cuda

auto uses GPU offload when the runtime reports a usable CUDA device and otherwise falls back to CPU.

cpu disables GPU layers explicitly.

cuda requires a usable CUDA backend and visible CUDA device, and fails with an actionable error when that requirement cannot be satisfied.

GPU layer control

GPU offload can be tuned with:

OGD_LLM_GPU_LAYERS

Accepted values are:

auto
all
<number>

This allows deployments to leave layer placement to llama.cpp, offload the complete supported model, or request a specific layer count.

CUDA build features

CUDA support remains a compile-time choice.

New feature profiles include:

cuda
cuda-no-vmm
full-cuda

For example:

cargo build --release --features full-cuda

A binary built without a usable GPU backend can still run the LLM service on CPU.

Explicit:

OGD_LLM_DEVICE=cuda

is reserved for deployments that require CUDA rather than merely preferring it.

Prompt-prefix reuse

Multi-step Agent runs often send the same long prompt prefix repeatedly:

system instructions
conversation
previous tool history

followed by a relatively small new suffix.

v0.80.2 can snapshot and reuse ephemeral llama.cpp sequence state for these repeated prefixes.

Conceptually:

turn 1
[ stable prefix ][ new content ]
       │
       └── capture sequence state

turn 2
[ same stable prefix ][ different suffix ]
       │
       ├── restore cached state
       └── decode suffix only

This cache is designed for reuse inside an Agent run rather than as durable conversation memory.

Bounded prefix cache

The host-memory budget is controlled with:

OGD_LLM_PREFIX_CACHE_MAX_BYTES

The default budget is:

1 GiB

Setting it to:

0

disables prefix-state caching.

Generation statistics can report:

cachedPromptTokens

so callers can observe how much prompt work was reused.

A more capable Agent tool contract

The Agent tool set now contains:

database.collections
database.describe
database.query
files.list

The new:

database.describe

tool allows the Agent to inspect a schemaless collection before inventing field names.

It returns the scoped collection size together with field paths and JSON types observed from a bounded prefix of visible documents.

Observed fields are treated as evidence, not as a strict schema guarantee.

Better database analysis

The Agent now distinguishes between:

looking at a few rows

and:

reasoning about a collection

For summaries, distributions and analytical questions, it can:

  1. discover or describe the collection;
  2. issue targeted read-only queries;
  3. gather compact evidence;
  4. perform a final synthesis turn.

This avoids treating:

limit 20

as if those 20 documents represented the complete collection.

Native query discipline

Agent database access continues to use the openglacier pipeline language.

Queries must start from the native DSL shape:

on <collection>

For example:

on orders
| group status
| sort count desc
| limit 20

The Agent is explicitly guided away from:

SQL
MongoDB syntax
JSON query objects

and from mutating operations.

System collections remain inaccessible through Agent database tools.

Bounded analytical loops

Database analysis now has its own query budget.

The default maximum is:

4

and can be configured with:

OGD_AGENT_MAX_DATABASE_QUERIES

The upper accepted bound is deliberately finite.

This gives the Agent room to gather more than one piece of evidence without allowing an open-ended query loop.

Separate planning and synthesis budgets

Tool-selection turns and final-answer turns do not have identical needs.

v0.80.2 therefore adds a separate synthesis budget.

The default final synthesis allowance is larger than the normal planning budget and can be configured through:

OGD_AGENT_SYNTHESIS_MAX_TOKENS

An explicit request-level maxTokens remains a hard cap.

This lets short tool-planning turns stay compact while giving the final answer enough room to summarize the evidence.

Smaller tool context

The default maximum tool-result context is reduced from:

32 KiB

to:

8 KiB

Database evidence is compacted even more aggressively between analytical turns.

The Agent keeps a small evidence ledger instead of carrying every verbose tool response indefinitely.

This is particularly important for small local models with limited context windows.

Adaptive context handling

When a synthesis request exceeds the model context window, the Agent can now adapt.

It can:

  • reduce the generation budget;
  • compact database-derived evidence;
  • shrink retained tool output;
  • retry with a smaller context footprint.

Only tool-derived database context is compacted by these emergency paths.

The user conversation and execution scope remain conceptually distinct from this temporary evidence reduction.

More deliberate tool choice

The Agent can now narrow the tool set and tool-choice policy according to the request.

Examples include requiring:

database.collections

for collection discovery, or:

database.describe

before a schema-oriented answer.

Analytical requests can require actual:

database.query

execution before synthesis.

Final synthesis turns use:

toolChoice = none

so the model is asked to answer from evidence rather than start another tool loop.

Agent provider timeout

Delegated Agent capability calls can legitimately wait longer than ordinary request/response operations, especially during local model generation.

A dedicated read timeout is now available through:

OGD_AGENT_PROVIDER_READ_TIMEOUT_MS

This separates long-running Agent provider behavior from the normal daemon connection timeout.

LLM and Agent debugging

Dedicated debug topics are now available for:

llm
agent

These paths expose more information around:

  • provider selection;
  • device selection;
  • GPU offload;
  • common/chat parsing;
  • tool calls;
  • prefix-cache behavior;
  • Agent orchestration.

Secret redaction continues to apply to debug output.

Compatibility

The new LLM request fields are additive to the capability introduced in v0.80.0.

Plain text generation remains valid without tools.

CPU execution remains available without CUDA.

CUDA requires an appropriate build and runtime environment.

Agent runs remain stateless and retain the inherited Place/AppInstance execution scope.

The most visible behavioral change is inside tool-oriented Agent execution:

the model is now driven through structured tool definitions and parsed native tool calls rather than the original private JSON directive protocol.

Examples

Build the full daemon with CUDA support:

cargo build --release --features full-cuda

Force CPU inference:

OGD_LLM_DEVICE=cpu \
cargo run --release --features full --bin ogd

Require CUDA and offload all supported layers:

OGD_LLM_DEVICE=cuda \
OGD_LLM_GPU_LAYERS=all \
cargo run --release --features full-cuda --bin ogd

Disable model thinking:

OGD_LLM_THINKING=off \
cargo run --release --features full --bin ogd

Disable prefix-state caching:

OGD_LLM_PREFIX_CACHE_MAX_BYTES=0 \
cargo run --release --features full --bin ogd

Increase the Agent database-analysis budget:

OGD_AGENT_MAX_DATABASE_QUERIES=8 \
cargo run --release --features full --bin ogd

In short

v0.80.2 makes the AI capabilities introduced in v0.80.0 substantially more native and more disciplined.

The LLM path gains:

common/chat · structured tool calls · thinking policy · CUDA selection · prompt-prefix reuse

and the Agent gains:

database description · bounded multi-query analysis · compact evidence · adaptive synthesis

The intent is not to turn the Core into an AI framework.

It is to let local models use existing openglacier capabilities through the same small, explicit operation contracts as everything else.

Full Changelog: v0.80.0...v0.80.2

Small core, long life.

Assets 40
Loading