runtime-llama 0.13.0a1

Public prerelease of the exact qualified runtime-llama==0.13.0a1 wheel. The earlier cross-replica numerical finding remains open; 0.11.1 remains the latest stable release.

• Automatic character-budget calibration now measures 2,500 hard original synthetic texts, 500 in each of five categories, selected reproducibly from the original 10,000 texts using native Nomic counts. Category recommendations use the selected category's weighted characters/prepared-token mean at 80%; mixed uses the complete selected mean at 90%. Only an explicit budget request starts calibration.
• Direct embedding and native-500 recovery make no /tokenize calls. A failed POST is isolated through bounded subbatches on the existing scheduler. Successful child vectors are retained and not replayed by isolation.
• EmbeddingChunkSplitRequiredError returns all successful original indices plus lossless Unicode/UTF-8 midpoint halves for isolated rejected inputs. A 500 does not prove context overflow, and one halving may not be sufficient. The consumer retries only the requested halves, keeps completed vectors and owns fragment identity and retry limits. The calibrated mean and character budget never change during fallback.
• Ordered asynchronous windows preserve earlier results and remain usable after a caught split-required error. Owner cancellation, finite deadlines, identity/topology validation, result credits, backpressure and drained close remain in the same engine. New regressions cover first/all rejected inputs, shared owners, actual concurrent HTTP connections, formatting cancellation and sanitized rejection causes.

Consumer migration: the automatic path now emits the new partial-results error rather than the prior exact-diagnostic overflow error. Update the handler before adopting this prerelease; do not retry the entire original window or discard successful vectors. The declared-model exact preparation API remains available. See the migration contract and examples and API guide.

Verification: 2,031 tests, Ruff check/format, source/wheel builds and clean installed-wheel checks. One loaded Nomic replica, context/native batch/microbatch 512 and one slot, processed 511 texts in 2.5971 seconds, using 2 embedding POSTs, 10 metadata GETs and zero tokenizer requests. Cold calibration made exactly 2,500 tokenizer requests in 12.5469 seconds. Recovery retained two valid vectors; two halving rounds completed the rejected source without changing the mixed 1,101-character recommendation. Same-alias text/token-ID and retained-vector comparisons had maximum absolute error zero with unchanged thresholds. Client wall timings include transport and validation; they are not pure GPU times or a comparison with previous two-replica runs.

The existing FIND-000007 remains open. A single-replica live pass and local two-alias HTTP barrier tests do not establish numerical equivalence across physical replicas. No tolerance, loaded profile, server build or consumer configuration was changed for this publication.

The wheel is byte-identical to the tested artifact:

c1cb36802579be6bc58624f657f87ad64d7d7305218b3d8bd022b9635c56ce3f  runtime_llama-0.13.0a1-py3-none-any.whl

Its embedded README retains historical local-candidate wording. The current repository, this prerelease and RELEASE-MANIFEST.json record the later publication authorization. All 39 runtime payload files match the tagged source. Optional tsnet archives reuse the verified v0.6.0 executables and license inventory, with no new native OS/VPN qualification. Apache-2.0, Davide Mele copyright and upstream attribution are unchanged.

Development and delivery are managed with Flower MCP. See the qualification and publication record. Verify all seven assets with SHA256SUMS. This publication performs no consumer installation, job, service restart or PyPI upload.

runtime-llama 0.12.0a1

Public prerelease of the exact 0.12.0a1 wheel already exercised by the diagnostic consumer. The numerical finding between the observed remote embedding replicas remains open. This release does not claim complete numerical qualification; 0.11.1 remains the latest stable release.

• Initial automatic character recommendations now use the worst observed characters/token ratio across the existing 10,000-sample profile, keeping category 80% / mixed 90% factors. Valid raw calibration caches are reused without retokenization.
• Successful direct embedding still makes no /tokenize requests. After a direct HTTP 500 drains, bounded exact diagnostics inspect only that owner's inputs actually sent in the failed POST. Unproven overflow preserves the original error and does not change the budget.
• Confirmed overflow returns EmbeddingInputOverflowError with original input/window indices, a budget snapshot and complete lossless Unicode/UTF-8 spans, each counted with the actual prefix and special-token policy. The consumer owns reprojection and an explicit retry; the library never silently changes vector cardinality or replays the POST.
• Learned absolute ceilings persist with thread/process locking, monotone minimum updates, revisions and private atomic publication. Safety percentages are not applied again. refresh_character_budget() observes current limits and reads completed state without starting calibration.
• The existing scheduler, resource bounds, cancellation, finite deadlines, model identity checks and drained shutdown remain. Additional regressions cover actual subbatch provenance, duplicated occurrences, shared owners, async window ordinals, tokenizer failures and replica-limit drift hidden by an unchanged pool minimum.

Verification: 1,963 tests, Ruff check/format, source/wheel builds and isolated installation. The direct 511-text live check used both replicas with 4 embedding POSTs, 16 metadata GETs and zero tokenizer calls, in 1.5471 seconds; vectors matched the exact path identically. The original 531-token input on context 512 returned verified 512/25-token fragments. Only the actual failed POST's 128 rows plus two fragments were counted. A separately authorized clean CodingCastle diagnostic ingestion published 126,045 vectors with one atomic commit and no corpus errors, in 707.626 seconds. Timings are client/application wall times, not pure GPU time or a controlled speedup claim.

Open numerical limitation: five real inputs in the 257-input retry comparison exceeded unchanged cross-replica thresholds: maximum absolute difference 0.0024030989764955157 and minimum cosine 0.9998033728657948, versus 1e-5 / 0.999999. On each individual replica, strings and exact native token IDs produced identical vectors. The difference follows the replica; its hardware/kernel cause is not established. Flower FIND-000007 stays open, with no tolerance relaxation or pool change.

The published wheel is byte-identical to the delivered candidate:

SHA256 85a0d6f57a20be595dc818448066cf97694ccfe286b1b8c06d8774ea06518b04

Its embedded README retains historical local-candidate wording; current repository documents and RELEASE-MANIFEST.json record the subsequent publication authorization and consumer diagnostic evidence. All 38 runtime payload files match the current tagged source. Optional tsnet archives reuse the verified v0.6.0 executables and license inventory; no new native OS/VPN qualification is claimed. Apache-2.0, Davide Mele copyright and upstream attribution are unchanged.

Lifecycle authority and host-reported evidence are managed with Flower MCP. See the API guide, qualification and open finding and prerelease record.

Verify all seven assets with SHA256SUMS. This publication performs no consumer update or PyPI upload.

runtime-llama 0.11.1

Hardens the automatic embedding boundary without changing the existing scheduler, category 80% / mixed 90% recommendations, consumer configurations or model profiles.

• AutomaticEmbeddingConfig exposes existing finite embedding_max_request_bytes, embedding_max_response_bytes, embedding_max_pending_bytes and embedding_max_pending_tokens controls. Defaults are unchanged. Decoded HTTP response size remains separate from retained Python result capacity. An explicit bounded pending budget can accommodate high-context direct batches without invented token counts.
• http_connection_policy="reuse" selects the existing keep-alive policy for an owned client. Default remains "close"; injected clients retain policy and ownership. No retry or transport fallback is added.
• Calling draining close from the same service's cancellation callback now raises a typed diagnostic before shutdown mutation, preventing self-deadlock across synchronous, calibration and streaming paths. Callbacks must remain side-effect free; external cancellation and drained shutdown are preserved.
• Explicit cold character-budget calibration checks actual private persistence feasibility before remote tokenization. Valid readable caches do not require directory writes, final publication remains atomic, and late timeout failures retain their type.
• A verified single-model statusless catalog can be attested by coherent read-only root /props, sharing authority with the declared adapter. Router, sleeping, ambiguous, unverified and inconsistent cases remain rejected.
• The mixed recommendation's cross-category limitation is documented and tested. No statistical input admission, implicit tokenizer, silent truncation or exact-preparation fallback is introduced.

Verification: 1,811 tests, Ruff check/format, normalized source/wheel builds and fresh isolated installation. New regressions exercise a real loopback HTTP response with 128 vectors × 4,096 dimensions, default 8 MiB rejection and explicit 16 MiB success, decoded gzip bounds, incomplete bodies, no replay, ordered results and drain; owned/borrowed connection policy; high-context resource capacity/fairness; callback self-close in supervised subprocesses; cache feasibility/cancellation/read-only reuse; and statusless discovery.

Installed-wheel live qualification used both already loaded Nomic replicas and unchanged numerical thresholds. 511 direct texts before calibration: 1.5078 seconds, 4 embedding POSTs + 16 metadata GETs, zero tokenizer calls. Exact preparation plus embedding: 3.7297 seconds, 511 tokenizer POSTs + 4 embedding POSTs + 15 metadata GETs. All 511 vectors matched exactly. Only the explicit cold budget request performed 10,000 tokenizations, in 53.8941 seconds; category selection and warm reuse needed no additional counts. These are client wall times including applicable metadata, preparation, HTTP and server work, not pure GPU timings or a controlled version-speed comparison.

Lifecycle and host-reported evidence are managed with Flower MCP. See the configuration/API guide, hardening qualification and public release record.

The portable Python wheel's 36 runtime payload files match the live-qualified candidate and public source byte-for-byte. Optional tsnet archives retain v0.6.0 executables and license inventory; no new native build, OS execution, VPN enrollment or VPN inference is claimed. Apache-2.0, Davide Mele copyright, NOTICE and upstream attribution remain unchanged. Consumer installations, code/config/pins, services, models and databases were not changed. No PyPI upload is included.

RELEASE-MANIFEST.json binds source and artifact provenance; verify the seven assets using SHA256SUMS.

runtime-llama 0.11.0

Adds optional direct automatic embeddings while preserving the existing exact adapter APIs and one shared persistent engine/scheduler.

• AutomaticEmbeddingService owns verified loaded-model and replica discovery, task prefixes, ordering, bounded concurrency, backpressure, cancellation, finite deadlines and drained close. Ordinary initialization, availability, synchronous embedding, queries and async windows send strings directly without per-input /tokenize or calibration/cache prerequisites.
• Only an explicit character_budget() request loads or performs the optional 10,000-sample calibration. Categories are code, documentation, prose, structured and multilingual, 2,000 original texts each; mixed aggregates all 10,000. Single categories use 80% of measured characters/token; mixed uses 90%. Complete raw measurements persist independently of margins.
• The consumer owns splitting and can save its character recommendation once, then send strings on subsequent starts. The recommendation is empirical; actual tokenization/context admission stays with the native server. No silent truncation or failed POST replay is added. Existing declared-model exact prepare_* / embed_prepared contracts remain available.

Verification: 1,730 tests, Ruff check/format, normalized builds and clean isolated installation. All 36 runtime package files match stable source; 35 are byte-identical to the live-qualified 0.11.0a1 candidate, with only the version literal changing in __init__.py. Corpus contents are unchanged.

Retained bounded live comparison on two already loaded verified Nomic replicas: 511 direct inputs in 1.5626 seconds, four embedding POSTs + 16 metadata GETs, zero tokenizer requests, both replicas used concurrently. Exact preparation plus embedding took 3.9873 seconds, 511 tokenizer POSTs + four embedding POSTs + 15 metadata GETs. All vectors matched exactly with unchanged numerical thresholds. Only the explicit budget request then performed 10,000 tokenizations in 54.3861 seconds; category selection and warm reuse needed no more. These are client wall times including applicable preparation, discovery, transport and server work, not pure GPU time or a general throughput guarantee. Stable publication adds no new inference benchmark or model administration.

The project lifecycle and evidence are managed with Flower MCP. See the API and consumer example, engineering qualification and stable release record.

Install the portable Python wheel below. The three optional tsnet archives reuse byte-identical v0.6.0 executables and license inventory; this release adds no native build, OS execution, VPN enrollment or VPN inference qualification. Apache-2.0, Davide Mele copyright, NOTICE and upstream MIT attribution are unchanged. Consumer installations/configuration/pins, services and model profiles remain unchanged by publication. No PyPI upload is included.

RELEASE-MANIFEST.json binds exact source and asset provenance; verify downloads using SHA256SUMS.

runtime-llama 0.10.3

Improves embedding recovery correctness and reduces coordinator/planning work while retaining the Python APIs, one shared scheduler, exact token counts and numerical thresholds.

• Recovery validates each pending record against its own historical limits and the full fresh server observation. Owner-local failure preserves other callers, original result indices and actual transport drain. Native failed or ambiguous POSTs remain non-replayable.
• embedding_max_response_bytes independently bounds decoded native JSON bodies, default 8 MiB.
• embedding_preparation_cache_entries and embedding_preparation_cache_bytes configure exact-token retention, default 1,024 entries / 8 MiB. Effective bytes remain capped by pending bytes / 4; zero disables retention.
• Remembered notifications and relevant deadlines replace the fixed global 10 ms coordinator poll. Arbitrary cancellation callbacks and custom coalescing clocks retain finite checks. Planning avoids unnecessary keys, scans, rotating copies and index searches while preserving fairness, topology, cancellation, backpressure and drained close.
• The qualification probe can select and report the existing close or opt-in reuse HTTP policy. The default stays close; no fallback or retry was added.

Verification: 1,578 retained tests, Ruff check/format, 33 publication packaging tests, normalized source/wheel checks, public manifest/license audit and a clean isolated installation. All 28 runtime payload files match the live-qualified candidate byte-for-byte.

The retained live pair used both already loaded, verified EmbeddingGemma 2 BF16 replicas and unchanged thresholds. Each 511-text pipeline used 539 HTTP requests = 511 exact tokenizations + 4 embedding POSTs + 24 metadata reads. All vectors matched exactly between policies and against retained 0.10.2 outputs. Separate prepared/raw repeat calls needed zero additional tokenizations after eight initial measurements.

Pipeline wall time was 4.8449 s with close / 4.6630 s with reuse. This single pair qualifies the current plain-HTTP route functionally; it does not establish a general latency improvement. Historical 0.10.2 profile digests differ, so this is not a controlled version-speed comparison. Measurements include preparation, client/server and transport work, not pure GPU time.

The lifecycle and evidence are managed with Flower MCP. See the engineering qualification and public release record.

Install the Python wheel below. The three optional tsnet archives retain byte-identical executables and license inventory from v0.6.0; this publication adds no new native build, OS execution or VPN enrollment/inference qualification. Apache-2.0 and upstream MIT attribution are unchanged. Consumer installations, pins, services and model configurations remain unchanged. No PyPI upload is included.

RELEASE-MANIFEST.json binds source and asset provenance. Verify downloads using SHA256SUMS. Public wheel SHA-256:

73870dc4f8ab421ad9d4ff4f944888a623d57756d1372aa6c8786ad0e7b9aecb

runtime-llama 0.10.2

Reduces redundant embedding metadata requests, safely reuses completed exact preparation and fixes interrupted synchronous waits while preserving the existing Python APIs and shared scheduler.

• Admission reuses its just-observed catalog. Frozen owner cohorts and dispatch rounds share fresh metadata without a TTL-based inference permit.
• A bounded adapter-local LRU retains completed exact token IDs, keyed by full prepared text, task/token policy, verified content/semantic identity and tokenizer build/vocabulary. Prepared seals, current limits and fresh model checks remain authoritative. Unknown identity/build bypasses reuse.
• Interrupted blocking waits retire only the affected owner, wake the coordinator and drain actual work before reraising the original exception. Shared POST survivors continue, including through repeated waiter interruptions.
• Incremental queue summaries, iterator packing and constant-time owner rotation avoid repeated growing-prefix scans. A late owner renews partial preflight membership instead of prematurely splitting the tail. The 128-arrival regression now examines 256 records across preview/dispatch instead of 8,256 growing-prefix visits.
• Sync/async APIs, configurable total-text batching, bounded backpressure, deadlines, ordering, identity/topology checks and drained close remain supported by the same scheduler.

Verification: 1,506 retained tests, Ruff check/format, 33 publication packaging tests, normalized wheel/source checks, source/privacy/license audit and a clean isolated public-wheel installation. All 28 runtime payload files match the previously live-qualified candidate byte-for-byte.

The retained 511-text qualification used both already loaded, verified EmbeddingGemma 2 BF16 replicas at unchanged profiles and numerical thresholds: 539 pipeline HTTP requests = 511 exact tokenizations + 4 embedding POSTs + 24 metadata reads, down from 50 metadata reads. All 511 vectors matched the baseline with zero coordinate difference. Separate prepared/raw reuse calls performed zero additional tokenizations after eight initial exact measurements.

Pipeline wall time was 4.6436 s, versus the retained 4.5116 s baseline. No elapsed-time speedup is claimed. These are client/HTTP measurements, not pure GPU timings. Publication adds no new inference benchmark, consumer installation, service restart or model administration.

The lifecycle and evidence are managed with Flower MCP. See the engineering qualification and public release record.

Install the Python wheel below. The three optional tsnet archives retain byte-identical executables and licenses from v0.6.0; this publication adds no new OS execution or VPN enrollment/inference qualification. Apache-2.0 and upstream MIT attribution are unchanged.

RELEASE-MANIFEST.json binds source and asset provenance. Verify downloads using SHA256SUMS. Public wheel SHA-256:

5a98e5c9195c71042fdd7ac62f138a4489e57923346294dafabff6c293e56fcf

runtime-llama 0.10.1

Corrects persistent embedding batching so compatible records accumulate across source windows and callers for each verified replica. Small delivery windows no longer reduce the configured per-replica HTTP target. One scheduler still controls global and replica admission.

• Preparation and capacity events fill batches with the default embedding_batch_wait_sec=0.0; no fill timer is required. Finite tails and resource pressure ensure progress.
• Shared POSTs preserve owner indices, typed errors, cancellation/deadlines, backpressure and actual drained closure. Stopping one owner preserves survivors. Backend/transport failure affects all participants of that POST; native failed or ambiguous requests are never replayed.
• Fixes cleanup of an owner's unsent backlog after worker-side failure. Preparation admission retires independently of retained HTTP and result ownership.
• Public sync/async signatures and configurable total-text batch_size remain unchanged. Input context, aggregate HTTP token budgets and native computation batch sizes remain distinct.

Verification: 1,401 tests passed, Ruff check/format, normalized wheel/source audit and clean public-wheel install. Bounded functional qualification used both already loaded content-verified EmbeddingGemma 2 BF16 replicas: 511 texts in eight windows became four mixed-window POSTs (128/128/127/128), with two active embedding requests, one per alias, and zero coalescing delay. Exact preparation, ordered outputs and unchanged numerical thresholds passed. No model, server or consumer changes were made.

No 0.9.0 comparison or throughput improvement claim is included because an LLM workload remained permitted. Reported wall measurements include client, network, runtime and instrumentation; they are not pure GPU timings. Consumer acceptance remains separate.

The engineering lifecycle and evidence were managed with Flower MCP. See the qualification record and public release record.

Install the Python wheel below. The three optional tsnet archives reuse byte-identical executables and licenses from v0.6.0 (also shipped in v0.7.0 and v0.10.0); this release adds no new OS or VPN execution qualification. Apache-2.0 and upstream MIT attribution remain unchanged.

RELEASE-MANIFEST.json binds source and asset provenance. Verify downloads with SHA256SUMS. Public wheel SHA-256:

f36527cee8002263e346eb9f6dc7fb83d69b7209ca5efa60f64eda0a12937ae3

runtime-llama 0.10.0

Python adapters for externally managed llama.cpp servers, with a shared persistent embedding queue engine.

• Async window streams with bounded intake and ordered delivery share the synchronous adapter engine and scheduler.
• Per-replica ready queues form batches at dispatch capacity; exact native token-ID inputs avoid retokenizing prepared text.
• Configurable text/token/byte budgets, isolated cancellation, finite deadlines, identity/topology checks and drained closure. Default coalescing delay is zero; positive coalescing remains optional.
• Existing generation, structured tool calls, discovery and optional Tailscale/Headscale transport remain available.

Validation: 1,372 deterministic tests; package/lint checks; clean public-wheel installation; bounded live checks on two loaded replicas with unchanged numerical thresholds and 511 ordered outputs per run.

Performance boundary: no speedup is established. The exploratory candidate timings were slower than the coalescing baseline while external LLM activity changed. These are client/HTTP/runtime wall times, not GPU time. See the release verification and detailed qualification.

The three optional helper archives reuse unchanged executables originally qualified in 0.6.0, with original provenance and notices. No new native OS or live VPN qualification is claimed.

Download the wheel/source archive and optional helper for your platform. Verify hashes using SHA256SUMS and RELEASE-MANIFEST.json. Distribution: runtime-llama; Python import: runtime_llama. Apache-2.0, copyright Davide Mele; upstream attribution retained.

Versions 0.8.0 and 0.9.0 were local candidates; this is the next public release after 0.7.0. Consumer installations and remote models are unchanged.

runtime-llama 0.7.0

Python client for native llama.cpp HTTP servers, with corrected embedding batching and bounded preparation, independent HTTP connections by default, and optional typed native generation timings.

• batch_size is the total text window, distributed over verified usable replicas. A 256-text window used two concurrent POSTs of 128 texts; native computation batches no longer impose an artificial aggregate HTTP token cap.
• Exact task-aware preparation, once-only prefixes/special tokens, oversized-input measurement, identity/topology checks, finite deadlines and drained cancellation remain available.
• Generation bookkeeping is reduced while preserving reasoning, tool calls and runaway guards. Keep-alive is explicit; native unpatched llama-server works with the default connection-close policy.

Qualification: 1164 deterministic Python tests and Ruff checks passed. On server build b11716-797de63b2, two EmbeddingGemma 2 BF16 HIP replicas with Flash Attention off passed the unchanged numerical thresholds: all 77 discriminating comparisons had zero raw coordinate difference, including 33 cross-replica comparisons. The clean-installed candidate wheel passed the historical 551-output SDK replay; the final wheel contains the identical qualified runtime payload. An independent offline audit passed 394 checks.

These are bounded deployment results, not a guarantee for every input/backend or a pure GPU throughput measurement. Build and Flash Attention were changed together, so the numerical improvement is not attributed to one isolated cause. Earlier failed qualifications remain historical evidence.

Install the wheel below or use the tagged source:

python -m pip install 'git+https://github.com/Damel91/llama-runtime.git@v0.7.0'

Distribution: runtime-llama; Python import: runtime_llama; Python 3.11+; Apache-2.0. Optional tsnet helper archives explicitly reuse the byte-identical 0.6.0 executables and include licenses and original build provenance. This release adds no new OS/VPN execution or CI coverage claim. No consumer installation, service or model configuration was changed.

See verification and limits. Use RELEASE-MANIFEST.json and SHA256SUMS to verify the downloadable artifacts.

llama-runtime 0.6.0: optional Tailscale/Headscale transport

Adds an optional userspace Tailscale/Headscale transport for the existing llama.cpp HTTP client. Applications own configuration, credentials and node state through TailnetConfig and TailnetHelper; the library does not require a system VPN installation. Existing package/import names and default HTTP behavior are preserved.

The helper provides an authenticated loopback proxy, verified peer routing, end-to-end TLS through CONNECT, per-request cancellation and bounded drained shutdown. Generation, embeddings and model discovery share the explicit route. Native helpers pin Go 1.27.1 and Tailscale 1.104.1, omit the web UI and contain no working-tree VCS metadata.

Assets include the universal Python wheel, normalized source archive, separate macOS ARM64 / Linux AMD64 / Windows AMD64 helper archives with complete license notices, RELEASE-MANIFEST.json and SHA256SUMS.

Validation: 967 Python tests; 121 focused tests on Python 3.14; Go 20 top-level tests plus 16 subtests, race detector, vet and formatting; clean installed-wheel and real native offline smoke; source, archive, license and privacy checks. Linux/Windows helpers were crossbuilt. This release makes no claim of live VPN enrollment/inference, execution on Linux/Windows or new CI coverage.

Consumer guide · Verification and limits