Loop Engineering v0.4.1 — verified alpha release

[chillum-codeX]

Loop Engineering v0.4.1 is live.

This release provides a Python runtime for repeatable agent loops with:

• validated state transitions
• token, cost, step, and retry budgets
• deterministic gates before model calls
• generator/evaluator separation
• human checkpoints and recovery handlers
• JSON/SQLite persistence
• CLI commands for init, audit, run, cost, and validate

Verification

• 97 tests pass locally and in GitHub Actions
• deterministic evaluator validation confirms oracle answers pass and empty negative controls fail
• one paid OpenRouter smoke request verified provider-reported token and cost accounting
• wheel and source distribution pass twine check
• Tests, Static Checks, and Documentation workflows are green on main

This is an alpha release. Historical mock benchmark outputs are explicitly not presented as model-quality or security evidence, and the project makes no SOTA or production-readiness claim.

• Repository
• v0.4.1 release
• Quickstart

Feedback on runtime semantics, adapters, and reproducible benchmark design is especially welcome.

[richardchen874-sys]

The provider behavior behind "Loop Engineering v0.4.1 verified alpha release" is what I would test first: latency, usage fields, tool calls, streaming, and fallback can diverge even when the first request succeeds.

I would log route choice, provider/model, tool-call support, latency, usage, and retry/fallback reason together. Agent failures are much easier to debug when the route decision is visible.

This overlaps with something I am testing: an OpenAI-compatible multi-model API layer around official Chinese models, especially around cost-aware routing and normalized usage behavior. For loop-engineering, is the harder problem provider compatibility, routing quality, or keeping per-run cost predictable?

[chillum-codeX]

Good question, and useful to be specific rather than diplomatic about it.

Routing quality is the actual gap — there isn't one yet. Today a loop registers one LLMClient per component; there's no logic that picks between providers/models, and no fallback path when one degrades mid-run. If you're already building cost-aware routing, that's genuinely ahead of where this project is.

Cost predictability is the most developed piece. Budget/BudgetGate/BudgetTracker track token/cost/step/time limits, and provider-reported usage gets wired back into the running budget via a callback (_consume_llm_usage in core.py), not estimated after the fact. That part I'd stand behind.

Provider compatibility is partial, and your point about post-first-request divergence is accurate. llm_client.py currently has zero streaming support and zero tool-call normalization — just plain request/response behind a shared generate()/generate_structured() interface. So yes, exactly the failure mode you're describing (works on the first call, diverges on streaming or tool use) is unhandled today, not solved.

On the logging suggestion: we have UsageRecord (provider/model/tokens/cost) and LLMResponse.latency, but they're separate objects, not one record per call joining route decision + latency + usage + retry reason. That's a fair gap — visibility into why a route was chosen is what makes agent failures debuggable instead of mysterious, and we don't have that today.

If your routing layer normalizes usage/latency across providers, I'd be interested in comparing notes on the schema — happy to look at whatever you're testing.

[richardchen874-sys]

That separation is exactly the boundary I am testing: one record for the logical request, then an ordered set of upstream attempts rather than forcing route choice, latency, usage, and retry outcome into separate objects.

The minimum fields I would compare are requested and resolved model, route policy/version, provider and model per attempt, attempt latency, provider-reported usage, terminal outcome, and the reason a fallback was or was not allowed. Billing evidence should stay separate from normalized operational usage.

Rather than turn this into a broad integration, would you be open to reviewing one sanitized single-attempt trace and one fallback trace against UsageRecord and LLMResponse? If that boundary maps cleanly, we can then decide whether a small non-production comparison is worth doing.

[chillum-codeX]

Yes, happy to review both traces against that boundary but I'll save you a round trip: I can already tell you where it maps and where it doesn't, since UsageRecord/LLMResponse are small enough to check by hand.

Maps cleanly: resolved model (model), provider (UsageRecord.provider), and usage/cost (input_tokens/output_tokens/cost, with estimated: bool already distinguishing provider-reported from approximated that part of your billing-vs-operational split already exists, at least at the single-call level).

Doesn't exist at all: requested-vs-resolved model (only one model field, no distinction), route policy/version (no routing concept anywhere in this client), terminal outcome, and fallback-allowed reason. LLMResponse.latency is per-response, not per-attempt there's no "ordered attempts" structure to attach it to, because a call here is a single shot: it either returns one LLMResponse or raises. No partial-attempt record on the way there.

So your instinct in the first comment was right this isn't a schema mismatch so much as a missing layer. UsageRecord/LLMResponse model a call, not a routed request, and your minimum field list needs the second thing.

Send the two sanitized traces (single-attempt and fallback) and I'll map them field-by-field against the above rather than guess further concrete beats hypothetical here. A comment with two JSON blocks in this thread is fine.

[richardchen874-sys]

Your mapping confirms the boundary precisely: UsageRecord and LLMResponse describe a call, while requested-versus-resolved model, ordered attempts, terminal outcome, and the fallback decision belong to a routed-request layer that does not exist yet.

Rather than asking you to review two JSON blocks that I generate, the cleaner comparison is for Loop Engineering to capture them directly from a real OpenAI-compatible endpoint. That keeps the evidence in your own parser and avoids adapting a trace after the fact. A bounded check only needs one normal request and one forced-fallback request, with prompt and response content removed.

I can provide limited non-production credit for that comparison through NovaRouteAI. Please register at https://novarouteai.com, then send only the registered account email privately to richardchen874@gmail.com so I can verify the test account and provide the exact model ID and endpoint details. Which channel do you normally use for technical coordination: email, Discord, or Telegram? I am equally happy to continue by email.

If those two traces expose the missing routed-request fields cleanly, you will have a concrete basis for deciding whether that layer belongs beside UsageRecord/LLMResponse or above LLMClient.