0.13.0

0.13.0

Added

• Online evaluation over live traces. reactifact.eval.online scores
  finished runs sampled from a TraceStore with the same Evaluators as
  offline evaluation, writing the verdicts back as tags (pass → eval, fail →
  eval:failed) for the dashboard, plus metrics
  (reactifact_online_eval_runs_total, reactifact_online_eval_score).
  OnlineEvalConfig controls sampling (sample_rate/strategy/seed),
  filters, tags and reference_fn; online_evaluate(store, config) runs one
  batch and OnlineEvaluator schedules it (run_once, run_forever,
  on_report). Because a trace is truncated, the source decides what is
  scoreable: trace_source() scores the RunTrace itself (typed path via
  trajectory_match, output_present, no_errors; unusable metrics are
  skipped, never a false zero), while context_source(run_fn) rehydrates the
  full Context for exact text/core_metrics/judge scoring. Ships a FastAPI
  router (create_online_eval_router: run/report/summary, lazy import) and the
  CLI python -m reactifact eval <traces.db>. Docs in docs/en|ru/eval.md and
  the reactifact-eval skill. The devops example demonstrates it:
  examples/devops/online_eval.py scores served runs with trace-level
  routed/routed_to evaluators and web.py mounts
  create_online_eval_router next to the trace dashboard.
• Trust & safety: guardrails, authorization and quota. Three opt-in,
  library-level layers on RuntimeResources, acting on typed artifacts and
  resource access (deterministic, testable) rather than around model messages.
  reactifact.guardrails.GuardrailPolicy runs ordered checks on every
  agent-produced Create/Update: PIIGuardrail (redacts via any Redactor),
  InjectionGuardrail, DenyListGuardrail, SizeGuardrail, the generic
  PatternGuardrail, with on_violation="block" (raises GuardrailViolation)
  or "flag" (records reactifact_guardrail_triggered_total). reactifact. authz adds a Principal + capability-based PermissionPolicy, enforced by
  the runtime on run/create/update/delete, by the tool loop on
  execute, and available to produces via resources.require_authorized(...);
  denial raises AuthorizationError. reactifact.quota adds cross-turn,
  per-principal usage limits (Quota/QuotaTracker/QuotaLLM) — checked at
  turn start (RunOutcome.QUOTA_EXCEEDED) and counted through the turn, with
  cache hits free and a pluggable QuotaStore (in-process by default). All off
  by default; docs in docs/en/safety.md and the reactifact skill's
  references/trust-and-safety.md. The devops example demonstrates it:
  examples/devops/guardrails.py defines a custom guardrail (a destructive
  production change must cite a CHG- ticket) plus secret redaction, wired
  onto RuntimeResources and screening the raw input in web.py; run
  examples/devops/safety.py for an end-to-end, no-LLM demo.
• Agent skills for building with reactifact. The package ships official
  Agent Skills in reactifact/skills/<name>/SKILL.md (version-matched, since
  they live in the package), installed into a project with
  python -m reactifact skills install [--target agents|claude|both] (also
  skills list / skills show). Eight skills cover the paradigm end to end:
  reactifact (mental model, invariants, working loop, routing, mistakes),
  reactifact-agents (produce/consume/effects/provenance), reactifact-llm
  (structured output, LLMAgent/HITL, tools, budgets), reactifact-rag
  (sources, fan_out_sources, recipes), reactifact-testing,
  reactifact-eval, reactifact-observability, and
  reactifact-from-langchain. Each has a workflow plus references/ for
  lookup; every runnable code block is executed in CI by tests/test_skills.py
  (# not-run marks illustrative fragments), so the examples cannot rot. A
  root AGENTS.md points library contributors at the skills and the repo
  commands.
• Prometheus metrics, dependency-free. reactifact.metrics adds a tiny
  in-process collector (Metrics) and a MetricsTracer that fills it from the
  tracing hook, plus create_metrics_router serving GET /metrics (the web
  extra). Counters: run/budget outcomes, agent runs, LLM calls/tokens/errors
  and (with MetricsTracer(metrics, pricer=…)) cost; histograms: agent/LLM
  latency. Metrics.render() emits the Prometheus text exposition directly —
  no prometheus_client dependency — and MetricsTracer composes with the
  existing sinks (tracer=[TraceStore(…), MetricsTracer(metrics)]). Span
  reads/writes/edges also give domain-adjacent counters
  (reactifact_artifacts_written_total{type,op},
  reactifact_artifacts_read_total{type},
  reactifact_relations_total{relation}), and RuntimeResources.metrics is a
  no-op-until-configured sink so a produce can record app-specific signals
  (call.context.resources.metrics.increment("route_total", route=…)).
• Dataset evaluation and LLM-as-judge. reactifact.eval grows from the
  deterministic metric core (§56) into a full offline-evaluation layer, with no
  hosted store. Dataset/Example hold inputs + optional
  reference_outputs/metadata (content-hashed version, .json/.jsonl
  loading); evaluate(dataset, target, evaluators, summary=…) runs a target
  over every example — the target may return a Context, a
  reactifact.testing.ScenarioResult (read via .context/.trace), an
  outputs mapping or a RunResult — and scores it. An Evaluator is any
  callable EvalInput -> Feedback | bool | float | {key: score} | list | None
  (None skips, never a silent zero); from_metric wraps the scoring metrics,
  trajectory_match(mode, steps=…) matches a run's typed path
  (strict/unordered/subset/superset over agents/events/reads/
  writes). llm_judge (plus judge_correctness/judge_relevance/
  judge_faithfulness) turns any LLMProvider into a judge with tolerant JSON
  parsing, continuous scores, choices snapping and reference-based grading —
  composable with CachingLLM/budget/metrics and faked with FakeLLM.
  EvalReport gains aggregate()/passed()/assert_passed() (the CI gate,
  EvalFailure), summary evaluators (summary_pass_rate, summary_mean) and
  assert_eval. reactifact.eval is now a package; the existing
  EvalCase/run_suite metrics API is unchanged.
• Provider resilience: exact LLM cache and failover. reactifact.cache. CachingLLM keys every complete() by the whole request (model, messages,
  temperature, …) and answers a repeat without calling the provider — no
  tokens, no latency; an optional KVCache over any KVBackend persists it.
  A hit is marked (raw={"reactifact_cache": "hit"}) so BudgetLLM does not
  charge it. Deliberately not semantic (an approximate match would answer a
  different question) and stream() is uncached. reactifact.routing. RouterLLM tries providers in order and falls back on failure
  (should_fallback= — e.g. retryable_only — and on_fallback=); only
  complete() fails over. Both are drop-in provider wrappers, composable:
  resources.llm = CachingLLM(RouterLLM([openrouter_llm(), groq_llm()])).
• Stable artifact type identity (schema evolution). reactifact.types
  decouples the persisted type string from Python's module layout:
  register_type(Model, type_id=…, aliases=[…], migrate=…) makes a model
  rename/move — or a field rename — no longer break loading old sessions,
  commits, events or recordings. type_id_of / resolve / migrate_payload;
  the default id stays the qualified name and pre-registry payloads keep
  loading, so it is fully backward compatible. Applies to Artifact.data_type,
  Create/Update.data_type and Event.artifact_type.
• Multi-turn conversations on typed artifacts. reactifact.recipes. conversation adds Conversation — generic over your message model (any
  model with role/text; ConversationMessage is a ready default) — with
  create_turn/add_message/transcript/turn_count/conversation_state,
  so apps stop hand-rolling a chat log and a prompt builder. For the common
  case of separate question/answer types, Transcript + MessageSpec
  render several typed turns (with per-type role and an optional renderer for a
  non-text assistant payload) into one Message list, leaving the artifacts
  untouched. ChatMemory on
  ChatAssistant wires bounded memory: keep (prune the raw thread via
  WindowPruner), summarize+summary_type (a growing digest via
  RollingDigestSummarizer), and compact_commits (Context.compact — bounds
  history without touching artifacts, safe for correlated/typed messages). The
  repair/devops/knowledge examples use compact_commits to bound
  long-lived sessions. Docs: a new Multi-turn & chat page
  (EN + RU) mapping the LangGraph pieces to reactifact, plus a minimal offline
  examples/chat demo of the single-model recipe.
• Context compaction. Context.compact(keep_commits=…, keep_versions=…)
  bounds a long-lived session: it collapses commits older than the last
  keep_commits into a baseline snapshot (and optionally trims each artifact's
  retained version history). The absolute version/head_id are preserved, so
  context_hash, commit version numbers and pending-trigger resume are
  unchanged. It is irreversible — checkout/diff below the baseline raise
  (Context.compacted_at reports it) — and it does not evict artifacts (the
  memory recipes still own that). Artifact gained an absolute version
  counter decoupled from len(history) so trimming history never changes
  context_hash; the baseline is persisted in Context.to_dict/from_dict,
  copied by clone/branch, and CommitLog replays correctly across it.
• Token and cost budgets. Budget gains max_tokens (total prompt +
  completion over the turn) and max_cost, with RunOutcome.budget_tokens_ exceeded / budget_cost_exceeded. Usage is counted by the runtime itself
  (BudgetLLM + BudgetTracker), independent of tracing — which is optional
  — so RunStats now reports prompt_tokens/completion_tokens/cost even
  without a tracer. Streaming is counted too: LLMResponseChunk carries an
  optional usage (the OpenAI-compatible base asks for it via
  stream_options, and Anthropic/Gemini surface it), and a token/cost budget
  stops the next generation or tool-loop step — an in-flight call may
  overshoot by one, like max_seconds. Cost needs an injected
  RuntimeResources(pricer=…) (reactifact.pricing); the framework ships no
  built-in price table on purpose. ScenarioReport gained cost (computed
  from the same pricer).
• Sessions persist pending triggers, so an interrupted run resumes after a
  process restart. Context.to_dict()/from_dict() now carry the event
  queue (Event.to_dict/from_dict), and a session-backed runtime saves once
  at each generation boundary after consuming its batch — settled
  generations included — so the saved queue is consistent: an already-consumed
  trigger is not replayed, and one whose consumer never ran is. Reopening a
  session and calling arun() continues from where it left off instead of
  seeing an empty queue. At-least-once: a produce that had already committed
  may run again, so use stable ids for idempotent produces. context_hash
  still excludes the queue (it is transient state, not provenance).
• reactifact.testing ergonomics: provenance assertions, sync API, golden
  snapshots, explain(), an opt-in pytest plugin, and richer fault injection.
  • Provenance is now testable directly: result.artifacts(Answer).linked( "supported_by", Evidence) and a RelationAssertions group
    (result.relations.has/none/count/outgoing/incoming) assert on the
    artifact graph's edges, not only artifact fields.
  • ScenarioLab.run_sync(...) / Scenario.turn_sync(...) mirror the async API
    (like Runtime.run), and ScenarioResult.explain() / Scenario.explain()
    dump the whole run (artifacts, relations, path, tools, LLM, errors).
  • Golden snapshots: assert_golden_file(context, path) seeds the snapshot
    when it is missing (or update=True/$REACTIFACT_GOLDEN_UPDATE=1) and
    fails on drift; result.assert_golden(path) / scenario.assert_golden(path)
    wrap it, plus result.context_hash.
  • Fault injection grew stubs and precision: stub_tool/stub_resource
    (returns/side_effect, unittest.mock-style), and fail/
    fail_resource gained when(args) and delay (slow-then-fail).
  • Opt-in pytest plugin (reactifact.testing.pytest_plugin via
    pytest_plugins = [...]): runs async def scenarios without
    pytest-asyncio, maps ScenarioSkip to a skip, adds a scenario_lab
    fixture defaulting mode= to $REACTIFACT_SCENARIO_MODE.
  • The reactifact scenario CLI renders failures pytest-style: a FAILURES
    section, the failing frame from the user's code with its source line, the
    exception, and the run state (ScenarioResult.explain(), tracked via the
    new reactifact.testing.current_result()); --no-state omits the dump.
  • Docs: a new Testing page (EN + RU) with the assertion
    groups, fault/stub reference, record/replay, golden and the plugin.

Changed

• Docs build is now strict in CI, and the public API is frozen.
  mkdocs build --strict runs as its own CI job (the docs dependency group),
  and every out-of-docs_dir link (to examples/**, the root README.md /
  CHANGELOG.md) now points at absolute GitHub URLs. tests/test_public_api.py
  pins reactifact.__all__ and each core submodule's __all__, so an export
  can only change deliberately; docs/en|ru/api.md §Stability documents the
  freeze, the deprecation window and the 1.0 checklist in release.md.
• OTLP docs cover LangSmith and SigNoz. The OTLPTracer section in
  docs/en|ru/observability.md gained copy-paste recipes (endpoints/headers)
  for LangSmith's OTLP ingestion and SigNoz (self-hosted and Cloud) — no code
  change, the tracer already targets any OTLP/HTTP collector. quota.py
  coverage rose 85% → 100%.

Fixed

• A failing generation no longer silently drops its triggers. The runtime
  used to drain_events() at the start of a generation, so an agent exception
  (default isolate_errors=False) — or any error during patch validation /
  commit — left the artifacts created but their consumers unqueued: the next
  arun() saw an empty queue and settled at zero runs, losing the work. The
  queue is now read non-destructively (Context.pending_events()) and the
  batch is consumed (Context.consume_events()) only after the generation's
  patches are committed, so a retry re-runs the work. Events the commits
  themselves emit (the next generation's triggers) are preserved. Paired with
  the session-persisted queue above (see Added), this makes a crashed run
  resume after a restart, not just retry in-process.