v0.1.0

The first public release. Evals.si is a self-hosted evaluation service: one set of evaluators for scoring outputs you have, running a model or agent on a dataset, watching live traces, and rewarding RL training.

Added

• Score, Run, Watch, Reward. Score existing outputs; run a target (model, RAG pipeline or agent) on a dataset with trials, gates and budgets; score live traces with online policies; use the same evaluators as RL rewards and in training-loop hooks (PyTorch, Lightning, Hugging Face Trainer and TRL).
• Evaluators with confidence intervals on every mean. Records an evaluator cannot judge are skipped, and evaluator failures are errors, never zeros. There is no default judge model.
• Agent runs on sandboxed tasks, including SWE-bench, τ-bench, Terminal-Bench and BFCL, over A2A, MCP, HTTP, an OpenAI Responses API or a CLI.
• Ingest and trace sources. OTLP ingest (OTel GenAI, OpenInference, OpenLLMetry, MLflow conventions) with redaction; sources that read MLflow, Phoenix and Langfuse, with score write-back; scheduled batch reports; agent-framework traces from LangGraph, CrewAI, the OpenAI Agents SDK, LlamaIndex and the Claude Agent SDK.
• Pipeline controls. JSONL and Parquet dataset upload, idempotent keyed writes, run deletion to recover quotas, and an SDK evalsi.log() that batches in the background.
• Adapters for lm-evaluation-harness, Inspect AI, RAGAS and DeepEval in isolated environments; custom evaluators as Python functions or sandboxed Wasm modules.
• Sandboxes that fail closed: Firecracker microVMs where /dev/kvm is usable, else bubblewrap, else Landlock, else hardened pods, never with weaker isolation than asked.
• Server and tools. A single evalsid binary (Connect, gRPC and REST), a web UI, guardrails, annotation queues, webhooks, MCP tools, authentication and per-project credentials, and the evalsi CLI and Python SDK.
• Kubernetes. Helm charts (evalsi, evalsi-crds, evalsi-sandboxd), an operator with EvalRun, OnlineEvalPolicy, Evaluator and SandboxClass CRDs, and an air-gapped install bundle.
• Release engineering. Linux amd64 and arm64 archives, Python distributions, multi-architecture images and OCI charts, signed keylessly with Sigstore, with scripts/verify-release.sh.

Security

• The API bounds every RPC message at 64 MiB, and reads at most 64 KiB of a request whose credential was rejected. Connect reads a unary request whole before authentication runs, so a caller without a valid credential could otherwise make the server buffer a body of any size.
• The operator lets a resource name a project other than its namespace (evals.si/project) only from the release's namespace or namespaces mapped with operator.projectNamespaces, since it acts through the API as an owner.
• The chart's bundled NATS requires a token, and a NetworkPolicy limits its client port to the release's namespace.
• Trace sources refuse shared address space (100.64.0.0/10, where some clouds serve instance metadata), 0.0.0.0/8 and NAT64 addresses of internal hosts unless listed in sources.allow_hosts.

Known limits

See docs/LEFTOVERS.md for the full list and SUPPORT.md for what has been verified. In particular, EKS and GKE have not been run, and the per-release sandbox-pool NetworkPolicy has not been exercised on a cluster that enforces NetworkPolicy.