Skip to content

v3.5.0

Choose a tag to compare

View all tags
@alikahwaji alikahwaji released this 05 Nov 05:14
· 111 commits to main since this release
v3.5.0
48d2f25
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

A minor release with no breaking changes. The guards take your own detectors and their accuracy is published, evaluation scores reach OpenTelemetry and Langfuse, and verifyCitations() checks the citations Claude and OpenAI return.

npm install @devilsdev/rag-pipeline-utils@^3.5.0

Security

  • Connector errors no longer carry the API key. On 3.4.0 and earlier, a failed call through AnthropicConnector, CohereConnector or OllamaConnector (a wrong key, an unknown model, a network error) threw an axios error that kept the request headers, so console.error(err) or an error reporter printed the API key. Errors are now plain Errors with status, code and the API's response, and their message says what the API said. OpenAIConnector was not affected. If your logs may hold such errors, rotate the key.

Highlights

  • Pluggable guard detectors. PreRetrievalGuard, RetrievalGuard and PostGenerationGuard take detectors (an injection classifier, a PII service), with the built-in patterns or instead of them (builtIn: false). Async detectors run through the new checkAsync() / filterAsync(); a detector that throws blocks by default. Detectors
  • Measured guardrails. rag-pipeline eval-guardrails <cases.jsonl> and evaluateDetectors() report precision, recall, F1 and confusion counts per detector, and fail CI below a --min. The Benchmarks page publishes the patterns' numbers, and a classifier's for comparison, on public test sets. Guardrail accuracy
  • Scores in OpenTelemetry and Langfuse. With telemetry on, each evaluation score is a gen_ai.evaluation.result event parented to the span it judges; result.telemetry carries the ids, and toLangfuseScores(result) returns langfuse.score.create() objects. No new dependency. Observability
  • Verify provider citations. fromAnthropicCitations() and fromOpenAIFileCitations() normalise a model's citations, and verifyCitations() checks each claim against the passages it cites, with number and negation checks and the optional judge. Citations

Fixed

  • AnthropicConnector works with current Claude models. On 3.4.0 the LLM judge failed with HTTP 400 on Claude Haiku 5.5, Sonnet 5.5 (the connector's default), Opus 5.5 and Fable 5.1, which reject the judge's temperature: 0, and generate() returned undefined when a model thought before answering. The connector now retries without temperature (with a warning), returns every text block, rejects a refusal with code: "MODEL_REFUSED", and generateWithUsage() reports tokens. Found by the new Claude benchmarks.
  • CostTracker and createModelRouter price Claude Haiku 5.5 and Fable 5.1 (they reported $0), and CostTracker's estimate counts the retrieved passages, not just the query (it was about 25 times too low).
  • The TypeScript declarations compile with @types/node 25 and later.
  • A strict GuardrailsPipeline block's error says why.

Changed

  • The retrieval span is named retrieval with kind internal, per the OpenTelemetry GenAI conventions (it was retrieval <data source>, kind client); only pgvector and Qdrant spans change name. SEMVER.md now states that telemetry may follow those conventions in minor releases, each change listed in the CHANGELOG.

The scoped packages (@devilsdev/rag-core, -connectors, -eval, -guardrails) and @devilsdev/rag-mcp-server are released at 3.5.0 too. Full details: CHANGELOG · Blog post

Assets 2
Loading