v2.5.0

Added

• Cancellation: createRagPipeline creates an AbortController per run
  and passes its signal to every component (retriever.retrieve({ signal }),
  embedder.embedQuery/embed(text, { signal }), reranker.rerank(q, docs, { signal }), llm.generate(q, docs, { signal })); a new options.signal
  lets callers cancel. All built-in connectors and rerankers honour it.
  AgenticPipeline and IterativeRetriever propagate it too.
• QueryPlanner.plan(query, { llm, signal }): LLM-proposed sub-queries with
  strict JSON validation and heuristic fallback (returns a Promise);
  AgenticPipeline option enableLlmPlanning (default false).
• checkNumericConsistency, extractNumbers, detectInjection and
  PRICING_METADATA exports.
• Opt-in detectInjection on RetrievalGuard to scan retrieved documents for
  indirect prompt injection.
• CostCalculator: pricing provenance (source, asOf, verified), a
  pricing override, one-time unknown-model warnings
  (RAG_UNKNOWN_MODEL_PRICING), longest-match model resolution.
• EmbeddingReranker cacheSize (LRU of document embeddings) and
  concurrency options; MemoryRetriever eviction: 'fifo' option.
• Plugin signatures: computeContentDigest(), a signed contentDigest that
  binds plugin code, opt-in requireContentDigest and requireSignedPlugins,
  and register()'s optional { content } argument.
• JWTValidator options singleUseTokens (default true) and
  maxTrackedJtis (default 100000).
• AuditLogger reads its key from encryptionKey or
  RAG_AUDIT_ENCRYPTION_KEY; requireEncryption: true makes a missing key an
  error.
• .env.example for docker compose.
• npm run test:serial and npm run format:check.
• Working CLI workflow: rag-pipeline init, ingest <path>, query <question> and run now do real work against a persistent local index.
  init writes a pipeline of built-in components (--template local | openai | ollama | anthropic, or interactive prompts in a terminal).
• FileVectorStore (exported): JSON-file vector index with atomic
  writes, embedder/dimension tracking and per-source re-ingestion.
• .ragrc.json pipeline resolution: entries { stage, name, config }
  resolve to built-in components, custom plugin modules under plugins, or
  plugins registered on pluginRegistry.
• rag-pipeline serve and a working DevServer: local HTTP API
  (GET /health, POST /query) bound to 127.0.0.1, JSON only, 64 KB body
  limit, no CORS.
• LocalEmbedder.getState() / setState().
• rag-pipeline hub install <source> installs plugins from a local
  directory, .tgz, single .js file, npm:<package> or a registry
  (sha256-verified). It contains extraction to .rag/plugins, runs a static
  security scan (high-risk findings block unless --allow-risky), checks the
  stage contract in a separate process, and registers the plugin in
  .ragrc.json. PluginHub.installPlugin() uses the same installer.
• rag-pipeline hub certify submit|check <path> scores a local plugin
  from its files (contract, security scan, tests, documentation) against the
  BASIC/VERIFIED/ENTERPRISE automated thresholds.
• rag-pipeline hub list lists plugins registered in the config.
• rag-pipeline init setup wizard: chooses the embedder, LLM, retriever and an optional reranker, using only built-ins or custom plugin modules. Connector options are pre-filled with the connector defaults. Each custom plugin is loaded and its stage method checked before it is accepted. The wizard reports which API key environment variables are missing, can run a connection test on request, and previews the JSON before writing.
• rag-pipeline ai, built on the configured pipeline and index:
  ai train embedding [path] refits the local TF-IDF embedder and
  re-embeds the index; ai benchmark reports measured p50/p95/mean latency
  per stage; ai feedback <question> <docId> --rating up|down stores votes in
  .rag/feedback.json and ai search applies them as a capped,
  explained score adjustment; ai multimodal process <path> reports type,
  characters and chunks for text files (images exit 1: no built-in connector
  accepts image input).

Deprecated

• Experimental modules moved to @devilsdev/rag-pipeline-utils/experimental:
  ModelTrainingOrchestrator, FederatedLearningCoordinator,
  MultiModalProcessor, AdaptiveRetrievalEngine, SSOManager,
  SAMLProvider, OAuth2Provider, ActiveDirectoryProvider, OIDCProvider,
  TenantManager, ResourceMonitor, DataGovernanceManager /
  DataGovernance, PluginSandbox and PluginCertification. They are
  simulated or incomplete (random metrics, SSO that accepts any code outside
  production and does not verify SAML or OIDC tokens, random quotas, erasure
  that deletes nothing, a non-functional sandbox, fixed certification
  scores); see the Experimental docs page. Their main-entry exports still
  work but emit a one-time DeprecationWarning (RAG_DEP_* codes) when
  used, are marked @deprecated in the types, and will be removed from the
  main entry in the next major release. The same classes are shared by both
  entries, so instanceof checks keep working.

Removed

• Unused CLI config templates under src/cli/templates/ (old format,
  referenced a schema path that did not exist).
• Ten unreachable duplicate modules under src/ai/ and the internal
  src/dag/enhanced-dag-engine.js (never reachable from the package entries;
  the exported classes are unchanged).
• semantic-release configuration and dependencies (releases use
  npm publish --provenance from the release workflow), duplicate
  .eslintrc.json and .babelrc, about 50 one-off fix scripts and their
  ci:* npm scripts, the tracked junit.xml, and 14 test suites that never
  exercised src/ (fake pipelines, mocks, empty timing loops).
• CLI placeholders that printed success without doing anything (ingest,
  query, doctor, validate, plugin search|install|publish) and
  options that had no effect (--trace, --stats, --parallel,
  --streaming, --max-memory, --preview, --validate, --explain,
  --abort-timeout, --quiet, config show --format).
• The interactive wizard built on hard-coded mock registry data.
• The unregistered src/cli/commands/ai-ml* command modules: federated
  learning, LLM fine-tuning, ai train status|deploy, ai dashboard,
  adaptive profiles and multimodal search/describe. They called methods
  missing from the src/ai engines, exited 0 on failure and printed
  hard-coded or simulated numbers. The src/ai library modules are unchanged.
• Unreachable modules that were never exported or registered: src/dx/
  (including a realtime debugger exposing an unauthenticated WebSocket),
  src/cli/commands/{dx,docs}.js, src/cli/input-sanitization-middleware.js,
  src/cli/plugin-marketplace-commands.js, src/query.js, src/ingest.js,
  src/pipeline/pipeline.js, src/core/pipeline-factory.js. The package
  exports map only exposes . and ./cli, so these were never public.

Fixed

• evaluateRagDataset scores the generated answer; it scored the
  retrieved documents, so every case failed with zero scores.
• RetrievalGuard with enforceACL also reads a top-level acl /
  requiredRole on results that have metadata (it read only metadata
  then).
• createRagPipeline: timeouts now cancel work and no stage starts after
  an abort (a timeout still resolves { success: false, error: 'Execution timeout' }; a caller abort resolves 'Execution aborted'). Without an llm
  it runs retrieval only and returns answer: null instead of 'ok'. Stream
  mode with citations or evaluate ends with one { token: '', done: true, citations, evaluation } chunk. The artificial 5 ms delay is removed.
• Connectors retry only transient failures (network, 408/409/425/429/5xx)
  with exponential backoff, jitter and Retry-After; SDK-internal retries are
  disabled. API keys are read at construction, and calls without a key fail
  fast with MISSING_API_KEY.
• MemoryRetriever enforces maxDocuments and rejects mismatched vector
  dimensions instead of truncating them.
• AuditLogger no longer encrypts with a random per-process key that made
  logs unreadable after a restart; legacy createCipher records decrypt on
  Node 22+.
• JWTValidator / jwt-config honour an explicit clockTolerance /
  notBeforeTolerance of 0 (it became 30); the expired-token test is enabled.
• loadPluginsFromJson awaits registration and verifies signed manifests
  and plugin files before importing them.
• rag-pipeline init: the connection test aborts its calls after the 15 s
  timeout and the CLI exits afterwards.
• Benchmarks accept --iterations=N, no longer abort on the retry
  benchmarks' deliberate warm-up failures, and exit when done.
• Docs generation no longer hangs: the JSDoc regex backtracked
  exponentially on an unterminated /**.
• Tests run in parallel again (a suite destroyed the Jest worker's own
  stdio streams), no longer rewrite tracked fixtures, and coverage works on
  Windows (the test setup no longer patches path.resolve).
• Package import: require() / import of 2.4.5 failed in every clean
  install with Cannot find module 'isolated-vm'. isolated-vm is now
  loaded only when PluginSandbox runs and is an optional peer dependency;
  pino-pretty is an optional peer as well. Added the missing ajv-formats
  and marked dependencies, pinned chalk to 4.x (5.x is ESM-only), removed
  unused dotenv. Importing the bundle no longer prints "Missing Contract"
  warnings. A tarball smoke test (npm run smoke:pack) now runs in CI and
  before publish.
• createRagPipeline: the embedder result was computed and discarded,
  so vector retrievers never received a queryVector. Plugins registered on
  the exported pluginRegistry are now resolvable by name, and the timeout
  timer is cleared when a run finishes.
• MemoryRetriever throws a clear error when no queryVector is given
  instead of crashing inside cosine similarity.
• README / docs quick start now passes an embedder; as previously
  written it could not retrieve anything from MemoryRetriever.
• Citations with zero retrieved documents no longer fail the run; every
  sentence is reported as unsupported.
• HybridRetriever passes the full request (including queryVector) to
  retrieve(), so vector results are no longer silently dropped, and throws
  when every retriever fails.
• Guardrails: the retrieval guard (ACL, relevance, freshness) now runs
  before generation via the new options.filterResults hook; removed
  documents are reported by id only; strict-mode blocks no longer return the
  raw answer; PII is redacted in citations and evaluation details too.
• AuditLogger: tenant/user/action filters never matched, so queries and
  compliance reports returned every tenant's records. Encryption now uses
  createCipheriv with a random IV per field (it threw on Node 22 and reused
  the nonce on Node 18/20). The integrity hash now covers action and tenant.
• InputSanitizer: a regex state bug let command-injection payloads
  through after a previously blocked input.
• LLMReranker sends an actual ranking prompt instead of the raw
  question, keeps documents the LLM omits, and honours topK.
• Agentic RAG: with an embedder, each sub-query and refined query is
  embedded on its own instead of reusing the original query vector.
• TypeScript: 52 runtime exports (citations, guardrails, hybrid
  retrieval, agentic RAG, connectors, rerankers, GraphRAG, ...) had no
  declarations. They are now generated from JSDoc at build time into
  dist/types/ and re-exported from dist/index.d.ts; the build fails if
  any export is undeclared, and the smoke test compiles a TypeScript consumer.
• doctor / validate report real findings (unknown components,
  missing stages, missing API keys, broken custom plugins) and exit 1 on
  errors. doctor --auto-fix no longer lets callers re-enable a shell, no
  longer strips Windows path separators, and runs npm on Windows.
• hub (alias plugin) commands are registered. There is no default
  registry (the old default host does not exist): pass --registry <url> or
  set RAG_PLUGIN_REGISTRY for registry operations.
• Hot reload ignore patterns work again (chokidar 4 dropped glob support)
  and only JavaScript changes trigger a reload.
• loadRagConfig() accepts a config file path as well as a directory and no
  longer logs to stdout.
• LocalEmbedder produced all-zero vectors for corpora of two or fewer
  documents (IDF is now smoothed).
• Logger: console-style calls such as logger.error("Failed:", msg) no
  longer log the message as one key per character.

Changed

• Documentation: the README is rewritten with use cases whose code runs
  in CI (__tests__/docs/readme-examples.test.js executes every README
  example), and .ragrc.json's JSON schema (.ragrc.schema.json, shipped in
  the package) now describes the pipeline + plugins format the CLI and
  loadRagConfig() actually use.

• Metric changes:

  • Context recall now measures how much of a reference (ground-truth) answer
    the retrieved documents support. Pass it via
    PipelineEvaluator.evaluate(result, { referenceAnswer }) or
    run({ options: { evaluate: true, referenceAnswer } }). Without one,
    scores.contextRecall is omitted, details.contextRecall is
    { score: null, skipped: true, reason }, and metadata.metricsSkipped
    lists it.
  • computeGroundedness is a weighted mean over the available components;
    without a reference it is (0.4 * faithfulness + 0.3 * contextPrecision) / 0.7.
  • Claim support checks numbers: a sentence whose numbers (digits, decimals,
    percentages, number words) are not in the source scores below 0.4 and is
    not grounded. This affects citations, faithfulness, recall,
    PostGenerationGuard and SelfCritiqueChecker. "20" equals "twenty".

• createRagPipeline results always include answer when an llm is
  configured.

• QueryPlanner splits only at clause or question boundaries, keeps
  "salt and pepper"-style phrases intact, and expands comparisons into one
  sub-query per entity plus the comparison; results carry source.

• Prompts: built-in connectors put retrieved context in the user message
  as delimited, untrusted reference data, with a system prompt telling the
  model to treat it as data. Answers may differ.

• PreRetrievalGuard returns the query unchanged (no partial deletion) and
  reports detections in injection; strict mode still blocks. Detection
  normalises input first. To cut false positives, bare "act as", "jailbreak",
  uppercase "DAN", mid-line "system:" and "you are now ready" are no longer
  flagged; chat role markers, prompt-exfiltration phrasing and normalisation
  bypasses now are.

• CostCalculator prices gpt-4o-mini-* and claude-3-5-sonnet-*
  correctly (previously gpt-4 pricing and $0); customPricing entries without
  unit default to per-1000 tokens. Built-in prices are marked unverified:
  their asOf is when they were added, not when they were checked.

• EmbeddingReranker limits concurrency to 4 by default; call
  clearCache() after retraining an embedder.

• AuditLogger with encryption enabled and no key now disables
  encryption with a warning instead of using a throwaway key.

• Plugin trust store is empty by default (the placeholder key is gone).

• docker compose requires POSTGRES_PASSWORD, REDIS_PASSWORD and
  GRAFANA_ADMIN_PASSWORD, publishes ports on 127.0.0.1 only, and drops the
  debugger port and the Prometheus lifecycle API; the image build fails on
  critical runtime vulnerabilities.

• CI: one ci.yml workflow (required checks Test & Lint (18|20|22))
  that also enforces format:check and the tarball smoke test, and runs for
  pull requests into any branch. npm test runs Jest in parallel.

• DevServer now really listens: start() binds port 3000 by default
  (pass port: 0 for a free port), so existing callers may see
  EADDRINUSE. Default host is 127.0.0.1 (was localhost), default
  watchPaths is [] (was ./src/plugins, ./src/mocks), and requests
  must use a loopback Host header while bound to loopback.

• loadRagConfig() error messages now include details
  (Config validation failed: ..., Invalid JSON in config file <path>: ...)
  and it no longer logs.

• rag-pipeline plugin / hub registry operations require
  --registry <url> or RAG_PLUGIN_REGISTRY. hub certify submit takes a
  local plugin path and reports computed scores; it previously issued
  certificates from fixed placeholder scores. PluginHub.installPlugin() now
  installs into <project>/.rag/plugins and registers the plugin in the
  config (it previously only wrote metadata.json to localCacheDir), and
  getInstalledPlugins() reads the config.

• FileVectorStore index files store mostly-zero vectors sparsely and
  round values to 7 significant digits (about 10x smaller for the local
  embedder).

• Grounding scores (citations, faithfulness, context precision/recall,
  groundedness, PostGenerationGuard, SelfCritiqueChecker) now measure the
  share of a sentence's content words found in a document, optionally
  IDF-weighted. They previously used Jaccard similarity against the whole
  document, which shrinks with document length: an answer copied verbatim
  from a ~70-word chunk scored 0. Scores will be noticeably higher for
  grounded answers.

• Default per-sentence similarity threshold raised from 0.3 to 0.5
  (CitationTracker.similarityThreshold, PipelineEvaluator.threshold,
  faithfulness/recall/hallucination-detector threshold). Aggregate
  thresholds (groundednessThreshold, approvalThreshold) are unchanged.

• The tokenizer keeps non-ASCII letters (Política no longer becomes
  poltica) and splits on newlines and hyphens instead of gluing words
  together. This also affects BM25, BLEU/ROUGE and the scoring reranker.

• RetrievalGuard with enforceACL now fails closed when
  context.userPermissions is missing.

• CitationTracker.track() non-array error message is now
  "retrievedDocs must be an array".

Security

• Production dependencies: npm audit --omit=dev reports no high or critical
  findings (axios ^1.21.0; overrides for fast-uri, follow-redirects,
  form-data, ws and zod).
• Experimental SSO: the development bypass that accepted any authorization
  code requires devBypass: true and a non-production NODE_ENV and is
  refused in production; OIDC id_tokens are validated (JWKS signature, iss,
  aud, exp, nonce); OAuth2/OIDC use PKCE (S256); redirect_uri comes only
  from configuration; callbacks must match the stored login request's
  provider.
• Plugin signatures: Ed25519 verification works (crypto.verify with
  KeyObjects); a signed contentDigest binds plugin code and is always
  enforced when present.
• AuditLogger compares HMAC signatures in constant time; JWTValidator's
  replay store is bounded and expires entries.
• Retrieved documents are no longer placed in the LLM system role; prompt
  injection detection normalises homoglyphs, leetspeak and zero-width
  characters.
• .secrets is no longer tracked in git (rotate anything it held) and is
  excluded from Docker builds.

Known limitations

• Grounding is lexical plus a numeric-consistency check. Not checked:
  negation ("not twenty days"), units ("20 hours" vs "20 days"), ordinals, a
  lone "one", differently split decimals, and claims that combine facts from
  several documents. Document-side matching is lenient ("twenty-five" in a
  source also supports "twenty" or "five").
• The openai SDK stays on 4.x: newer majors need Node 20+ and will come with
  3.0. csv-parse has one moderate advisory whose fix is a major release.
• Experimental SSO cannot refresh OIDC sessions, and SAMLProvider refuses
  to authenticate.
• AuditLogger's HMAC signing key is random per process unless signingKey
  is passed, so verifyIntegrity flags records written before a restart.

Upgrading: see the 2.5.0 release post and Migration guide. Deprecated exports keep working in 2.5 with a one-time warning and are removed from the main entry in 3.0.