Release v2.5 · verifywise-ai/verifywise

What's new

2.5 is a large release. The headline additions are a new template-first Reporting stack, OpenTelemetry observability across the services, and a migration of frameworks to a plugin/extension model. It also carries a security-hardening wave, advances AI Agents and Agent Control, adds a typography and theme token foundation, refactors the policy editor and the LLM Evals experiment flow, and continues raising test coverage.

Template-first Reporting — build reports from templates, with schedules, runs, delivery, and framework-aware content
OpenTelemetry observability — OTel instrumentation across the backend, EvalServer, and AI Gateway, with a dedicated observability compose stack
Frameworks as extensions — custom frameworks and the existing framework plugins migrated to an extension model
AI Agents and Agent Control — add agents manually from the empty state, distinguish manual vs synced agents, per-agent policy scope, decision provenance, a per-agent activity view, and run-grain stat cards on the Runs page
Security hardening — row-level security, HTTPS and DAST/CI scanning, admin registration lockdown, support for more than one superadmin, an SSRF fix, ReDoS remediation, and a sweep of open code-scanning alerts
Typography and theme tokens — a typography token foundation, chip and badge tokens, and theme spacing migrated to MUI spacing units
Policy editor refactor — the policy editor split into co-located modules, with optional team-member input
LLM Evals refactor — the new-experiment modal broken into pure helpers and per-step modules (dataset, model, judge), and EvalServer models moved to Pydantic
Dashboard and risk fixes — metrics revalidate when returning to the dashboard, and a project's current risk level now syncs to its computed residual
AI Gateway — the prompt editor model selector is now searchable, and the litellm SDK was updated (1.89.3 to 1.97.0)
Reliability and access fixes — password reset by stored email with anti-enumeration preserved, org-level file access authorized by organization, and AI Trust Center logo creation on first upload
User preferences API — self-scoped GET/PATCH of the current user's preferences
Test coverage — accessibility end-to-end checks with axe-core, contract tests for raw SQL list endpoints, a critical-journey e2e suite, flaky-test stabilization, and frontend line coverage raised toward 50%

What's Changed

fix(mrm): review follow-ups — sweep read caching, role query reuse, alert copy/logging by @gorkem-bwl in #4272
4029 add accessibility e2e checks with axe core by @DeboraSerra in #4274
4028 add contract tests for raw sql list endpoints by @DeboraSerra in #4275
feat(themes): add typography token foundation by @Innagl in #4277
GitHub Actions Speed-Up by @MuhammadKhalilzadeh in #4273
docker(frontend): bump nginx from 1.31.2-alpine to 1.31.3-alpine in /Clients by @dependabot[bot] in #4281
ci: bump actions/setup-node from 6 to 7 by @dependabot[bot] in #4282
deps(eval): update anthropic requirement from >=0.111.0 to >=0.117.0 in /EvaluationModule by @dependabot[bot] in #4283
deps(eval): update mistralai requirement from >=2.4.9 to >=2.7.0 in /EvaluationModule by @dependabot[bot] in #4284
deps(eval): update deepeval requirement from >=4.0.3 to >=4.1.1 in /EvaluationModule by @dependabot[bot] in #4285
deps(eval): update tqdm requirement from >=4.68.2 to >=4.69.0 in /EvaluationModule by @dependabot[bot] in #4286
ci: bump actions/cache from 4 to 6 by @dependabot[bot] in #4288
ci: bump torch from 2.10.0 to 2.13.0 in /EvalServer in the pip group across 1 directory by @dependabot[bot] in #4299
deps(backend): bump nodemailer from 8.0.9 to 9.0.3 in /Servers by @dependabot[bot] in #4297
deps(backend): bump @azure/msal-node from 3.8.10 to 5.4.1 in /Servers by @dependabot[bot] in #4295
deps(backend): bump @slack/web-api from 7.16.0 to 8.0.0 in /Servers by @dependabot[bot] in #4293
deps(backend): bump @types/node from 25.9.1 to 26.1.1 in /Servers by @dependabot[bot] in #4296
Admin Registration Lockdown by @MuhammadKhalilzadeh in #4279
refactor(themes): migrate theme spacing to MUI spacing units by @Innagl in #4301
deps(backend): bump axios from 1.16.1 to 1.18.1 in /Servers by @dependabot[bot] in #4302
chore(deps-dev): bump brace-expansion from 5.0.5 to 5.0.7 in /GRSModule/ui/frontend in the npm_and_yarn group across 1 directory by @dependabot[bot] in #4303
deps(backend): bump shell-quote from 1.8.4 to 1.10.0 in /Servers by @dependabot[bot] in #4305
deps(backend): bump body-parser from 2.2.2 to 2.3.0 in /Servers by @dependabot[bot] in #4306
chore(deps): bump pillow from 12.2.0 to 12.3.0 in /GRSModule in the uv group across 1 directory by @dependabot[bot] in #4307
chore(deps): bump js-yaml from 4.2.0 to 4.3.0 in /GRSModule/ui/frontend in the npm_and_yarn group across 1 directory by @dependabot[bot] in #4308
fix(dashboard): revalidate metrics when returning to the dashboard by @gorkem-bwl in #4309
docs(mrm): note email delivery and auto-opened findings in ingestion side effects by @gorkem-bwl in #4310
ci(security): make the security scanners actually run by @gorkem-bwl in #4312
fix(risk): sync current risk level to the computed residual by @gorkem-bwl in #4311
docs(api): regenerate OpenAPI spec to fix drift by @gorkem-bwl in #4313
ci(security): unblock tenant-isolation job and tune gitleaks allowlist by @gorkem-bwl in #4315
test(audit): cover refresh_tokens in the tenant-isolation audit by @gorkem-bwl in #4314
Add agents manually from the empty state, rename to AI Agents, and distinguish manual vs synced by @gorkem-bwl in #4319
Agent Control: per-agent policy scope, decision provenance, and per-agent activity view by @gorkem-bwl in #4320
Removing EU AI Act from Use cases by @MuhammadKhalilzadeh in #4321
feat(agent-control): run-grain stat cards on the Runs page by @gorkem-bwl in #4322
fix(ai-apps): correct empty-state copy to match shipped functionality by @gorkem-bwl in #4323
Feat: Migrate localStorage callers by @Br0wnHammer in #4265
Feature/chip badge tokens by @Innagl in #4318
fix(themes): add missing typography token file by @gorkem-bwl in #4333
fix(ai-gateway): make prompt editor model selector searchable by @gorkem-bwl in #4332
fix(auth): reset by stored email, trim lookups, preserve anti-enumeration by @gorkem-bwl in #4185
fix(file-manager): authorize org-level file access by organization_id by @gorkem-bwl in #4192
fix(ai-trust-center): create logo row on first upload instead of failing by @gorkem-bwl in #4335
RLS, Auth, HTTPS & DAST/CI Scanning by @MuhammadKhalilzadeh in #4304
chore(deps): bump the pip group across 1 directory with 3 updates by @dependabot[bot] in #4327
chore(deps): bump the uv group across 2 directories with 5 updates by @dependabot[bot] in #4328
Refactor/policy editor split 1 by @Innagl in #4337
Refactor/policy editor split 2 by @Innagl in #4339
Refactor/policy editor split 3 by @Innagl in #4341
Policy Editor: Optional Team Member's Input by @MuhammadKhalilzadeh in #4383
docker(frontend): bump nginx from 1.31.2-alpine to 1.31.3-alpine in /Clients by @dependabot[bot] in #4342
ci: bump actions/github-script from 7 to 9 by @dependabot[bot] in #4343
ci: bump actions/setup-node from 6 to 7 by @dependabot[bot] in #4344
ci: bump actions/setup-python from 6 to 7 by @dependabot[bot] in #4345
deps(eval): update mistralai requirement from >=2.4.9 to >=2.7.2 in /EvaluationModule by @dependabot[bot] in #4348
deps(eval): update pyarrow requirement from >=24.0.0 to >=25.0.0 in /EvaluationModule by @dependabot[bot] in #4349
deps(eval): update datasets requirement from >=4.8.5 to >=5.0.1 in /EvaluationModule by @dependabot[bot] in #4350
deps(backend): bump @types/node from 25.9.1 to 26.1.1 in /Servers by @dependabot[bot] in #4352
deps(backend): bump @slack/web-api from 7.16.0 to 8.0.0 in /Servers by @dependabot[bot] in #4353
deps(backend): bump @azure/msal-node from 3.8.10 to 5.4.2 in /Servers by @dependabot[bot] in #4358
deps(frontend): bump @testing-library/jest-dom from 6.9.1 to 7.0.0 in /Clients by @dependabot[bot] in #4360
deps(frontend): bump concurrently from 9.2.4 to 10.0.4 in /Clients by @dependabot[bot] in #4361
deps(frontend): bump typescript from 6.0.3 to 7.0.2 in /Clients by @dependabot[bot] in #4362
deps(eval): update openai requirement from >=2.44.0 to >=2.49.0 in /EvaluationModule by @dependabot[bot] in #4363
docker(evalserver): bump python from 3.12-slim to 3.14-slim in /EvalServer by @dependabot[bot] in #4364
deps(eval): update deepeval requirement from >=4.0.3 to >=4.1.4 in /EvaluationModule by @dependabot[bot] in #4365
ci: bump softprops/action-gh-release from 2 to 3 by @dependabot[bot] in #4366
docker(aigateway): bump python from 3.12-slim to 3.14-slim in /AIGateway by @dependabot[bot] in #4367
deps(evalserver): bump google-api-core from 2.28.1 to 2.33.0 in /EvalServer by @dependabot[bot] in #4369
deps(evalserver): bump multidict from 6.7.0 to 6.7.1 in /EvalServer by @dependabot[bot] in #4372
deps(aigateway): update presidio-analyzer requirement from >=2.2.0 to >=2.2.364 in /AIGateway by @dependabot[bot] in #4371
deps(aigateway): update spacy requirement from >=3.7.0 to >=3.8.14 in /AIGateway by @dependabot[bot] in #4373
deps(aigateway): update python-dotenv requirement from >=1.0.0 to >=1.2.2 in /AIGateway by @dependabot[bot] in #4374
deps(evalserver): bump yarl from 1.22.0 to 1.24.5 in /EvalServer by @dependabot[bot] in #4375
deps(aigateway): update pydantic requirement from >=2.5.0 to >=2.13.4 in /AIGateway by @dependabot[bot] in #4376
deps(aigateway): update httpx requirement from >=0.27.0 to >=0.28.1 in /AIGateway by @dependabot[bot] in #4377
chore(deps): bump the npm_and_yarn group across 4 directories with 10 updates by @dependabot[bot] in #4379
refactor(ai-detection): split ScanDetailsPage into co-located modules by @Innagl in #4382
Potential fix for code scanning alert no. 1181: Server-side request forgery by @MuhammadKhalilzadeh in #4386
Roster of Agents by @MuhammadKhalilzadeh in #4390
Dependency Security Patch by @MuhammadKhalilzadeh in #4418
Dependency Security Patch by @MuhammadKhalilzadeh in #4419
Remediate CodeQL polynomial ReDoS Alerts by @MuhammadKhalilzadeh in #4420
Security & Quality Patch by @MuhammadKhalilzadeh in #4426
Added observability with otel by @HarshP4585 in #4223
refactor(evals): extract pure helpers from NewExperimentModal by @Innagl in #4388
Fix superadmin logger error by @HarshP4585 in #4391
4024 add e2e coverage for a critical user journey by @DeboraSerra in #4393
deps(eval): update sentence-transformers requirement from >=5.6.0 to >=5.6.1 in /EvaluationModule by @dependabot[bot] in #4394
Security & Quality Patch by @MuhammadKhalilzadeh in #4437
feat(reporting): template-first reporting stack — templates, schedules, runs, delivery and framework selection by @haJ1t in #4389
docs(claude): add project-specific build, PR, and UI component rules by @gorkem-bwl in #4444
deps(eval): update transformers requirement from >=5.12.1 to >=5.14.1 in /EvaluationModule by @dependabot[bot] in #4397
deps(eval): update anthropic requirement from >=0.111.0 to >=0.120.2 in /EvaluationModule by @dependabot[bot] in #4398
deps(aigateway): update presidio-anonymizer requirement from >=2.2.0 to >=2.2.364 in /AIGateway by @dependabot[bot] in #4399
deps(aigateway): update asyncpg requirement from >=0.30.0 to >=0.31.0 in /AIGateway by @dependabot[bot] in #4400
deps(evalserver): bump regex from 2025.11.3 to 2026.7.19 in /EvalServer by @dependabot[bot] in #4401
deps(evalserver): bump posthog from 7.28.0 to 7.35.0 in /EvalServer by @dependabot[bot] in #4402
deps(aigateway): update fastapi requirement from >=0.120.0 to >=0.141.1 in /AIGateway by @dependabot[bot] in #4403
deps(evalserver): bump importlib-metadata from 8.7.0 to 9.0.0 in /EvalServer by @dependabot[bot] in #4406
deps(backend): bump google-auth-library from 10.6.2 to 11.0.0 in /Servers by @dependabot[bot] in #4410
deps(backend): bump jsdom from 29.1.1 to 30.0.1 in /Servers by @dependabot[bot] in #4414
deps(frontend): bump jsdom from 29.1.1 to 30.0.1 in /Clients by @dependabot[bot] in #4417
deps(evalserver): bump redis from 7.0.1 to 8.1.0 in /EvalServer by @dependabot[bot] in #4408
chore(deps): bump the npm_and_yarn group across 1 directory with 2 updates by @dependabot[bot] in #4442
deps(eval): update textblob requirement from >=0.20.0 to >=0.20.1 in /EvaluationModule by @dependabot[bot] in #4445
ci: bump actions/setup-python from 6 to 7 by @dependabot[bot] in #4446
ci: bump actions/setup-node from 6 to 7 by @dependabot[bot] in #4448
ci: bump actions/checkout from 6 to 7 by @dependabot[bot] in #4449
deps(eval): update numpy requirement from <3.0.0,>=1.26.4 to >=2.5.1,<3.0.0 in /EvaluationModule by @dependabot[bot] in #4396
deps(evalserver): bump tabulate from 0.9.0 to 0.10.0 in /EvalServer by @dependabot[bot] in #4404
deps(aigateway): update redis requirement from >=5.0.0 to >=8.1.0 in /AIGateway by @dependabot[bot] in #4407
deps(backend): bump bullmq from 5.77.6 to 6.0.0 in /Servers by @dependabot[bot] in #4413
Migrate plugins (custom frameworks) by @HarshP4585 in #4443
deps(backend): bump ai from 6.0.191 to 7.0.58 in /Servers by @dependabot[bot] in #4415
Feat: Project Risk Validation by @Br0wnHammer in #4425
refactor(evals): extract Dataset step from NewExperimentModal by @Innagl in #4427
Refactor/new experiment split 3 by @Innagl in #4430
4030 audit and stabilize flaky tests by @DeboraSerra in #4435
Refactor: Policy Editor by @Br0wnHammer in #4436
Refactor/project datasets split by @Innagl in #4439
deps(aigateway): update uvicorn requirement from >=0.32.0 to >=0.52.1 in /AIGateway by @dependabot[bot] in #4450
ESLint Auto lint check before commit by @MuhammadKhalilzadeh in #4456
Code-Scanning Remediation Report by @MuhammadKhalilzadeh in #4460
Allow more than 1 superadmin by @HarshP4585 in #4457
Fix/approval automations validation by @Innagl in #4459
Feat: Optimistic updates and targeted cache by @Br0wnHammer in #4461
Fix: Standardize API Error by @Br0wnHammer in #4462
fix(approvals): stretch workflow step divider when validation error appear by @Innagl in #4463
docker(evalserver): bump python from 3.12-slim to 3.14-slim in /EvalServer by @dependabot[bot] in #4464
docker(aigateway): bump python from 3.12-slim to 3.14-slim in /AIGateway by @dependabot[bot] in #4465
deps(eval): update anthropic requirement from >=0.120.2 to >=0.122.0 in /EvaluationModule by @dependabot[bot] in #4466
deps(eval): update numpy requirement from <3.0.0,>=2.5.1 to >=2.5.2,<3.0.0 in /EvaluationModule by @dependabot[bot] in #4467
deps(eval): update sentence-transformers requirement from >=5.6.1 to >=5.7.0 in /EvaluationModule by @dependabot[bot] in #4468
deps(eval): update mistralai requirement from >=2.7.2 to >=2.9.3 in /EvaluationModule by @dependabot[bot] in #4469
deps(eval): update transformers requirement from >=5.14.1 to >=5.15.0 in /EvaluationModule by @dependabot[bot] in #4470
deps(evalserver): bump aiohappyeyeballs from 2.6.1 to 2.7.1 in /EvalServer by @dependabot[bot] in #4471
deps(evalserver): bump pytest-asyncio from 0.25.3 to 1.4.0 in /EvalServer by @dependabot[bot] in #4472
deps(evalserver): bump grpcio-status from 1.71.2 to 1.83.0 in /EvalServer by @dependabot[bot] in #4473
deps(aigateway): update opentelemetry-api requirement from >=1.27.0 to >=1.44.0 in /AIGateway by @dependabot[bot] in #4474
deps(aigateway): update pydantic-settings requirement from >=2.0.0 to >=2.15.0 in /AIGateway by @dependabot[bot] in #4475
deps(aigateway): update uvicorn requirement from >=0.52.1 to >=0.52.3 in /AIGateway by @dependabot[bot] in #4476
deps(aigateway): update alembic requirement from >=1.13.0 to >=1.19.1 in /AIGateway by @dependabot[bot] in #4477
deps(aigateway): update spacy requirement from >=3.8.14 to >=3.8.15 in /AIGateway by @dependabot[bot] in #4478
deps(evalserver): bump google-api-core from 2.33.0 to 2.34.0 in /EvalServer by @dependabot[bot] in #4479
deps(evalserver): bump certifi from 2025.10.5 to 2026.7.22 in /EvalServer by @dependabot[bot] in #4480
deps(backend): bump @ai-sdk/anthropic from 3.0.79 to 4.0.38 in /Servers by @dependabot[bot] in #4482
deps(backend): bump ioredis from 5.11.0 to 6.0.0 in /Servers by @dependabot[bot] in #4484
deps(backend): bump eslint from 9.39.5 to 10.8.1 in /Servers by @dependabot[bot] in #4485
deps(backend): bump @types/jsdom from 28.0.3 to 30.0.0 in /Servers by @dependabot[bot] in #4487
deps(frontend): bump typescript from 5.9.3 to 7.0.2 in /Clients by @dependabot[bot] in #4489
deps(frontend): bump eslint from 9.39.5 to 10.8.1 in /Clients by @dependabot[bot] in #4490
deps(frontend): bump @types/pako from 2.0.4 to 3.0.0 in /Clients by @dependabot[bot] in #4491
deps(backend): bump cron-parser from 4.9.0 to 5.9.0 in /Servers by @dependabot[bot] in #4483
deps(backend): bump @ai-sdk/openai from 3.0.65 to 4.0.42 in /Servers by @dependabot[bot] in #4486
deps(frontend): bump lint-staged from 15.5.2 to 17.3.0 in /Clients by @dependabot[bot] in #4492
deps(frontend): bump @mui/material from 7.3.11 to 9.3.1 in /Clients by @dependabot[bot] in #4493
feat(vendors): add placeholders on vendor name and contact person by @Innagl in #4495
Feat: Evalserver Pydantic by @Br0wnHammer in #4494
4051 raise frontend line coverage from 30 to at least 50 by @DeboraSerra in #4496
feat(users): add self-scoped GET/PATCH /users/me/preferences by @Innagl in #4510
fix(security): resolve open code-scanning alerts by @MuhammadKhalilzadeh in #4511
Migrate the existing plugins to extension by @HarshP4585 in #4512
add frontend unit tests for shared component directories by @DeboraSerra in #4513
chore(aigateway): bump litellm 1.89.3 -> 1.97.0 by @gorkem-bwl in #4514
docs(whats-new): add v2.5 release notes by @gorkem-bwl in #4516
Merge develop into master (Aug 20) by @HarshP4585 in #4517
Merge develop into master (Aug 20) by @HarshP4585 in #4518
Merge develop into master (Aug 20) by @HarshP4585 in #4519
Merge develop into master (Aug 21) by @HarshP4585 in #4520
Fix Trivy image scan issue by @HarshP4585 in #4521
Merge develop into master (22 Aug) by @HarshP4585 in #4523

Full Changelog: v2.4.1...v2.5

Navigation Menu

Uh oh!

v2.5

Choose a tag to compare

Sorry, something went wrong.

Sorry, something went wrong.

Uh oh!

No results found

What's new

What's Changed

Contributors

Uh oh!