UI performance and scaling improvements

[sriaradhyula]

Goal

Use this discussion as the umbrella for CAIPE UI performance and scaling work.

The goal is to identify where the UI/BFF path stops scaling, make small measurable improvements, and verify each change with repeatable Locust benchmarks.

PRs in flight

• perf(ui): cache health and RBAC gate checks #1949 — perf(ui): cache health and RBAC gate checks
  • Short-lived BFF response caching for high-repeat UI endpoints.
  • In-flight request coalescing so concurrent identical requests share one upstream computation.
  • OpenFGA store-id discovery cached per UI process.
• fix(rbac): address admin and chat regressions #1950 — fix(rbac): address admin and chat regressions
  • Keeps the optimized RBAC/admin path correct for super-admin and chat navigation flows.
  • Ensures benchmark results reflect usable UI behavior, not just faster broken paths.
• docs(audit): document audit performance benchmarks #1951 — docs(audit): document audit performance benchmarks
  • Captures benchmark evidence and the honest before/after story.
  • Tracks where improvements are real and where p95/p99 tail latency still needs work.

Benchmark docs and runbooks

PR #1951 adds the first checked-in benchmark docs and run helpers for this discussion:

• docs/docs/evaluations/ui-performance-benchmark-results.md — consolidated UI performance benchmark results, including the 5/10/50/100/300/500/1000 user test plan and the honest readout that 300-user behavior improved while 1000-user p95/p99 tail latency still needs follow-up.
• docs/docs/security/rbac/audit-log-performance.md — audit-service performance context, backend comparison, local-vs-S3 behavior, and operational interpretation.
• scripts/locustfile.py — repeatable Locust workload for UI/BFF navigation and audit-service scenarios.
• ui/mint-test-session.mjs — local helper for minting a test browser/session token before running UI benchmarks.
• docs/docs/specs/2026-06-18-audit-logs-local-storage/ — supporting spec, data model, env vars, and task notes for the audit-service storage split.

Merge-readiness note for #1951:

• Rebased onto current main.
• CI is green, including Docusaurus build, RBAC docs sync, lint/test, Playwright RBAC regression, CodeQL, DCO, and image builds.
• The PR is MERGEABLE, CLEAN, and APPROVED.

These docs should become the canonical place to record future benchmark runs from this discussion. For each new optimization PR, add a before/after row with user count, throughput, p50, p95, p99, error rate, slowest endpoints, and the health state of audit-service/OpenFGA/Mongo/Keycloak/RAG/dynamic-agents during the run.

Improvement areas

1. BFF response caching and request coalescing

Reduce duplicate work from repeated UI navigation, tab switches, health polling, and refresh bursts.

flowchart LR
  Browser["Browser UI"] --> BFF["Next.js BFF"]
  BFF --> Cache{"Short-lived JSON cache\nkeyed by URL + caller headers"}
  Cache -->|hit| Browser
  Cache -->|miss| Inflight{"In-flight request map"}
  Inflight -->|same key already running| Shared["share upstream result"]
  Inflight -->|first caller| Compute["compute response once"]
  Compute --> OpenFGA["OpenFGA"]
  Compute --> Mongo["MongoDB"]
  Compute --> Health["health probes"]
  Compute --> DA["dynamic-agents"]
  Compute --> Cache
  Shared --> Browser

Loading

Expected impact:

• Fewer repeated OpenFGA checks for admin/RBAC gates.
• Fewer repeated platform-health probes during page load and polling.
• Less fan-out from identical concurrent requests under Locust load.
• Better p50 and throughput at medium concurrency.

2. OpenFGA store discovery cache

Avoid rediscovering the same store id on every tuple/check helper call.

sequenceDiagram
  participant BFF as Next.js BFF
  participant Cache as Store ID cache
  participant FGA as OpenFGA

  BFF->>Cache: get store id
  alt OPENFGA_STORE_ID set
    Cache-->>BFF: configured store id
  else first request
    Cache->>FGA: list/read stores by OPENFGA_STORE_NAME
    FGA-->>Cache: store id
    Cache-->>BFF: store id
  else later request
    Cache-->>BFF: cached store id
  end
  BFF->>FGA: check/read/write with store id

Loading

Expected impact:

• Removes repeated store lookup latency from hot RBAC paths.
• Reduces OpenFGA request volume during admin navigation and benchmarks.

3. Separate audit logs into an audit-service backend

Audit storage should be owned by a dedicated audit-service, not by the UI container. The UI/BFF, OpenFGA authz bridge, and Python services become audit event producers; audit-service becomes the only component that writes, reads, filters, exports, and purges audit log storage.

flowchart LR
  subgraph Producers["Audit event producers"]
    UI["Web UI / BFF"]
    Bridge["OpenFGA authz bridge"]
    Py["Python services"]
    Bots["Slack / Webex bots"]
  end

  UI -->|POST /v1/audit/events\nJSON batch| AuditSvc["audit-service FastAPI"]
  Bridge -->|POST /v1/audit/events\nJSON batch| AuditSvc
  Py -->|POST /v1/audit/events\nJSON batch| AuditSvc
  Bots -->|POST /v1/audit/events\nJSON batch| AuditSvc

  AuditSvc --> Normalize["validate + normalize"]
  Normalize --> Queue["bounded asyncio.Queue"]
  Queue -->|full / service down| Drop["warn + drop\nnon-breaking audit path"]
  Queue --> Worker["background flush worker"]

  Worker --> Select{"AUDIT_SERVICE_BACKEND"}
  Select -->|local default| Local["local NDJSON.gz\n/var/lib/caipe-audit-service"]
  Select -->|s3 when configured| S3["S3 Parquet objects\ntime/date partitions"]
  Local --> Purge["local retention purge\ndefault 1 day"]

  AuditUI["RBAC Audit UI / operators"] -->|GET /v1/audit/events| AuditSvc
  AuditUI -->|GET /v1/audit/events.zip| AuditSvc
  AuditSvc --> Local
  AuditSvc --> S3

Loading

Backend ownership split:

• UI no longer writes local files, S3 objects, MongoDB audit records, DuckDB files, or Parquet files directly.
• audit-service owns local disk writes, S3 writes, retention cleanup, read queries, and ZIP export.
• Producers send normalized JSON batches to audit-service and treat failures as non-breaking: warn in health/status, drop audit messages when audit-service is offline, and never block core product flows.
• The RBAC Audit UI reads from audit-service, including time-window filters and raw JSON ZIP downloads.
• Storage selection is runtime configuration: local disk by default; S3/Parquet when bucket and credentials are configured.
• Local disk has a default one-day purge so development and local Compose do not fill the host disk.

sequenceDiagram
  participant User as Browser user
  participant UI as Web UI / BFF
  participant Audit as audit-service
  participant Store as local disk or S3
  participant Health as platform health

  User->>UI: perform RBAC/admin/chat action
  UI-->>User: respond on product path
  UI-)Audit: enqueue audit JSON batch
  alt audit-service healthy
    Audit->>Audit: validate + enqueue
    Audit-)Store: background flush
  else audit-service offline or queue full
    UI->>Health: surface warning status
    UI-->>UI: drop audit event without failing action
  end

Loading

Expected impact:

• UI navigation does not block on local disk, DuckDB, Parquet, MongoDB audit writes, or S3 write work.
• Audit write latency moves out of the synchronous UI request path.
• Audit read/search/export load is isolated behind one service owner.
• Health status can clearly say whether audit storage is local, S3, healthy, degraded, or unavailable.

4. Admin and chat regression fixes

Performance improvements must preserve behavior for super-admin and chat flows.

flowchart TD
  Login["Signed-in user"] --> Gates["BFF auth + RBAC gates"]
  Gates --> Admin["Admin dashboard tabs"]
  Gates --> Chat["Chat conversation state"]
  Gates --> Audit["RBAC Audit UI"]
  Gates --> Deny["fail closed only when truly unauthorized"]

  Fixes["#1950 regression fixes"] --> Gates
  Cache["#1949 cache/coalescing"] --> Gates

Loading

Expected impact:

• Super-admin access should remain correct while gates are cached/coalesced.
• Chat should not get stuck in conversation-state checks due to stale or incorrect gate behavior.
• Benchmarks should measure valid product flows.

5. Benchmark documentation and repeatability

Keep the benchmark story reproducible and honest.

flowchart LR
  Code["PR under test"] --> Compose["docker-compose.dev.yaml"]
  Compose --> Locust["Locust run"]
  Locust --> Report["HTML/CSV results"]
  Report --> Docs["docs/docs benchmark notes"]
  Report --> Followups["focused follow-up issues/PRs"]

Loading

Expected impact:

• Each PR can be evaluated independently where possible.
• We can separate 300-user wins from 1000-user tail-latency problems.
• Future performance work starts from measured bottlenecks, not anecdotes.

What we want to learn

• Which UI surfaces create repeated backend work during normal navigation?
• Which backend dependencies dominate latency under load: OpenFGA, MongoDB, Keycloak, audit-service, RAG, dynamic-agents, or health probes?
• Where do p50 improvements hide p95/p99 tail-latency regressions?
• Which requests serialize under concurrency and should be cached, coalesced, batched, or moved off the critical path?
• What breaks first in full Docker Compose as concurrency increases?

Benchmark plan

Use Locust to run repeatable UI/BFF load tests at these user counts:

• 5 users
• 10 users
• 50 users
• 100 users
• 300 users
• 500 users
• 1000 users

For each run, capture:

• Throughput / requests per second
• p50, p95, and p99 latency
• Error rate
• Slowest endpoints
• Container CPU and memory hotspots where available
• Whether audit-service, OpenFGA, MongoDB, Keycloak, RAG, and dynamic-agents were healthy during the run

Candidate UI flows to stress

• Home page load and platform health polling
• Admin dashboard tab navigation
• Security & Policy / RBAC Audit view
• OpenFGA ReBAC/admin gate checks
• Chat page load and conversation state checks
• Dynamic agent availability and picker loading
• Knowledge/RAG status surfaces
• Audit event search and ZIP download

Success criteria

• Each PR has a clear before/after benchmark story.
• Improvements are measured independently when possible.
• 300-user results should show meaningful navigation improvement without increasing errors.
• 1000-user results should separate throughput/p50 gains from p95/p99 tail-latency issues.
• Any remaining bottleneck becomes a focused follow-up PR or issue, not a vague performance bucket.

Current hypothesis

#1949 should reduce duplicate BFF work from repeated health, RBAC gate, stats, config, and dynamic-agent availability calls. Audit-service offload should keep audit writes and reads out of the UI container's hot path. Together, those should improve normal navigation and medium-load behavior.

At 1000 users, tail latency may still be dominated by full-compose saturation, serialized downstream checks, cold probes, audit read fan-out, or backend dependency contention; Locust runs should tell us which one.