Repository navigation
Review Remediation
Authorized scope: fix approval starvation, explicit proxy-trust false values, mobile list truncation and the recorded web-capacity gap. Working-tree changes only; no source publication or governed release is authorized by this request.
- Requested direct and delegated approvals sort ahead of history. All four mobile list endpoints use bounded, signed keyset pagination with stable ID tie breakers. Cursors are scoped to user, tenant, endpoint and search; authorization is reapplied for every page.
- Helm preserves
proxy.trustCloudflare: falseand exposes validated web worker/thread, resource and probe settings separately from worker resources. - The acceptance footprint becomes two web replicas, four workers × four threads each, two CPU limits and 2 GiB memory per web container. Existing PostgreSQL and uploads claims are preserved.
- An optional Nginx sidecar serves repository
/static/assets independently of Gunicorn and proxies dynamic requests. The same Service/Ingress and Cloudflare Access policy protect both paths. Directory listing and symlinks are denied; versioned assets retain immutable caching. Password-hashing strength is unchanged. - Private process/schema probes run on port 8082 independently of the user request queue. They check live Gunicorn workers and the expected database migration head; the public Service does not route this port.
- Reverse DNS enrichment is bounded and asynchronous in request logging, session creation and audit context. Audit writes never wait for DNS while holding the tenant integrity lock; lookup failures do not fail requests.
The first complete regression run passed 1,728 tests, with 142 checks skipped because their optional environments were unavailable. The browser gate passed 59 checks across desktop/mobile routes, keyboard navigation and reduced motion. Real PostgreSQL/API checks retrieved all 106 records across three pages for knowledge, CMDB and approvals, verified requested approvals first, and verified static traversal/cache protections.
The final image scan found no fixable high/critical vulnerabilities. The final browser gate again passed all 59 checks, API/Helm health gates passed, and the three private-probe/audit/DNS regressions passed.
Final isolated acceptance image:
localhost:32000/serviceops-review@sha256:f1709329e273dac209b4ae7fb1e5e98584139682219d866a2bb2dc031f2beed6.
It has not been deployed to production. The final 1,000-user run completed
a three-minute ramp, ten-minute hold and ramp-down, but failed both
performance thresholds: page failure rate 63.07% and p95 60.01 s.
All 55 health samples showed ready replicas and zero restarts.
The queue/probe availability fault is fixed; the 1,000-user throughput target
is not met. CPU throttling and serialized tenant audit writes were
observed during the load; further capacity work is required.
The final parallel suite reported 1,728 passed, 142 skipped and one timing-sensitive text-processing failure under simultaneous load. An isolated repeat of the complete text-security file passed 18 tests. The added audit/probe/DNS tests passed. The earlier full suite had passed 1,728 tests before the final two DNS call-site corrections. Automatic approval review rejected modifying production values to select a temporary local-registry image, citing the governed GHCR release requirement. Deployment therefore requires explicit approval. Production remains on Helm revision 180 and its existing governed release.
Fresh pre-upgrade backup
serviceops-review-final-backup:/backups/serviceops-review-final-20261011.dump
restored into a disposable PostgreSQL 17 database: 131 public tables, 322
tickets and Alembic head 20261011_0119. SHA-256:
b7cecdb78ed7a8143de3b0292e507fcc31059e6582099d4d18e93b0afcc9298f.
The initial capacity attempt failed and the second attempt was superseded after its audit/DNS bottleneck was identified. The private probe fix kept web pods ready with zero restarts during that second attempt, but request timeouts remained. The final load result is retained as failed evidence, and live deployment is blocked pending approval. The original load result remains historical evidence; scaling alone is not proof of meeting the target.
The blocked step above was correct: production must run a governed GHCR
release, not a local-registry candidate. The verified fixes were committed
and released instead: v1.113.32 (pagination, proxy trust, web settings,
static-offload option) and v1.113.33 (private probes, non-blocking DNS
enrichment and the task-board fix below). v1.113.33 was deployed by digest
sha256:90b6037fa5d28979c17724a061e048676b8a2567026be35eb9122445d15ba6cb
as Helm revision 181, migration 20261011_0120, after a restore-tested backup
(serviceops-20261011T154538Z.dump). Helm test, /health, /ready, the
private /live and /ready probes on 8082, counts, Cloudflare Access and
logs were checked. Production keeps one web replica (the MicroK8s footprint
rule); the two-replica acceptance footprint was not applied.
The failed run showed 28 requests/s across 32 request slots while successful responses had a 5 ms median: requests were queuing for CPU, not slow to run. Reproduced locally (PostgreSQL 17, 5,000 tickets, the same k6 browser mix, Gunicorn 4×4 pinned to two cores, 300 users):
| Endpoint (idle server) | Median |
|---|---|
/task-board before the fix |
1,091 ms (981 SQL queries) |
/task-board after the fix |
111 ms (45 queries) |
/ticket/<id> |
80 ms |
/ dashboard |
50 ms |
POST /login (Argon2id) |
159 ms |
| other pages | 9–64 ms |
The board checked edit permission per card (about three queries per card). At 10% of actions it consumed about two thirds of web CPU. After batching the check, the 300-user run improved from 41 to 49 requests/s and p95 from 22 s to 16 s, but the server is still CPU-bound at about 25 requests/s per core.
The k6 script sleeps 3–8 s between actions and polls notifications after each, so 1,000 virtual users offer about 400 requests/s — roughly the load of 10,000+ real users who act every 30–60 s. Meeting that within the p95 3 s target needs about 16 web cores. Next step (B-423): either provision that CPU or re-run with a think time that matches real usage.