Repository navigation
Implementation Plan
Released: v1.113.40 (release commit 7411091, image ghcr.io/awijesundara/serviceops-server@sha256:eaceee78…ea0b, SLSA provenance verified). It contains B-438 to B-441 and migration 20261013_0122.
Running now: candidate 60fc1cb (the same features before final QA fixes), Helm revision 188, migration 20261013_0122, events service and nginx edge enabled. A restore-tested backup for the release upgrade was taken (serviceops-20261013T143803Z.dump: 1,038 users, 322 tickets, 132 tables). The release upgrade waits for the owner's go-ahead.
Done (2026-10-12 to 2026-10-13)
| Item | Where |
|---|---|
| CI owning-team field only on main pickers; stale rack bar hidden (B-433) | 1.113.34 |
Whole-application review: MFA lockout and single-use codes, equal-time unknown usernames, no-store, complete OpenAPI, worker PUBLIC_BASE_URL (B-434) |
1.113.35 |
| Malformed input answers 400, never 500; improvement owner tenant check; stability sweep tool (B-435) | 1.113.36 |
| 500-user evaluation; dashboard counts in SQL; compiled database driver (B-436) | 1.113.37, 1.113.38 |
| Review fixes: atomic TOTP replay guard, reset on re-enrolment, length check on changed values only (B-437) | 1.113.39 |
| Compact "View location" link; assignee profile avatar; plain name links; aligned profile panels; My tasks shows only my teams (B-438) | 1.113.40 |
Live communication: ETag'd checks, one checking tab per browser, server-sent events via PostgreSQL LISTEN/NOTIFY, worker wake-up, browser Web Push (B-439) |
1.113.40 |
| Task board drop into empty lanes; approval chains link records, larger history toggle (B-440) | 1.113.40 |
| Optional RT reference on changes with automatic fetch (B-441) | 1.113.40 |
| Search indexes and faster ticket search, per-tenant asset tags, SCIM paging, live work lists, dead MFA helper removed (B-442) | Committed, next release |
Deep QA before v1.113.40 (2026-10-13)
| Check | Result |
|---|---|
| Full unit and integration suite (4 chunks) | 1,789 passed, 0 failed |
| Browser smoke: desktop, mobile, wide, keyboard, focus, reduced motion, 320 px reflow, axe | 100 passed (found and fixed: pages requested /events/stream where it is not routed, a console 404) |
| Opt-in browser tests: AI (PostgreSQL), rack location, admin assignment | 29 passed; two AI tests described the earlier interface and were updated |
| Feature browser QA (board, approvals, RT fetch, no script errors) | 15/15 |
| PostgreSQL live signals, events service end to end, worker wake | Passed |
| Stability fuzz, every page/form/POST/API, four roles, SQLite and PostgreSQL | 0 failing routes |
| Cross-tenant sweep of every id-based route | 3,710 requests, no leaks |
| Lint (ruff), CodeQL, supply-chain gate | Passed |
Waiting on the owner
- Approve the production upgrade to v1.113.40 (backup ready; values prepared).
- Scale production for 500 users: 4 web pods (1 CPU each) and PostgreSQL at 2 CPU. Needs both nodes and departs from the one-replica rule. Performance
- Commit and release the request-scoped permission caches (reviewed, all tests pass, 3–9 fewer queries per page). They are uncommitted in the local checkout. Consider a smaller Gunicorn
max-requeststhan 10,000. - Grant the deployment token write access to
awijesundara/k8sso deployment history can be pushed. - Delete superseded GitHub releases and tags (v1.113.30 to v1.113.38) and completed one-off Jobs in
operations. - Plan
API_TOKEN_PEPPER(re-issues every API and mobile token), setMETRICS_TOKEN, and document key rotation. - Enable Web Push (
WEB_PUSH_ENABLED) and the RT change link (RT_CHANGE_LINK_ENABLED) when wanted; both are off by default.
Project management (started 2026-10-14)
A twelve-phase programme brings every Jira Software project management capability into ServiceOps (scope narrowed to Jira by the owner on 2026-10-14), connected to incidents, problems and changes: Project management plan, with a complete Jira inventory (B-443, B-449 to B-459). Phases 1 and 2 (projects and work items; Jira work item completeness) are implemented and tested on a branch; phase 3 (JQL and filters) is next.
Next engineering work
The previous list is done (B-442, committed locally for the next release after v1.113.40 is deployed). Remaining candidates:
| Work | Why |
|---|---|
| Live updates for approvals and notifications pages | They still rely on the bell and timed checks |
| Translations for strings added since 2026-10-11 | Other languages show English for the new live-update, Web Push and RT texts |
B-436 sized ServiceOps for 500 all-day users on an isolated copy, fixed the two largest CPU costs (dashboard counting, pure-Python database driver) and deployed v1.113.38 at Helm revision 185. Production still runs one web pod; moving to four pods and a 2-CPU database is a pending decision. Performance.
B-435 makes malformed input answer 4xx instead of 500 across forms, POST routes and the API, on SQLite and PostgreSQL, with a reusable sweep (tools/stability_fuzz.py). v1.113.36 is deployed at Helm revision 184.
B-434 re-reviewed the application and cross-checked every wiki page against the source and the running deployment. Second-factor sign-in now locks accounts on repeated wrong codes and refuses replayed codes; unknown usernames take as long as real ones; signed-in responses are not stored; the OpenAPI document lists every /api/v1 route and a test keeps it complete. v1.113.35 is deployed at Helm revision 183, migration 20261012_0121, after a restore-tested backup and provenance verification. Remaining operational items are listed in the backlog entry.
B-433 limits the CI owning team field to the main CI pickers, hides a stale rack location bar while a different CI is picked, and labels the Affected CIs relationship select. v1.113.34, Helm revision 182.
B-430 implements the four review corrections. API/chart regressions, 59 isolated browser checks, real PostgreSQL pagination checks and the final image scan pass; a fresh backup restored 131 tables and 322 tickets at head 20261011_0119. Private process/schema probes and asynchronous audit/session DNS remove request-queue and DNS availability bottlenecks. The final 1,000-user run failed the performance gate (63.07% page failures; p95 60.01 s), while replicas stayed ready without restarting. The repeated suite passed 1,728 tests with one timing-sensitive failure; its entire text-security file passed 18 tests when repeated in isolation. Further throughput work remains. Production remains at Helm revision 180: automatic approval review rejected selecting a temporary local-registry image, citing the governed release requirement. No source publication or governed release is claimed. Scope and evidence.
B-428 adds document-only live previews, reset and explicit saved/preview feedback; the existing account form remains the save boundary. Five additional palettes expand the picker to eleven choices. The review also covers shared surfaces, keyboard focus, non-color selection indicators, responsive preferences and embedded rack themes. No schema change. Ten desktop/mobile theme and workspace checks passed, including maximum account font scale; 43 localization/existing-theme checks passed. Lint, generated dark stylesheet and version consistency checks passed. The full suite passed: 1,705 tests, 141 environment/opt-in checks skipped. The signed source commit is 667ae7a9d08f93328e5c7f232e4cf1703e079c7d. Governed v1.113.30 passed source/versioned application, browser/accessibility, dependency/image security and all five RPM build/install gates. The release commit and tag are verified. Independent image signature/provenance and all five RPM checksum/provenance checks passed. v1.113.30 is deployed at Helm revision 179, image digest sha256:f0e28a090b89e9ae1f1d6a94ec069ff94a137773021ec1b1cf861f802a0d78aa. Web/outbox/AI are ready with zero restarts; retained Helm test, health/readiness, public API and Access checks passed. The pre-upgrade backup restored 131 public tables and 322 tickets at unchanged schema head 20261011_0118. PostgreSQL/uploads storage identities are preserved. Delivery evidence. Following explicit user approval, 17 older releases and 38 older tags were removed; only verified v1.113.30 remains. Source commit history and GHCR images are preserved. Design and review.
B-427 fixes the confirmed draft-to-form omission: rich draft fields remain server-side and transfer by an owned run reference, avoiding URL truncation. CI selections come from supplied evidence and are reauthorized on open; owning teams must be eligible. Planned timestamps use the requester profile timezone and convert to the existing UTC form convention. Incomplete drafts list missing fields. Risk remains calculated; no ticket is created before normal human submission. Explicitly named visible CIs are retrieved ahead of generic inventory matches; conflicting draft source selections are rejected. The full local suite passed (1,700 tests); explicit desktop/mobile Chromium checks verify all fields, CI chips and submission. The narrow-screen lifecycle bar now scrolls internally.
B-426 stores authenticated mobile origin on comments and record activity, shows an iOS badge alongside AI-assisted, and returns source metadata to API clients. Migration 20261011_0118 adds nullable columns to preserve existing records; migration preservation/idempotency and forged-header negatives passed. The signed v1.113.29 release is deployed at MicroK8s Helm revision 178. Source and versioned application/security/browser checks, all five RPM targets and independent checksums/provenance passed. The pre-upgrade backup restored 131 tables and 321 tickets at schema head 20261011_0118. Web/outbox/AI are ready on the verified image digest with zero restarts; retained Helm test, public API, health/readiness, Access and storage identity checks passed. The approved live Google Gemini draft-only test selected the correct CI and team, returned profile-timezone dates and all three plans, and retained them in the review form. No ticket was created; temporary conversations were removed. Delivery evidence.
B-425 requires a deletion reason and preserves a deleted notice with the original comment. The ticket event retains filename, reason, actor/time and a versioned comment reference, without modifying the original comment or requiring a migration. File bytes are removed after the deletion event commits. Invalid reasons and denied permissions preserve the file; failed commits roll back; unexpected failures are logged with sanitized user feedback. Threaded comment notices and API metadata are escaped/serialized normally. Browser checks at 1440px and 390px passed with keyboard and CSRF enabled. Included in signed v1.113.29, deployed at Helm revision 178; source and release checks, migration/rollout, health, public API and Helm smoke acceptance passed.
B-424 fixes the API omission confirmed for INC0010559: its stored CI link now appears in the authenticated ticket response, with a usable rack destination. The versioned detail contract exposes authorized Web UI information: record fields, CIs/services, governance and reviews, SLAs, approvals, tasks, related records, status updates and event history. Ticket lists stay compact; create/update responses retain details. Tenant, role, scope, related-record visibility and CI-class checks remain enforced. No schema migration.
The iOS ticket view adds expandable sections, exact rack navigation, visible retry feedback and server-authorized edit/state controls. Legacy responses still decode. 31 backend regressions, the simulator build and decoding checks passed. A synthetic SwiftUI component capture was visually checked; physical-device and VoiceOver acceptance remain pending.
Earlier MicroK8s revision 176 was the local acceptance deployment, built from governed v1.113.25 plus this isolated API change. A fresh dump restored successfully (131 tables, 321 tickets); scratch DB and verification credentials were removed. Public authenticated ticket/details/rack calls return 200, anonymous API access returns 401, website Access returns 302, health/readiness and retained Helm test pass, and all three workloads are ready with zero restarts. Schema and PVC identities are preserved. Reviewed backend source published in commit b869d69; local regression checks passed (406 tests, plus 13 detail/error tests after the review adjustment). This temporary candidate was superseded by governed releases; current runtime is verified v1.113.29 at revision 178. The results below retain the earlier acceptance history. Evidence. Client/API guide.
Professional web themes and the previously accepted mobile rack API are published in v1.113.25. Signed source/release commits and tag are GitHub Verified. Governed QA passed: 1,674 application tests, 100 browser checks and two administration regressions; all five RPM build/install targets and their stable checksum assets passed. Independent image provenance verification succeeded.
MicroK8s Helm revision 175 uses the exact release chart and GHCR digest, superseding revisions 171–174. Web/outbox/AI are ready with zero restarts, health/readiness return 200, migration head is unchanged, Helm smoke test passed and public Access returns 302. Backup restore and original PVC identities were verified. Live server rendering confirmed Navy, Slate and Forest. Authenticated public-browser theme acceptance was not repeated. Release/deployment evidence. Earlier acceptance images and releases below remain historical evidence.
B-421 adds Executive Navy, Slate and Forest alongside Classic Light, Dark and Match system. Accessible radio cards reuse account preference persistence; no schema migration. Palette changes retain semantic status colors and existing accessibility settings. Eight targeted functional/browser regressions passed at desktop and 390px widths; an additional five-test run verifies contrast, keyboard selection and persistence. The acceptance image adds only theme files to the previously accepted rack image, excluding unrelated working-tree edits. MicroK8s revision 172 is accepted: fresh restore-tested backup, unchanged schema head, ready web/workers, passing Helm test, HTTP 200 health/readiness, public Access 302 and preserved PVCs. Evidence. Earlier deployments below remain historical.
B-420 adds front/rear rack elevations from recorded capacity and placements, selected-server highlighting, device details, and explicit warnings for missing or overlapping positions. The read-only rack API enforces tenant and CI-class permissions; unmarked space is not presented as proof of an empty rack. No schema change.
Verification: 26 backend tests, the iOS build, placement checks and a SwiftUI component render passed. The live authenticated rack route returned four readable devices; unauthenticated access returned 401. MicroK8s revision 171 was accepted, with a restore-tested backup, passing Helm test, healthy web/workers and preserved storage. Physical-device navigation/accessibility acceptance remains pending. This is a local acceptance image, not a new governed release. Evidence. Component render with test placements.
B-419 adds grouped navigation, searchable asset details and ServiceOps connection information. The mobile CMDB response uses existing fields, preserves tenant/class read policies, excludes arbitrary attributes and supports older iOS clients. No migration change. The iOS build and legacy/rich-response decoding checks passed; 25 backend regressions passed. The local acceptance image is deployed at Helm revision 170, with a restore-tested backup, healthy web/workers and passing Helm/Access checks. This supersedes revision 168/169 as current runtime state; no new governed release or GitHub publication is claimed. Final visual/accessibility and physical-device acceptance remain pending. Evidence.
ServiceOps v1.113.21 is deployed to MicroK8s in namespace operations, Helm revision 168, using the chart from the same release and a provenance-verified image pinned by digest. Web, outbox worker and AI worker are ready with zero restarts. Health/readiness returned HTTP 200; migrations and the retained Helm test passed; public Cloudflare Access returned HTTP 302. PostgreSQL, uploads and backup PVCs were preserved.
The pre-upgrade dump was restored into a disposable database: 131 tables, 21 tickets and migration head 20261009_0117 verified; the scratch database was removed. Existing configuration warning: API_TOKEN_PEPPER falls back to SECRET_KEY. Authenticated production browser/SSO journeys were not rerun. Deployment evidence. Earlier release and deployment results below remain historical evidence.
Started 2026-10-09; finalized 2026-10-10 (Asia/Tokyo). Scope: B-416–B-418, authorized review corrections and administration UI improvements. No schema change. GitHub publication was authorized on 2026-10-10 and completed as v1.113.13; live deployment was explicitly authorized and verified on 2026-10-10.
| Step | Dependencies | Outcome | Status |
|---|---|---|---|
| 1. Reproduce | Current source and disposable database | Three isolated failing regression cases; review evidence | Complete |
| 2. Correct contract behavior | Step 1 | Retain selected references; stream/filter contracts with bounded rendered results; keep recipient-less notices retryable | Implemented |
| 3. Improve authority administration | Existing tenant/role/audit policies | Assigned users only; searchable Add user dropdowns and per-user removal; independent executive quorum changes | Implemented |
| 4. Preserve save context | Explicit validated section and group panel | POST/redirect/GET to the same page, open panel, visible accessible success message; no dependence on Referer | Implemented |
| 5. Verify | Steps 2–4 | Regression/security/full-suite checks, desktop/mobile browser and accessibility evidence | Complete |
| 6a. Publish governed release | Step 5, signed app and companion documentation commits | GitHub quality/security/browser gates, signed GHCR provenance and SBOM, RPM assets, stable release | Complete: v1.113.13 |
| 6b. Deliver locally | Step 5, fresh restore-tested backup, immutable image, idle Helm release | Helm --wait without --atomic; retained PVCs; migrations/readiness/workers/public access/log checks | Complete: Helm revision 167 |
| 7. Close and clean | Verified results | Current backlog status and evidence; remove confirmed temporary artifacts; preserve credentials, data, backups and historical requirements | Complete |
Permission assignment screens show assigned people, with searchable dropdowns to add eligible users and explicit removal controls. Do not enumerate every user with a Grant/Reject toggle. Keep role-level permission matrices where they express role policy rather than user assignment. Incremental changes preserve other assignments and approval rules. Save actions retain the originating page and open editor/group context and show a clear success confirmation; errors remain visible. Validate return targets on the server.
Local focused validation: 64 passed, 2 environment-gated skips. Localization and regression gate: 53 passed. Real Chromium at 1440×1000 and 390×844: 2 passed with CSRF enabled, keyboard picker selection, assignment/removal, same-page group save, visible confirmation in the viewport, no page errors, no horizontal overflow, and no axe WCAG A/AA violations on CCB/executive pages. Production-image Python 3.14 smoke: login and CCB/executive/contracts routes passed against disposable SQLite with external networking disabled. Ruff, JavaScript syntax, generated localization inventory, Helm lint and document anchor checks pass.
The first complete local Python 3.12 run recorded 1,640 passes, 135 skips and one stale generated localization source failure; source.json was regenerated and its focused gate passed. The first Python 3.14 container run recorded 1,638 passes, 137 skips and one stale generated dark-theme failure after the confirmation styles changed; dark.css was regenerated. The final clean Python 3.14 container suite passed: 1,640 passed, 137 environment-gated skips, 15 warnings, with two workers and external networking disabled. The 100-executive cap regression is included. These intermediate runs are retained as historical diagnostic evidence and are not claimed as fully passing.
Pre-upgrade backup: serviceops-backup-review-20261009:/backups/serviceops-20261009T131054Z.dump. Restore into disposable serviceops_review_restore_20261009 succeeded: Alembic 20261009_0117, 131 public tables, 21 tickets, 8 users; the test database was dropped. PostgreSQL, uploads and backups PVC UIDs were recorded before upgrade.
Production user data is excluded from mutation tests. External-provider, authenticated public-browser/SSO and other environment-gated tests remain separate verification gaps.
Current outstanding work retains unresolved requirements and verification gaps. Historical engineering and AI plans remain reference evidence; previous dates, release versions and estimates do not represent current deployment state.
Published v1.113.13, release commit 984653a3bb03a383da96015640c8a379794e4eb4. GitHub verified the application, documentation and workflow-generated release commit signatures. Quality gate, CodeQL and governed release completed successfully. Release validation: 1,640 passed, 137 environment-gated skips; browser journeys 100 passed, 6 skips; new desktop/mobile administration tests 2 passed. All five RPM targets passed build/install checks; downloaded RPM SHA-256 sidecars and build attestations were independently verified. GHCR image signing, SLSA provenance and CycloneDX SBOM gates passed; image attestation was independently verified. See structured release evidence and downloaded checksums.
The initial hosted quality attempt and rerun failed before testing because Docker Hub rate-limited Python pulls and PostgreSQL registry authentication timed out. CI now uses Google's public registry mirror with unchanged pinned digests; the mirror returned the exact Python and PostgreSQL manifests. The first notes CI run found logs excluded by the generic ignore rule; the referenced final passing logs were explicitly committed and subsequent checks passed.
The hosted Kubernetes deployment workflow was skipped. The user subsequently authorized direct local MicroK8s deployment, verified below. Older GitHub releases/tags remain unchanged in this deployment-only request.
The user explicitly authorized deployment on 2026-10-10, resolving the earlier automatic approval rejection. ServiceOps v1.113.13 is deployed in namespace operations, Helm revision 167, using the chart extracted from immutable tag v1.113.13 and image ghcr.io/awijesundara/serviceops-server@sha256:40c8c8f4445bfcbc8b4e6c9919d4bd90f85061b58b5933e44e8aeecaaebfe570. Helm upgrade used --reuse-values and --wait, without --atomic. Environment values were updated locally; no new GitHub publication was requested in this deployment turn.
Fresh backup: serviceops-backup-pre-11313-20261010:/backups/serviceops-20261009T225640Z.dump. The isolated restore succeeded with migration 20261009_0117, 131 tables, 21 tickets and 8 users; its scratch database was dropped. Initial rehearsal connection-format failures were corrected before upgrade. Migration hook completed; web, outbox worker and chart-enabled AI worker each run one ready replica, with the exact verified image and zero restarts. Full logs from all new pods and their init containers had zero error/critical/traceback/HTTP 5xx matches. /health and /ready returned 200; retained Helm test succeeded, including database, migrations, audit encryption, uploads and worker checks. Public Cloudflare Access returned 302. Live ticket/user counts remain 21/8.
PostgreSQL, uploads and backup PVC identities are unchanged and Bound. The previous local candidate was never deployed. Authenticated public-browser/SSO journeys were not rerun against production; the release's hosted desktop/mobile and administration accessibility gates passed. See structured deployment evidence.
Cleanup: duplicate active backlog rows consolidated into 234 unique historical/current IDs; previous requirements and stable anchor destinations retained in the single frozen history file. Disposable restore Jobs and QA image removed; source caches and temporary build/export scripts cleared. The verified backup, production data, credentials and final acceptance image remain available. Administration UI convention was saved in the user-requested memory update note.
Final raw evidence: Python 3.14 suite, desktop/mobile browser, focused regressions, localization, immutable image. Live /health and /ready are HTTP 200 and public Cloudflare Access returns HTTP 302 for the deployed v1.113.13 release.