Skip to content

Add scenario runner seam for runtime summaries #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Pan14ek merged 1 commit into from
Jun 1, 2026
Merged

Add scenario runner seam for runtime summaries #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion backend/lined/.beads/issues.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,15 @@
{"_type":"issue","id":"lined-62z","title":"Add backend Docker image","description":"Add reproducible Docker image support and documented build/run flow for the Spring Boot backend as the experiment/backend-containerization task.","acceptance_criteria":"Dockerfile builds the backend image reproducibly; container run flow is documented; docs index routes to the containerization guide; backend behavior is unchanged.","status":"closed","priority":2,"issue_type":"task","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-23T17:35:06Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-23T17:41:10Z","started_at":"2026-05-23T17:35:17Z","closed_at":"2026-05-23T17:41:10Z","close_reason":"Implemented Dockerfile, .dockerignore, and containerization docs; Gradle check and JaCoCo passed. Docker daemon-backed build/run verification is tracked separately in lined-azh because the local Docker socket was unavailable.","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-51s","title":"Add scenario runner seam","description":"Architecture review bug bug/scenario-runner-seam: runtime evidence generation is split across procedural docs, kubectl ordering, k6 execution, telemetry collection, and hand-built summaries. Add a narrow runner seam that accepts scenario, workload, and output root, coordinates Kubernetes/k6/state adapters, and writes sanitized collector-ready runtime-summary artifacts.","acceptance_criteria":"A CLI under load-tests/runtime-scenarios accepts allowlisted scenario/workload inputs, coordinates kubectl and k6 through testable adapters, writes runtime-summary.json only for successful runs, writes sanitized manifests, includes unit tests, and documents the workflow.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-06-01T15:08:34Z","created_by":"Oleksii Makieiev","updated_at":"2026-06-01T15:08:49Z","started_at":"2026-06-01T15:08:49Z","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-0zb","title":"Deepen calendar time-window handling","description":"Architecture review bug: event flows pass raw OffsetDateTime pairs and repeat start/end validation across create, update, list, conflict, and user-conflict paths. Add one validated calendar time-window path and keep overlap/conflict rules local to the event scheduling module without changing public API shape.","status":"closed","priority":2,"issue_type":"bug","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-06-01T14:06:34Z","created_by":"Oleksii Makieiev","updated_at":"2026-06-01T14:19:13Z","started_at":"2026-06-01T14:06:46Z","closed_at":"2026-06-01T14:19:13Z","close_reason":"Completed calendar time-window refactor with validated window seam, conflict analyzer, header-bound requester checks, critic review, and ./gradlew check.","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-z7y","title":"Fix account provisioning policy seam","description":"Architecture review bug from docs/experiment-tasks.md: AccountApplicationService exposes provisioning booleans and hard-coded default role/free-plan policy, while role resolution knowledge is duplicated between user and role modules. Implement one clear registration provisioning path, centralize default role/plan policy, and share role resolution.","status":"closed","priority":2,"issue_type":"bug","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-30T16:16:47Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-30T16:21:50Z","started_at":"2026-05-30T16:16:55Z","closed_at":"2026-05-30T16:21:50Z","close_reason":"Implemented account provisioning policy seam and shared role resolver; focused tests and ./gradlew check pass.","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-9pq","title":"Define SLO constraint thresholds","description":"Define initial local experiment SLO and constraint thresholds for classifying runtime-summary evidence for experiment/slo-constraint-thresholds. Keep backend behavior and scoring unchanged; add docs/config only.","status":"closed","priority":2,"issue_type":"task","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-30T05:18:03Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-30T05:24:19Z","started_at":"2026-05-30T05:18:18Z","closed_at":"2026-05-30T05:24:19Z","close_reason":"Completed SLO threshold docs and versioned config","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-vbf","title":"Add runtime scenario summary workflow","description":"Implement experiment/runtime-scenario-summaries: add a repeatable local workflow for running kind deployment scenarios under k6 and producing sanitized runtime-summary.json artifacts for collector ingestion.","status":"closed","priority":2,"issue_type":"task","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-29T08:25:25Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-29T08:40:48Z","started_at":"2026-05-29T08:25:33Z","closed_at":"2026-05-29T08:40:48Z","close_reason":"Implemented runtime scenario summary docs and CLI with critic review, verification, and Notion write-back.","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-8jq","title":"Add Prometheus telemetry pipeline","description":"Implement experiment/prometheus-telemetry-pipeline from docs/experiment-tasks.md by adding local kind Prometheus manifests and documentation for collecting Actuator runtime metrics.","status":"closed","priority":2,"issue_type":"task","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-28T20:30:07Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-28T20:36:12Z","started_at":"2026-05-28T20:30:16Z","closed_at":"2026-05-28T20:36:12Z","close_reason":"Implemented Prometheus telemetry pipeline manifests and documentation","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-hgb","title":"Add runtime fitness extension","description":"Implement experiment/fitness-runtime-extension by documenting runtime-aware fitness inputs and adding optional summarized runtime metrics ingestion without changing the existing structural fitnessScore.","acceptance_criteria":"Runtime fitness design is documented and linked; collector optionally reads summarized runtime metrics without failing when absent; existing fitnessScore semantics and analyzer compatibility are preserved; Notion research pages are updated and verified.","status":"closed","priority":2,"issue_type":"task","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-28T06:21:08Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-28T06:31:02Z","started_at":"2026-05-28T06:21:22Z","closed_at":"2026-05-28T06:31:02Z","close_reason":"Implemented runtime fitness design docs, optional collector runtime summary ingestion, critic review loop, verification, and Notion write-back.","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-byj","title":"Add load-test baseline","description":"Implement experiment/load-test-baseline by adding a repeatable k6 workload for users, lobbies, tasks, and events against the local kind backend baseline.","acceptance_criteria":"k6 baseline script creates synthetic data and exercises valid workflows; documentation explains kind port-forward, k6/Docker commands, synthetic data, and Prometheus metrics verification; no Spring business behavior changes.","status":"closed","priority":2,"issue_type":"task","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-25T19:40:17Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-26T04:33:19Z","started_at":"2026-05-25T19:40:27Z","closed_at":"2026-05-26T04:33:19Z","close_reason":"Completed: added bounded k6 load-test baseline script, documented smoke/baseline workflows, Docker fallback, synthetic data behavior, and runtime metrics verification; Gradle test/check and static script checks passed.","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-cy8","title":"Runtime metrics baseline","description":"Implement experiment/runtime-metrics-baseline from docs/experiment-tasks.md: expose Prometheus-compatible backend metrics and document key runtime signals so /actuator/prometheus can be collected for latency, error, and resource analysis.","status":"closed","priority":2,"issue_type":"task","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-25T19:11:10Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-25T19:22:23Z","started_at":"2026-05-25T19:11:18Z","closed_at":"2026-05-25T19:22:23Z","close_reason":"Implemented runtime metrics baseline with Prometheus scrape metadata, documentation, and local verification.","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-stn","title":"Add backend Kubernetes health probes","description":"Implement experiment/backend-health-probes by configuring the kind backend Deployment with Spring Boot Actuator readiness and liveness probes, documenting verification steps, and preserving backend business behavior.","acceptance_criteria":"Backend Deployment has readinessProbe and livenessProbe using Actuator endpoints; readiness and liveness health groups are explicit; kind baseline docs explain probe verification; Gradle quality gates and kind rollout verification pass.","status":"closed","priority":2,"issue_type":"task","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-25T18:08:23Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-25T18:14:45Z","started_at":"2026-05-25T18:08:35Z","closed_at":"2026-05-25T18:14:45Z","close_reason":"Completed: configured Actuator readiness and liveness health groups, added Kubernetes probes to the kind backend Deployment, documented verification steps, and verified Gradle test/check/JaCoCo plus kind rollout and Actuator smoke tests.","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-wq2","title":"Deploy backend baseline to kind","description":"Implement experiment/kind-postgres-backend-baseline by adding local kind Kubernetes manifests for PostgreSQL and the Spring Boot backend, plus documentation for building, loading, applying, port-forwarding, and verifying the Actuator health endpoint.","acceptance_criteria":"PostgreSQL and backend Kubernetes manifests exist under k8s/kind; docs explain the reproducible kind workflow; backend health can be verified at /actuator/health after port-forwarding; Java business behavior is unchanged.","status":"closed","priority":2,"issue_type":"task","assignee":"Oleksii Makieiev","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-23T17:57:50Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-23T18:03:39Z","started_at":"2026-05-23T17:57:59Z","closed_at":"2026-05-23T18:03:39Z","close_reason":"Completed: added k8s/kind manifests, documented the kind baseline workflow, and verified Docker build, kind load, rollout status, and /actuator/health smoke test.","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-62z","title":"Add backend Docker image","description":"Add reproducible Docker image support and documented build/run flow for the Spring Boot backend as the experiment/backend-containerization task.","acceptance_criteria":"Dockerfile builds the backend image reproducibly; container run flow is documented; docs index routes to the containerization guide; backend behavior is unchanged.","status":"open","priority":2,"issue_type":"task","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-23T17:35:06Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-23T17:35:06Z","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-15y","title":"Remove redundant calendar requesterId query parameter","description":"PR #43 review follow-up: conflict endpoints now bind requester identity to X-User-Id, making the requesterId query parameter redundant. Deprecate and remove requesterId from /api/calendar/conflicts and /api/calendar/user-conflict in a compatibility-focused cycle.","status":"open","priority":3,"issue_type":"task","owner":"alexmakeev2703@gmail.com","created_at":"2026-06-01T14:40:15Z","created_by":"Oleksii Makieiev","updated_at":"2026-06-01T14:40:15Z","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-oss","title":"Document Notion knowledge-base workflow","description":"Add backend documentation for using Notion as the durable knowledge base, including write-back checklist, verification after write, fallback policy, and entry template.","status":"closed","priority":3,"issue_type":"task","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-26T20:53:14Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-26T20:53:24Z","closed_at":"2026-05-26T20:53:24Z","close_reason":"Completed: added Notion knowledge-base workflow doc, linked it from backend docs index and AGENTS routing, and verified the documentation diff.","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"lined-azh","title":"Verify backend Docker image with running daemon","description":"Docker build and container smoke checks for experiment/backend-containerization could not run in this session because Docker daemon socket /Users/oleksii_makieiev/.docker/run/docker.sock was missing. Start Docker Desktop or another daemon, run docker build -t lined-backend:local ., then run the image against PostgreSQL and verify /actuator/health and /swagger-ui.html.","acceptance_criteria":"docker build succeeds; container starts against PostgreSQL; actuator health and Swagger UI are reachable from localhost.","status":"open","priority":3,"issue_type":"task","owner":"alexmakeev2703@gmail.com","created_at":"2026-05-23T17:39:18Z","created_by":"Oleksii Makieiev","updated_at":"2026-05-23T17:39:18Z","dependency_count":0,"dependent_count":0,"comment_count":0}
1 change: 1 addition & 0 deletions backend/lined/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ changes.
| Runtime Metrics | Prometheus-compatible Actuator metrics baseline and runtime signal map. | `runtime-metrics-baseline.md` | Use before collecting backend runtime metrics or designing runtime-aware fitness inputs. | `/actuator/prometheus`, latency/error/resource signals, Prometheus scrape metadata. |
| Load-Test Baseline | Repeatable k6 workload for users, lobbies, tasks, and calendar event flows. | `load-test-baseline.md` | Use before running baseline workload traffic against the local kind backend. | k6 smoke run, local workload profile, synthetic data behavior, runtime metrics checks. |
| HPA Resource Scenarios | Local kind variants for backend resource requests/limits, fixed replicas, and CPU-based HPA behavior. | `hpa-resource-scenarios.md` | Use before comparing deployment/runtime trade-offs under k6 workload traffic. | Resource pressure, fixed replica comparison, HPA prerequisites, scenario cleanup. |
| Runtime Scenario Summaries | Scenario-runner seam for producing sanitized runtime-summary artifacts from one scenario and workload. | `runtime-scenario-summaries.md` | Use before generating collector-ready runtime evidence for local scenario comparisons. | Scenario runner CLI, k6 summary export, Kubernetes state summaries, provenance manifest. |
| Runtime Fitness Extension | Runtime-aware fitness metric contract and optional collector input shape. | `runtime-fitness-extension.md` | Use before adding runtime-aware scoring or attaching runtime summaries to metrics documents. | Structural/runtime score separation, runtime metric schema, normalization, compatibility. |
| Prometheus Telemetry Pipeline | Local Prometheus deployment and scrape workflow for kind backend metrics. | `prometheus-telemetry-pipeline.md` | Use before collecting persistent-enough Prometheus samples from local scenario runs. | Prometheus pod discovery, Actuator scrape verification, PromQL checks, telemetry cleanup. |
| SLO Constraint Thresholds | Initial runtime SLO and constraint thresholds for classifying local experiment variants. | `slo-constraint-thresholds.md` | Use before interpreting runtime-summary evidence or adding runtime-aware scoring. | Latency, error-rate, availability, restart, readiness, and resource-pressure constraints. |
Expand Down
14 changes: 10 additions & 4 deletions backend/lined/docs/runtime-fitness-extension.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,11 @@ Store summarized values only. Do not store raw Prometheus exposition text,
full pod YAML, secrets, environment variables, or generated load-test output in
Cosmos DB.

Use the scenario runner in `docs/runtime-scenario-summaries.md` to produce
collector-ready local runtime summaries. The runner owns local Kubernetes/k6
orchestration and sanitized artifact generation; the collector still only
validates and attaches an explicit summary JSON file.

## Metric Contract

| Field | Unit/range | Better direction | Source |
Expand Down Expand Up @@ -170,10 +175,11 @@ deploy kind, call Kubernetes, or scrape Actuator endpoints by itself.

Before using a runtime summary in an experiment:

1. Run a k6 smoke workload from `docs/load-test-baseline.md`.
2. Verify Actuator metrics from `docs/runtime-metrics-baseline.md`.
3. Record the active deployment scenario from `docs/hpa-resource-scenarios.md`.
4. Produce a summarized runtime JSON file.
1. Verify the active scenario and workload prerequisites from
`docs/runtime-scenario-summaries.md`.
2. Run the scenario runner for one scenario/workload pair.
3. Inspect the generated `runtime-summary-manifest.json`.
4. Use the generated `runtime-summary.json` as the explicit collector input.
5. Run the collector with `RUNTIME_METRICS_JSON` set.
6. Confirm the stored document preserves `fitnessScore` and adds
`metrics.runtime_metrics` separately.
Loading
Loading