Config TOML and workflow UX improvements#31
Merged
Conversation
Config: - Unify backend + applicative config into config.toml (KESTREL_CONFIG_FILE); overlay watched_repos, trigger_label, reconcile_interval_seconds, verify_checks, max_verify_iterations (file wins, env fallback). Secrets stay in the environment. KESTREL_BACKENDS_FILE kept as a deprecated alias. - Rename backends.toml.example -> config.toml.example; update docs, compose, .env.example, .gitignore. Workflow steps: - Introduce a StrEnum Step (refine/design/code/verify) as the single typed source of truth; wire it through the service and router. UI (WorkflowPanel): - Pulse the actively-running step chip (theme-token keyframe, reduced-motion fallback). - Render the code deliverable in a real diff viewer (diff2html) via a derived deliverable_format, instead of markdown. - Spin an activity indicator on active runs in the sidebar. - Show a depleting progress ring on the verify chip for remaining code<->verify iterations (new verify_round field + column + migration 0009, verify_max_iterations exposed in the detail payload). Tests: config overlay/alias/env-fallback; frontend diff/pulse/verify/sidebar. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reorganise backend/.env.example to promote the TOML config file: lead with KESTREL_CONFIG_FILE, keep secrets (github/webhook/jira/code-host tokens) as the primary env entries, and comment out the applicative settings that now live in config.toml (watched_repos, trigger_label, reconcile_interval_seconds, verify_checks, max_verify_iterations) so they read as env fallbacks rather than the recommended path. Group the remaining env-only settings separately. Document the split in docs/configuration.md: config.toml is the preferred home for non-secret config; .env holds secrets and not-yet-migrated settings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mechanically enforce structural limits across Python and JS/TS/Vue to fight module bloat, parameter/branch creep, and copy-paste: - ruff: +C90/PL/B/SIM/ARG/RET with mccabe + pylint thresholds - pylint: module length <= 500 - vulture + knip: dead code - import-linter + dependency-cruiser: layering + no import cycles - eslint (flat config) + sonarjs: file/function size + complexity - jscpd: copy-paste <= 3% Single entrypoint `task quality` (tools/quality.sh); per-file PostToolUse agent hook (tools/agent-check.sh); pre-commit hooks; and a [quality-override] commit-trailer guard on the threshold config (tools/check-config-guard.sh). Existing violations are grandfathered with TODO(quality) markers; new files get no exemptions. CI job wiring is in a follow-up commit (needs a workflow-scoped push). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the guardrail suite (tools/quality.sh) and the threshold-config guard (tools/check-config-guard.sh) into the reusable testing workflow as separate jobs. Not yet pushed: requires a token with the `workflow` scope. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… Settings Change set A: two urgent fixes for the broken Jira deployment: 1. Migrate JiraClient.search() from deprecated GET /search (removed by Jira Cloud) to POST /rest/api/2/search/jql with token-based pagination (nextPageToken → loops until isLast). Fixes the 410 Gone error blocking all Jira polling. Also fixes the latent >50-issue truncation, so the dismissal-clear logic stays correct. 2. Move KESTREL_HOST/PORT/RELOAD into Settings so .env values are honoured (they were read via os.environ.get, bypassing pydantic-settings). Now all three are real Settings fields, consistent with the FastAPI kit rule. Both changes are non-breaking to ingestion behaviour (SC-006 in feature 004); existing 002/003 tests pass with zero observable change. Verified against pinned ruff@0.15.20 + pylint@4.0.6 (per-file). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…config) Feature 004: Extract the per-source ingestion config from the 002/003 scalars into one uniform [[task_sources]] list; unify poll cadences; broaden Jira repo resolution to web/remote links; add a \`python -m app poll\` dry-run CLI. Includes spec.md (4 user stories, 17 FRs + SC-007a for dismissal scoping), plan.md (9 design decisions, Constitution Check), research.md (D1-D9), data-model.md (TaskSourceConfig with per-source code-host config + key for dismissal-scoping), 3 contracts (config-schema, cli-poll, jira-remote-link), quickstart.md (4 runnable scenarios), and tasks.md (37 tasks across 7 phases). Remediated per /speckit-analyze: F1 (dismissal-scoping via per-source 'key'), E1 (poll_interval_seconds into _CONFIG_FILE_FIELDS), U1 (code-host from entry), C1/G1/D1 (minor cleanups). This is 004-task-source-config on the master branch. Feature is ready for /speckit.implement; recommended: Set A (code) → Set B Phase 1 → Set B Phase 2 (foundational) → Set B Phase 3 (US1 MVP) → stop and validate before US2-US4. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Feature 004-task-source-config. Replaces the per-source scalar ingestion config (002/003) with one file-only [[task_sources]] list, unifies the poll cadence, broadens Jira repo resolution, and adds a dry-run poll command. Breaking config change, no back-compat (accepted; single-user tool). - config: new TaskSourceConfig model (app/config_models.py, alongside the extracted BackendConfig); Settings gains task_sources (file-only) + poll_interval_seconds; removes watched_repos/trigger_label/jira_*/code_host*/ the two interval keys; adds github_sources/jira_sources/github_source_for helpers; warning validators retargeted per source. Tokens stay in the environment via each entry's token_env (mirrors BackendConfig.secret()). - consumers rewired to the list: ingestion.is_watched, github_webhook router, reconcile (one service per github source), jira_poll (one per jira source, whole JQL, key-scoped dismissals), workflow source/code-host registry, and the lifespan (one loop per configured source). - poll CLI: `python -m app poll` lists what each source matches with no side effects (app/cli.py + app/services/poll_source.py; list_work_items reuses the live query). __main__.py is now a thin shim. - Jira repo resolution: JiraClient.get_remote_links + _repo_from_url; _resolve_repo split into helpers, field-first then a title-matched web link; field optional. - WorkItem lives in app/ports.py to avoid a poll_source<->services import cycle. Tests: 451 pass; task quality clean (jscpd 2.15% py). Existing 002/003 behaviour tests updated to the new shape and still green (SC-006). New tests cover config parsing/validation/token resolution, web-link URL parsing + fallback, the non-ingesting listings, and the CLI. Verified against pinned ruff@0.15.20 + pylint@4.0.6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update config.toml.example, backend/.env.example, and the configuration / GitHub-workflow / Jira-workflow docs for feature 004: the [[task_sources]] list, unified poll_interval_seconds, token_env secret handling, web-link repository resolution, and the `python -m app poll` dry-run. Removes the migrated scalar keys (watched_repos, trigger_label, jira_*, code_host*, the two interval keys). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Check off tasks.md for feature 004 and record that WorkItem was implemented in app/ports.py (not poll_source.py) to avoid an import cycle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pydantic-settings reads backend/.env into Settings fields, but the dynamic named-secret lookups — a task source's token_env, a backend's api_key_env, and the standard OTEL_* vars — resolve against os.environ, which .env does not otherwise populate. A secret placed only in .env was therefore invisible to token()/secret(), so ingestion silently had no token. Load .env into the process environment via python-dotenv (now a direct dependency) in app/__main__.py — the real `python -m app` entrypoint, not library code, so tests stay isolated. Existing environment values still win (override=False), matching pydantic's env>dotenv precedence. Adds a test that reproduces the bug (a .env-only token resolving through token_env only after load_env) and documents the behaviour. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jira only allows http(s) URLs in the web-link field, and a link can be a deep link rather than the bare repo URL. Make _repo_from_url host-aware and stricter: reject non-http(s) schemes; for github keep the first two path segments (so .../owner/name/issues/5 resolves to owner/name); for gitlab/gitea keep the subgroup path and truncate a /-/ deep-link tail; tolerate a trailing .git or slash. The clone still uses the configured code_host_base_url as the host — the link only supplies the project path (no schema change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Self-hosted Jira/GitLab often present an internal or self-signed CA that the process does not trust, and httpx uses its own bundled CA set (not the OS trust store), so the poll/resolve/notify REST calls fail TLS verification. Add verify_ssl (default true) to TaskSourceConfig. When false, the source's httpx REST clients are built with verify=False: the Jira client, the code-host client (GitLab/Gitea, and the GitHub client used for the source), across the poll factory, the reconcile factory, and the workflow run-time registry. A test asserts verify=False actually reaches the httpx SSL context (CERT_NONE). Scope: this covers REST/API calls only. It does NOT change git clone/fetch/push, which use the system trust store — document installing the internal CA there for git. Example config and docs updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…eadless) A Jira→GitLab run failed at clone: git launched a GitLab OAuth browser flow and hung on an interactive prompt the headless backend can never answer. Two root causes: 1. GitService._auth always used the service's default token (GitHub's); the per-run code-host token override existed but was never wired. So a GitLab clone was sent the GitHub PAT, GitLab 401'd, and git fell into its OAuth credential helper. 2. Nothing disabled interactive prompting, so git blocked on the prompt. Fixes: - The CodeHost now supplies its git credential — x-access-token:<token> for GitHub, oauth2:<token> for GitLab — threaded into clone/fetch/push via the run's code host (replacing the never-wired default-token path). provision_ worktree is inlined at its two callers as ensure_mirror(cred) + add_worktree to keep methods within the argument limit. - _git now runs with GIT_TERMINAL_PROMPT=0 and prepends `-c credential.helper=` so no inherited helper (OAuth/browser) runs and git fails fast on a 401. - _git also redacts the injected auth header in the INFO log (it previously logged the base64 credential — a token leak). Docs note the code-host PAT scopes (GitLab read_repository + write_repository). 461 tests pass; task quality clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A Jira-sourced run has no numeric issue id (it uses task_ref), so persisting its gate/terminal notification failed with "NOT NULL constraint failed: notification.issue_number" — the notification bell never showed the awaiting- input gate. The notification table was created NOT NULL in feature 002 (GitHub-only) and was never relaxed when Jira (003) made workflow_run.issue_number nullable. Migration 0010 makes notification.issue_number nullable; the ORM row, the store signature, the Notification dataclass, the NotificationOut schema, and the frontend Notification type all follow (identify the run via workflow_id). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The sidebar list was fetched only on mount/create, with live updates that only
synced existing items' status — so a run started by background ingestion
(GitHub webhook / Jira poll) never appeared until a manual page reload.
Add a list-level channel to WorkflowBus that every publish() ticks (creation
included), a GET /api/workflows/events SSE endpoint that re-serialises the
summary list on each tick, and a frontend subscription (startList/stopList) so
the sidebar live-adds background-created runs. Declared before /{workflow_id}
so the static path isn't captured as an id. Mirrors the notification bus
pattern (push, don't poll).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both /api/workflows/events and /api/notifications/events were failing with ERR_EMPTY_RESPONSE in the browser once a real Jira run started cloning/editing files. Root cause: `task backend` runs with KESTREL_RELOAD=1, and the run workspace defaults to `./.kestrel-workspaces` — inside the same directory uvicorn's --reload watches by default. A run writes real .py files there (git clone, agent edits mid-turn), which uvicorn's watcher picks up and restarts the *entire server process* on — killing every open SSE connection at once, which is exactly why both streams broke simultaneously and only once a workflow was actually running. Reproduced directly: writing a .py file under backend/.kestrel-workspaces/ while curl held an SSE connection open triggered "WatchFiles detected changes ... Reloading..." and the connection was cut. Fix: cmd_serve now creates the workspace directory up front (so watchfiles treats it as a directory-prefix exclude rather than a non-matching glob) and passes its absolute path via reload_excludes. Verified the same repro no longer restarts the server and the SSE connection survives. The packaged Docker image was never affected: KESTREL_RELOAD is unset there and KESTREL_WORKSPACE_ROOT=/workspaces already sits outside /app. Also fixed docs/development.md, which recommended invoking `uvicorn --reload` directly — that bypasses cli.cmd_serve and would reintroduce the bug; steered it to `python -m app` instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both the workflow sidebar and the notification bell were populated only by their SSE stream connecting successfully — the workflow list lost its refresh() fallback in the live-list commit, and NotificationCenter was already SSE-only before that (pre-existing). A stream that fails to connect (ERR_EMPTY_RESPONSE reported in a real browser, though not reproducible via curl against the same backend/data) then shows nothing at all instead of a static, correct list. Call refresh() (plain GET, already proven reliable) alongside start()/ startList() on mount in both components. The SSE stream still upgrades to live updates when it connects; if it doesn't, the UI shows the last-known- good list instead of blank. Defensive regardless of the stream's root cause, which needs further live-browser diagnostics to pin down. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A frontend/backend port mismatch (KESTREL_PORT set in the backend env but not mirrored to the frontend's VITE_API_BASE) silently blanked the whole UI — every fetch failed at the network level with no visible signal beyond a console error, costing real debugging time to trace. Add a connectivity seam to the shared api request() function (mirrors the existing setUnauthorizedHandler pattern): fires false when fetch() itself throws (network-level failure, not just a non-2xx response) and true whenever a request resolves at all. A new useConnectivity() composable wires this to a reachable ref and polls GET /livez every 5s while unreachable so the signal self-heals once the backend comes back, with no manual reload needed. App.vue renders a persistent, non-dismissible banner naming the configured API base when unreachable — registered before any child component's onMounted fetch runs, so it's live for the very first request. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The config.toml.example was using outdated step names (plan, implement) that no longer exist in the backend implementation, which uses the canonical steps: refine, design, code, verify. This mismatch caused silent fallbacks to default backends when operators copied the example. Changes: - Update config.toml.example to use correct step names: refine, design, code, verify - Add validation to Settings that fails fast with a clear error message if invalid step names are configured, preventing silent fallbacks - Update docs/backends.md to reflect current step names - Update backend/app/policy.py docstring examples - Add comprehensive test coverage for step_backends validation, ensuring valid steps are accepted and invalid ones are rejected with helpful error messages The validator runs after config file load and supports partial configurations and sub-step names like "refine.reconcile". Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Fix _validate_step_backends: a dotted sub-step key (e.g.
"refine.reconcile") was wrongly rejected even when its top-level step
("refine") was valid, because main_invalid didn't check the prefix
against valid_steps before flagging it. Also fixes the test's own TOML
fixture, which set "refine" as a scalar then re-opened it as a table via
bare-dotted-key syntax — invalid TOML; the intended literal dotted key
needs quoting ("refine.reconcile" = ...).
- Rename jira_srcs -> jira_sources (cspell flagged the abbreviation).
- prettier --write the 4 files format:check had drifted on.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
setup-jira-workflow.md:48 was 81 columns; rewrap the paragraph. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges accumulated work from feat/config-toml-and-workflow-ux into master —
this branch was ready but hadn't been merged yet, which caused feat/005
(behavioral verify evidence) to be planned/implemented against a stale
master and now needs reconciling on top of this.
🤖 Generated with Claude Code