This repository is a monorepo for the data-movement onboarding toolchain. It contains one primary product and the supporting pieces it builds on:
- DDA Agent (
dda-agent/) — the primary product. A FastAPI backend + React SPA that automates data-movement onboarding end to end: given a Jira key it parses and validates the requirements, creates child Jiras, registers records with the org system's APIs, resolves target repositories through pattern-defined slots, generates artifacts (Mongo-stored templates and a column-details → BigQuery DDL pipeline), commits them per repository in approved order (GitHub Enterprise via OAuth, github.com via PAT), triggers Jenkins / GitHub Actions builds, and reports the outcome back to Jira — all behind four human approval gates over a resumable, crash-safe session (a Mongo-checkpointed state machine with a side-effect intent journal). - CDM-Next Prework Excel form (
excel-form/+mock-server/,portal/,api/,tools/,docs/) — a self-contained, config-driven macro-enabled Excel intake form (.xlsm). The DDA Agent reuses this project's pure-Python parser/validator for Mode A intake, so the two ship together.
This is the single, authoritative README for the whole repository (it replaces the former
README.md and DDA_AGENT_README.md).
- Repository map & setup files
- Prerequisites
- Quick start
- DDA Agent
- CDM-Next Prework Excel form
- Supporting services
- Testing
- Glossary
d:\Project\excel is the single git root. The DDA Agent lives in dda-agent/ as a self-contained
subproject; the Excel-form product and its services are siblings. No package is hoisted to the root —
the root Makefile, a consolidated Python manifest (pyproject.toml +
requirements.txt), and this README are the entry point that ties them together.
| Path | Purpose |
|---|---|
| dda-agent/ | Primary product. FastAPI backend (backend/, package dda_agent), React SPA (frontend/), JSON config (backend/config/), ops script (scripts/), tests (tests/). |
| excel-form/ | CDM-Next Prework intake form: JSON authoring config (src/config/), VBA source (src/vba/), the pure-Python build/verify/parse pipeline (build/), standalone tests (test/). |
| mock-server/ | Zero-dependency Node backend implementing the Excel-form HTTP contract + serving the portal. |
| portal/ | Static download portal (lists published workbooks, SHA-256 integrity, download). |
| api/ | The Excel-form HTTP contract: contracts.md, openapi.yaml, and a copy-in Spring Boot reference (java/). |
| tools/ | Check-FormReadiness.ps1 — read-only customer-machine readiness check. |
| docs/ | Long-form docs. DDA Agent deployment: deployment.md, org-apis-onboarding.md, org-api-designs.md. Excel form: add-pattern.md, field-config.md, customer-enable-macros.md, storage-mongodb.md. |
| Makefile | Root task runner (setup / run / test / lint); see §3.1. |
| .github/ | GitHub Copilot config: copilot-instructions.md, instructions/, prompts/, agents/ (custom agents). |
CLAUDE.md + .claude/ |
Claude Code config: engineering conventions + design skills (the Copilot .github/ files mirror these). |
The setup is intentionally per-subproject; this table is the map.
| File | Location | Scope |
|---|---|---|
pyproject.toml / requirements.txt (root) |
pyproject.toml, requirements.txt, requirements-dev.txt | Consolidated manifest: unions all Python deps across both subprojects. Aggregator only — builds no package (see §3.2). |
pyproject.toml (DDA backend) |
dda-agent/backend/pyproject.toml | Source of truth for the installable dda_agent package + [dev] extras (dependency ranges). |
requirements.txt / requirements-dev.txt (DDA) |
dda-agent/requirements.txt, dda-agent/requirements-dev.txt | Pinned lockfiles for reproducible DDA installs. |
requirements.txt (Excel form) |
excel-form/requirements.txt | Excel-form build/test Python deps. |
.env.example |
dda-agent/backend/.env.example | Every DDA_* setting; copy to .env. |
start.ps1 / start.sh |
dda-agent/backend/start.ps1, dda-agent/backend/start.sh | Convenience backend launchers (venv + uvicorn). |
package.json |
dda-agent/frontend/package.json, mock-server/package.json | Frontend (Vite/React) and mock-server Node projects. |
vite.config.ts / tsconfig.json |
dda-agent/frontend/ | Frontend build + TS config (dev proxy /api → 127.0.0.1:8100). |
ruff.toml / pytest.ini / .flake8 |
dda-agent/ | Lint / test / cognitive-complexity config (rootdir dda-agent/). |
pyrightconfig.json |
pyrightconfig.json (repo root) | The enforced type-check gate (basic mode, 0 errors). |
| mypy config | [tool.mypy] in dda-agent/backend/pyproject.toml |
Advisory only. |
config/*.json |
dda-agent/backend/config/ | pattern-behaviors.json, org-apis.json, type-mappings.json, templates-seed.json (+ schemas). |
- Python ≥ 3.11 — the DDA backend (
requires-python = ">=3.11"). The Excel-form build needs only Python ≥ 3.9 (developed on 3.11). - Node.js ≥ 18 — the DDA frontend (Vite 5 / Vitest 2), the mock server, and the DDA validator
contract test (needs
nodeon PATH). - MongoDB reachable at
DDA_MONGO__URIfor the DDA backend inenv=dev/prod(env=testruns fully in memory). MongoDB ≥ 6.0 recommended: the partial-unique indexes using$infilters need it; on older servers index creation logs a warning and application-level checks take over. - Windows desktop Excel — only to run the CDM-Next Prework workbook (VBA). Building the workbook is pure Python and needs no Excel.
Service accounts / registrations for a production DDA deployment (all optional in dev/test,
which degrade gracefully until each stage is reached):
- Jira service account — username + password for Basic auth (
DDA_JIRA__SERVICE_ID/DDA_JIRA__SERVICE_PASSWORD), used for Jira reads: it must read the source issue and its attachments and resolve create-issue metadata for the anchor issue's own project (child Jiras are always created on the user's board — there is no configured target project). - Per-user Jira PAT — Jira writes go as the acting user: each user supplies a personal
access token (prompted at the first write stage,
AWAITING_JIRA_PAT, held in the in-memory broker for ~24 h) that creates/updates child issues and adds comments/labels/links plus the transitions inDDA_JIRA__TRANSITIONS. - Jenkins user + API token (
DDA_JENKINS__USER/DDA_JENKINS__API_TOKEN). The owner → node URL mapping collection ({_id: <owner>, jenkinsNode, _class}, name viaDDA_MONGO__JENKINS_OWNER_NODES_COLLECTION) is CI-team data, but an unmapped owner is no longer a dead end: when it is the only thing blocking Gate-2 approval, the user is asked for the node URL (restricted to the canonical controller's domain) and the mapping is written back. - GHE OAuth App — the GHE org admin registers an OAuth application whose Authorization
callback URL exactly equals
DDA_GHE__REDIRECT_URI(<backend origin>/api/v1/auth/ghe/callback); scopesrepo workfloware requested at authorize time; put the client id/secret inDDA_GHE__OAUTH_CLIENT_ID/DDA_GHE__OAUTH_CLIENT_SECRET. - Tachyon credentials (
DDA_TACHYON__*) — required at boot forenv=prod(api_key,chat_completion_base_url,usecase_id), plusDDA_LLM__PRIMARY_MODEL/DDA_LLM__FAST_MODEL. The gateway runs over the org-internaltachyon-sdkpackage, installed separately into the backend venv (not apyproject.tomldependency; a missing package surfaces asCONFIG_INVALIDat the first LLM call, not at boot). - Existing-system API endpoints — config/org-apis.json
ships with placeholder
https://existing.example/...URLs. At deployment, copy it, replace everyurlTemplate/getUrlTemplatewith real endpoints, pointDDA_ORG_APIS__PATHat the copy, setDDA_ORG_APIS__CLIENT_ID, and mapauthRefnames to tokens viaDDA_ORG_APIS__AUTH. Without the file, registration/record stages run in deferred mode. - The
releasescollection (name viaDDA_MONGO__RELEASES_COLLECTION) holds the org's release documents, keyed_id = {releaseDate}_{fixVersionName}_{applicationName}. A missing or link-incomplete release is created/completed by the agent (prefilled from the Jira fixVersion, links gathered from the user), so seeding is a bootstrap aid, not a prerequisite.
The Makefile is a thin orchestration layer over the nested projects — the files are
not moved to the root. Each target runs the exact command documented in the sections below.
Because the repo is Windows-primary and make is not guaranteed on Windows, the copy-paste
PowerShell equivalent is listed for every target (run these from the repo root
d:\Project\excel).
The Makefile assumes the checked-in Windows venv layout (
backend/.venv/Scripts/). On POSIX, run any target withmake VENV_BIN=bin <target>.
make target |
What it does | PowerShell equivalent |
|---|---|---|
make setup |
Create the backend venv + install [dev], npm install the frontend, install the Excel-form deps. |
(see §3.2, §3.3, §3.4) |
make seed |
Seed artifact templates into Mongo. | cd dda-agent; backend\.venv\Scripts\python scripts\seed_templates.py --mongo-uri mongodb://localhost:27017 --db dda_agent |
make seed-jenkins-nodes FILE=<seed.json> |
Seed owner→Jenkins-node rows (copy + fill config/jenkins-owner-nodes.example.json). | cd dda-agent; backend\.venv\Scripts\python scripts\seed_jenkins_nodes.py --mongo-uri mongodb://localhost:27017 --db dda_agent --file <seed.json> |
make seed-releases FILE=<seed.json> |
Seed org releases (copy + fill config/releases.example.json); stop-gap until the org release sync owns the collection. | cd dda-agent; backend\.venv\Scripts\python scripts\seed_releases.py --mongo-uri mongodb://localhost:27017 --db dda_agent --file <seed.json> |
make preflight |
Deployment readiness report — settings, config cross-refs, templates, reachability, seed counts, placeholder detection (docs/deployment.md). | cd dda-agent\backend; .venv\Scripts\python -m dda_agent.preflight |
make backend |
Run the FastAPI backend on 127.0.0.1:8100. |
cd dda-agent\backend; .venv\Scripts\python -m uvicorn --factory dda_agent.main:create_app --host 127.0.0.1 --port 8100 |
make frontend |
Run the Vite dev server (http://localhost:5173). |
cd dda-agent\frontend; npm run dev |
make build |
Type-check + production-bundle the frontend. | cd dda-agent\frontend; npm run build |
make test |
Backend pytest + frontend vitest. | (both rows below) |
make test-backend |
Backend test suite. | cd dda-agent; backend\.venv\Scripts\python -m pytest |
make test-frontend |
Frontend component/store tests. | cd dda-agent\frontend; npm test |
make lint |
Ruff over backend + tests. | cd dda-agent; backend\.venv\Scripts\python -m ruff check . |
make complexity |
Cognitive-complexity gate (flake8, CCR001 ≤ 15). | cd dda-agent; backend\.venv\Scripts\python -m flake8 backend\src |
make typecheck |
The enforced pyright gate (0 errors), run from the repo root. | dda-agent\backend\.venv\Scripts\python -m pyright |
make form-build |
Build the CDM-Next Prework workbook. | python excel-form\build\build_workbook.py |
make mock |
Run the mock server + portal on :8080. |
node mock-server\server.js |
cd dda-agent
python -m venv backend\.venv
backend\.venv\Scripts\python -m pip install -e "backend[dev]"
(POSIX: python -m venv backend/.venv && backend/.venv/bin/pip install -e "backend[dev]".)
Editable install (-e) is the source of truth for dependency ranges
(backend/pyproject.toml). For a reproducible pinned install use
the lockfiles instead: python -m pip install -r requirements-dev.txt (dev — includes runtime) or
-r requirements.txt (runtime only). The org-internal tachyon_sdk (LLM gateway) is not on
PyPI — install it from your internal index separately; without it, LLM calls degrade
deterministically.
To install every Python dependency across both subprojects in one shot, use the consolidated
root manifest: python -m pip install -r requirements.txt (or -r requirements-dev.txt), or
python -m pip install -e ".[dev]" from the repo root. That aggregator installs third-party deps
only; the editable dda_agent package still comes from pip install -e "dda-agent/backend[dev]"
(what make setup runs).
IDE / type-checking. Point your editor's Python interpreter at
dda-agent/backend/.venv/Scripts/python.exe. The repo ships pyrightconfig.json
and .vscode/settings.json (repo root) that pin this interpreter and teach Pylance the src
layout and the tests root (tests import from helpers.…). The enforced gate is pyright/Pylance
in basic mode → 0 problems across src and tests:
backend\.venv\Scripts\python -m pyright # run from the repo root d:\Project\excel
backend\.venv\Scripts\python -m mypy backend\src # advisory only, from dda-agent\
Configure. Copy backend/.env.example to .env in the
directory you launch from (settings read ./.env), or export DDA_* variables directly.
Seed the artifact templates (dev/prod; env=test seeds its in-memory store from the same file
automatically):
backend\.venv\Scripts\python scripts\seed_templates.py ^
--mongo-uri mongodb://localhost:27017 --db dda_agent ^
[--file backend\config\templates-seed.json] [--dry-run]
The script validates every template before touching Mongo (placeholders must parse;
requiredPlaceholders ⊆ placeholders(content ∪ pathTemplate)), then upserts by
{templateId, version}; marking a version active deactivates that template's other versions
first. --dry-run validates without writing. Exit code 2 = validation refused.
Run the backend (app factory — settings load at call time, never at import):
cd dda-agent\backend
.venv\Scripts\python -m uvicorn --factory dda_agent.main:create_app --host 127.0.0.1 --port 8100
Startup performs, in order: (1) AppSettings() — reads env/.env; env=prod fails fast listing
all missing required keys at once; (2) build_container — validates pattern-behaviors.json,
type-mappings.json, the excel-form configs, and org-apis.json (when configured); a broken
config file never boots; (3) lifespan — Mongo index creation (idempotent) → template cross-check
(every referenced templateId must exist and be active) → recovery startup sweep (repairs
dangling effect intents, releases stale writer locks, auto-resumes Jenkins-only POLLING_CI
sessions).
cd dda-agent\frontend
npm install
npm run dev # Vite dev server (default http://localhost:5173); proxies /api -> 127.0.0.1:8100 (REST + WS)
npm run build # tsc --noEmit + production bundle into dist/
npm run preview # serve the production bundle locally
npm test # vitest component/store tests
For a dev setup with the SPA on 5173 calling the backend origin directly, set
DDA_CORS_ORIGINS=["http://localhost:5173"] (the dev proxy otherwise avoids CORS).
Building the workbook is pure Python (no Excel needed):
# 1. Install Python dependencies
pip install -r excel-form/requirements.txt
# 2. (Only if you changed VBA) rebuild the template
python excel-form/build/build_template.py
# 3. Build the workbook (bakes ONE environment's base URL; default env: "dev")
python excel-form/build/build_workbook.py
python excel-form/build/build_workbook.py --env uat # pick a different environment
# 4. Structural verification (no Excel needed)
python excel-form/build/verify_xlsm.py
# 5. Open excel-form/dist/CDM-Next-Prework.xlsm in Excel, enable macros, fill the
# form, and click "✓ Validate & Submit".
# 6. Extract submissions from a filled-and-saved copy
python excel-form/build/parse_output.py path/to/filled.xlsm -o out.json
A zero-dependency Node (≥ 18) implementation of the Excel-form HTTP contract that also serves the portal — everything works end-to-end with no Java and no MongoDB:
node mock-server/server.js # or: npm start (from mock-server/)
# → http://localhost:8080 (API at /api/v1, portal at /)
It requires excel-form/dist/config.bundle.json (build it first) and reads only the PORT
environment variable (default 8080).
A user opens a session and gives the agent a Jira key. Two intake modes exist, detected from the issue itself:
- Mode A —
MASTER_EXCEL: a master Jira carrying a CDM-Next Prework.xlsmattachment. The workbook is downloaded (size-guarded, stored in GridFS) and parsed by reusing the siblingexcel-form/buildpure-Python parser (parse_output.py/config_loader.py, loaded read-only via an importlib bridge). Each workbook row becomes a requirement; the agent creates one child Jira per valid requirement. - Mode B —
SINGLE_JIRA: a single pre-created requirement Jira. The description is parsed by a deterministic codec first, with a hardened LLM fallback (LLM-filled fields are badged in the Gate-1 preview). The issue itself is the child; missing internal fields are collected conversationally (GATHERING_INTERNAL).
Field validation is the org's pattern-validate API when the org-system client is wired —
the flat payload carries patternName, every General-section field the form config defines
(except the attachment) plus the Jira identity / release / feature metadata captured at intake;
failed form keys re-open the edit card, failed non-form keys park with INTAKE_VALIDATION_FAILED.
The attachment is validated inside the agent (accept / maxSizeKb / content rules), so it never
rides the payload. Without the client the local rules stand in (LocalValidator mirrors the
authoritative VBA / config bundle), and they always still gate card answers and gate edits. Every external side effect is journaled for
exactly-once semantics, and progress is checkpointed to MongoDB so a crash or restart resumes
where it left off.
| Gate | State | Card (pendingAction.gate) |
Approves |
|---|---|---|---|
| 1 | AWAITING_JIRA_APPROVAL |
jira_preview |
Child-Jira drafts (summary, description, labels, custom fields, adoption of existing issues) |
| 2 | AWAITING_REPO_APPROVAL |
repo_plan |
Repository slot bindings, commit order, branch name, skips |
| 3 | AWAITING_ARTIFACT_APPROVAL |
artifact_preview |
Generated artifact bytes (content + diffs against branch head) |
| 4 | AWAITING_BUILD_APPROVAL |
build_trigger |
The CI job list (enable/disable per job) |
Every gate supports approve (optionally with structured edits), revise (comment-driven rework, revision bump) and abort (terminates the session with cleanup).
flowchart TD
A[CREATED] -->|jiraKey given| B[FETCHING_JIRA]
A -->|no key| K[AWAITING_JIRA_KEY] --> B
B -->|mode A: .xlsm attached| C[DOWNLOADING_ATTACHMENT --> PARSING_WORKBOOK]
B -->|mode B: requirement Jira| D[PARSING_DESCRIPTION]
C --> V[VALIDATING]
D --> V
V -->|invalid fields| G[GATHERING_INPUTS] --> V
V -->|mode A valid| G1{{Gate 1: jira_preview}}
G1 --> CJ[CREATING_CHILD_JIRAS] --> RJ[REGISTERING_JIRA]
V -->|mode B valid| RJ
RJ -->|mode B| GI[GATHERING_INTERNAL] --> RR[RESOLVING_REPOS]
RJ -->|mode A| RR
RR -->|credentials missing| CW[AWAITING_GITSAAS_PAT / AWAITING_GHE_AUTH] --> G2
RR --> G2{{Gate 2: repo_plan}}
CJ -->|write credential missing| WC[AWAITING_JIRA_PAT / AWAITING_ORG_TOKEN] --> CJ
G2 --> CR[CREATING_RECORDS --> UPDATING_RECORDS] --> GA[GENERATING_ARTIFACTS]
GA --> G3{{Gate 3: artifact_preview}}
G3 -->|revise| GA
G3 --> CM[COMMITTING]
CM --> G4{{Gate 4: build_trigger}}
G4 --> TC[TRIGGERING_CI] --> PC[POLLING_CI] --> RP[REPORTING] --> Z[COMPLETED]
RP -->|any job failed| UF[AWAITING_USER_FIX: retry / accept_partial] --> TC
Park states (AWAITING_USER_FIX, BLOCKED_ONBOARDING), FAILED and CANCELLED are reachable
from every non-terminal state and are omitted above for readability.
| Component | Role |
|---|---|
FastAPI backend (backend/src/dda_agent) |
REST + WebSocket API, workflow engine, adapters. App factory dda_agent.main:create_app; default bind 127.0.0.1:8100. |
React SPA (frontend/) |
Vite + React 18 + TypeScript + zustand. Session list, chat, stage rail, gate cards, credential prompts, build status. Dev server proxies /api (REST + WS) to the backend. |
| MongoDB | All persistent state (sessions, requirements, records, artifacts, templates, side-effect journal, audit events, messages, commit audits) + GridFS for attachments/oversize blobs. Two org collections (releases, jenkins_owner_nodes) are read at onboarding and written back with user-supplied details (Spring _class markers preserved). |
| Jira Data Center | Intake source and reporting target. Service-account Basic auth. |
| GitHub Enterprise (GHE) | Repo platform for platform: "GHE" slots. User authorizes via an OAuth App (browser popup). |
| github.com (GITSAAS) | Repo platform for platform: "GITSAAS" slots. User supplies a PAT with the repo scope. |
| Jenkins | CI for ci.type: "JENKINS" slots. Service user + API token; per-owner node URLs from Mongo. |
| GitHub Actions | CI for ci.type: "GITHUB_ACTIONS" slots, via workflow_dispatch on the session's GITSAAS credential. |
| Existing-system APIs | Config-driven HTTP choreography (backend/config/org-apis.json) for record registration. Optional: without the config file these stages run in deferred mode. |
| Tachyon (LLM gateway) | Org LLM gateway behind the LLMGateway port. Wired when DDA_TACHYON__API_KEY or DDA_TACHYON__CHAT_COMPLETION_BASE_URL is set; otherwise llm=None and every LLM-dependent path degrades deterministically. env=test uses a deterministic ScriptedGateway. |
Ports and adapters. The backend is strictly ports-and-adapters: ports/ holds one ABC per
external dependency; adapters/ holds the concrete implementations; container.py is the only
place adapters are constructed. Domain and orchestration code import only the ports. env=test
wires in-memory adapters throughout; tests inject scripted doubles through container overrides.
State machine, checkpointing, journal.
- 32 states (
orchestration/states.py) with an explicit legal-transition table. The engine (orchestration/engine.py) loads the session, runs the current state's handler, applies the result, and routes failures through the remediation policy. - Every advance is a
findOneAndUpdate {_id, version}compare-and-swap; the newversionnumber is the auditseq, gluing the audit trail and the WS replay stream to the session version. There are no multi-document transactions; domain writes happen first (idempotent), the session CAS is the single commit point, the audit event is appended after. - Every external side effect follows intent → call → commit(result) under a deterministic
idempotency key (
side_effectscollection). Re-running a stage adopts committed effects. Ambiguous failures leave the INTENT row for per-kind orphan checks to resolve (adopt vs. fresh attempt) — the exactly-once discipline.
All settings are environment-driven with prefix DDA_ and __ for group nesting
(DDA_JIRA__BASE_URL → jira.base_url), or come from a .env file. Secrets are SecretStr —
never in repr() or logs. Dict/list values are JSON. Inside a matched group an unknown key fails
loudly at startup (extra="forbid"); unknown top-level DDA_ variables are ignored.
env=prod requires (boot fails otherwise, listing every missing key): jira.base_url,
jira.service_id, jira.service_password, tachyon.api_key,
tachyon.chat_completion_base_url, tachyon.usecase_id, state_signing_key. dev/test
tolerate empty values.
| Variable | Type | Default | Description |
|---|---|---|---|
DDA_ENV |
dev|test|prod |
dev |
test wires fully in-memory adapters (no Mongo); dev/prod need MongoDB; prod adds fail-fast key validation. |
DDA_LOG_LEVEL |
str | INFO |
Python/structlog level. |
DDA_STATE_SIGNING_KEY |
secret | "" |
HMAC key signing the OAuth state + browser-binding cookie. GHE flow raises CONFIG_INVALID without it. |
DDA_CORS_ORIGINS |
JSON list | [] |
Enables CORS for the listed SPA origins. |
DDA_PATTERN_BEHAVIORS_PATH |
path | <backend>/config/pattern-behaviors.json |
Pattern behavior config (validated at boot). |
DDA_CAPABILITIES_PATH |
path | <backend>/config/capabilities.json |
Which flows this deployment can run, which artifact provider each uses, and who may answer in the chat box (chatResponders). Cross-checked at boot: a capability naming an unwired provider fails the process. |
DDA_FIELD_SOURCES_PATH |
path | <backend>/config/field-sources.json |
Where each non-form session value is read from on the Jira issue (captures, linked-issue reads, write-back plans). Validated at boot; every problem listed at once. |
DDA_REPORT_CATALOGS_PATH |
path | <backend>/config/report-catalogs.json |
The per-requirement catalogs the final report reads from the org system, and the columns each one shows. A catalog's api must exist in org-apis.json (boot cross-check). |
DDA_TYPE_MAPPINGS_PATH |
path | <backend>/config/type-mappings.json |
Source→BigQuery type maps for DDL. |
DDA_EXCEL_FORM_BUILD_DIR |
path | <repo>/excel-form/build |
Directory with parse_output.py + config_loader.py (read-only reuse). |
DDA_TEMPLATES_SEED_PATH |
path | <backend>/config/templates-seed.json |
Seed file for the env=test template store and seed_templates.py's default --file. |
| Variable | Default | Description |
|---|---|---|
DDA_SERVER__HOST / DDA_SERVER__PORT |
127.0.0.1 / 8100 |
Bind host/port. |
DDA_MONGO__URI / DDA_MONGO__DB |
mongodb://localhost:27017 / dda_agent |
Motor connection + database. env=prod refuses the localhost default. |
DDA_MONGO__RELEASES_COLLECTION |
releases |
Org releases collection, keyed _id = {releaseDate}_{fixVersion}_{applicationName}. The agent reads it at validation and writes back: a missing release is created (prefilled from the Jira fixVersion, links gathered on a release_details card), blank links are completed. Bootstrap via scripts/seed_releases.py. |
DDA_MONGO__JENKINS_OWNER_NODES_COLLECTION |
jenkins_owner_nodes |
Org owner→Jenkins-node mapping ({_id: <owner>, jenkinsNode, _class}, exact-match key). Unmapped owners are gathered from the user at Gate 2 on a jenkins_node card and written back. Bootstrap via scripts/seed_jenkins_nodes.py. |
DDA_MONGO__JIRA_STORY_INTAKE_COLLECTION |
jiraStoryIntake |
The org intake application's collection (_id = jiraStoryIntakeId). The agent's only write is the mode-A master link at registration: $set of masterJiraId + updatedDttm on the found record — no upsert, _class and every org field untouched. |
DDA_MONGO__SERVER_SELECTION_TIMEOUT_MS / __CONNECT_TIMEOUT_MS / __SOCKET_TIMEOUT_MS |
5000 / 5000 / 30000 |
Driver bounds — an unreachable Mongo surfaces as a timely error, never an unbounded stall. |
DDA_MONGO__MAX_POOL_SIZE |
50 |
Motor connection-pool ceiling. |
DDA_LIMITS__MAX_WORKBOOK_MB / __MAX_INLINE_ATTACHMENT_MB |
20 / 10 |
Input-size ceilings; oversized inputs park USER_FIXABLE before any decode/parse. |
DDA_SESSIONS__PARKED_TTL_H |
336 |
Sessions idle in a WAIT/park state longer than this are reaped (releases the unique anchor for re-onboarding); 0 disables. |
DDA_SESSIONS__REAPER_INTERVAL_S |
3600 |
Reaper cadence. |
DDA_SESSIONS__RESUME_CONCURRENCY |
5 |
Bound on post-restart auto-resume tasks (no thundering herd against Mongo/Jenkins). |
DDA_AUTH__MODE |
dev |
dev (fixed dev-user, header honored if present) or header (identity header required, 401 without). Both spoofable — internal networks only. |
DDA_AUTH__HEADER_NAME |
X-User-Id |
The identity header name. |
| Variable | Default | Description |
|---|---|---|
DDA_JIRA__BASE_URL |
"" |
Jira DC base URL. Unset ⇒ adapter unwired (/health/deps → not_configured). |
DDA_JIRA__SERVICE_ID / DDA_JIRA__SERVICE_PASSWORD |
"" |
Service-account Basic auth. |
DDA_JIRA__ISSUE_TYPE |
Requirement |
Issue type for child Jiras (also the createmeta preflight target). The project is never configured — children are created on the anchor issue's own project. |
DDA_JIRA__LINK_TYPE |
Relates |
Link type between child and anchor issue. |
DDA_JIRA__CUSTOMFIELDS |
{} |
Logical field key → customfield_NNNNN. A missing required field fails Gate-1 createmeta (JIRA_CREATEMETA_MISSING_FIELD). |
DDA_JIRA__MAX_ATTACHMENT_MB |
50 |
Attachment download guard (ATTACHMENT_TOO_LARGE beyond it). |
DDA_JIRA__TRANSITIONS |
{} |
Optional transition names per lifecycle event (onCommitted, onBuildSuccess, onBuildFailure); unavailable = warning, never error. Created children always mirror the MASTER's current status (a leftover onChildCreated entry is ignored). |
| Variable | Default | Description |
|---|---|---|
DDA_GHE__BASE_URL / DDA_GHE__API_URL |
"" |
GHE web base + REST API base (e.g. .../api/v3). |
DDA_GHE__OAUTH_CLIENT_ID / DDA_GHE__OAUTH_CLIENT_SECRET |
"" |
OAuth App client id/secret. |
DDA_GHE__REDIRECT_URI |
"" |
Must exactly match the OAuth App registration: <backend origin>/api/v1/auth/ghe/callback. |
DDA_GITSAAS__API_URL |
"" |
API base for GITSAAS slots (PAT per session). Empty ⇒ the adapter targets the public https://api.github.com; an explicit value also opts GitSaaS into /health/deps readiness. |
DDA_JENKINS__CANONICAL_URL |
"" |
Canonical Jenkins URL — always the crumb issuer + fallback base. Unset ⇒ not_configured. |
DDA_JENKINS__USER / DDA_JENKINS__API_TOKEN |
"" |
Service user + API token (crumb-exempt). |
In dev/prod the container builds a TachyonGateway whenever DDA_TACHYON__API_KEY or
DDA_TACHYON__CHAT_COMPLETION_BASE_URL is set (fail-loud on partial config — the three required
keys validated at once). With neither set, llm=None and every LLM-dependent path uses its
deterministic fallback. The gateway imports the org-internal tachyon-sdk lazily at the first
completion.
| Variable | Default | Description |
|---|---|---|
DDA_TACHYON__API_KEY |
"" |
Tachyon API key (secret; prod-required). |
DDA_TACHYON__CHAT_COMPLETION_BASE_URL |
"" |
Chat-completions base URL (prod-required). |
DDA_TACHYON__USECASE_ID |
"" |
Use-case identifier (prod-required). |
DDA_TACHYON__APIGEE_URL / __CONSUMER_KEY / __CONSUMER_SECRET / __CERT_PATH / __USE_API_GATEWAY |
"" / false |
Optional Apigee gateway settings. |
DDA_LLM__PRIMARY_MODEL / DDA_LLM__FAST_MODEL |
"" |
Per-tier models. Empty at the first call of that tier is CONFIG_INVALID. |
DDA_LLM__PRIMARY_TIMEOUT_S / DDA_LLM__FAST_TIMEOUT_S |
120 / 30 |
Per-tier timeouts. |
With DDA_DIVA__STREAM_URL set, general chat questions — anything the intent classifier does not
recognise as being about the current run — are answered by DIVA, streamed, and attributed
(author: "diva" on the message frames). Empty ⇒ the responder is not wired and every question
keeps its in-session answer. A half-set block fails the boot in every environment — a partial
DIVA disables the responder while looking configured. The service credential is exchanged per
request (never cached) and never persisted or logged.
| Variable | Default | Description |
|---|---|---|
DDA_DIVA__STREAM_URL |
"" |
The enable switch. The SSE chat endpoint (https required). |
DDA_DIVA__AUTH_URL |
"" |
Basic-auth token endpoint (https required). |
DDA_DIVA__CONSUMER_KEY / __CONSUMER_SECRET |
"" |
Service credential (the secret is SecretStr). |
DDA_DIVA__TENANT_ID / __ORCHESTRATION_ID |
"" |
Tenant and orchestration the questions run under. |
DDA_DIVA__STREAM_TIMEOUT_S |
120 |
Whole-stream budget per question. |
Verify with the preflight diva section — a real token exchange, not a ping.
| Variable | Default | Description |
|---|---|---|
DDA_ORG_APIS__BASE_URL |
"" |
The enable switch. Absent ⇒ REGISTERING_JIRA / CREATING_RECORDS / UPDATING_RECORDS (and kind: api-external tools) run in DEFERRED mode, and a pattern routed to the org artifact engine produces nothing. Set it to turn the org system on. |
DDA_ORG_APIS__PATH |
<backend>/config/org-apis.json |
Path to a deployment copy of org-apis.json. The shipped file is used when unset — this does NOT gate DEFERRED mode; BASE_URL does. |
DDA_ORG_APIS__CLIENT_ID |
"" |
This app's x-wf-client-id UUID, sent on every org-system call. |
DDA_ORG_APIS__AUTH |
{} |
Secret JSON map authRef → static bearer token (fallback for no-UI runs; the per-session org-framework Bearer — carried on the socket's opening auth frame, or entered on the credential card — takes precedence). An unmapped authRef sends no Authorization. |
DDA_ORG_APIS__USE_PING_TOKEN |
"" |
Secret. A hand-generated framework token that overrides the per-session broker token on every call (no-UI/dev). |
DDA_ORG_APIS__MAX_CONCURRENT |
8 |
Bulkhead: concurrent org-system calls across ALL sessions. |
DDA_ORG_APIS__BREAKER_THRESHOLD |
5 |
Consecutive dependency faults before the circuit opens. |
DDA_ORG_APIS__BREAKER_RESET_S |
30.0 |
How long the circuit stays open before a trial call is admitted. |
- config/capabilities.json — the flows this
deployment can run, as data: each capability's pipeline, its gate policy, and optionally the
artifactProviderits patterns generate through (a flow that runs the same patterns through a different generator is the reason that field exists). Validated at boot againstcapabilities.schema.json, and cross-checked against the wired providers and stage handlers — a capability naming a provider nothing wires, or a gate no stage honours, fails the process rather than a session. - config/pattern-behaviors.json — per-pattern
repoSlots(position/label/platform/artifactTypes/ci),artifacts(tool specs:kind: template | ddl | api | api-external,slot,templateId,pathTemplate,expand,dialect,typeMapping,apiRef),columnDetails(how to read the per-requirement column workbook),descriptionSource. Validated at boot againstpattern-behaviors.schema.jsonplus registry cross-checks.- User-chosen slot platform — a slot may declare
platforms: ["GHE", "GITSAAS"](withplatformas the pre-selected default); the session then asks the user where each such repo lives on aplatform_choicecard before any credential prompt, since the pick decides which credentials (GHE OAuth vs GitSaaS PAT) are collected. Absentplatforms, the slot is fixed toplatformexactly as before. - Per-platform CI —
ciis either one CI object (serving the default platform only) or a map keyed by platform ({"GHE": {…JENKINS…}, "GITSAAS": {…GITHUB_ACTIONS…}}). A chosen platform with no CI entry builds no CI job for that slot. Boot rejects a default platform not offered inplatforms, CI keyed on a platform the slot can never target, and mergeable slots (same position + platform across patterns) that disagree on CI.
- User-chosen slot platform — a slot may declare
- config/org-apis.json — per-API
{method, urlTemplate, getUrlTemplate?, authRef?, payloadMapping, responseMapping, idempotency, timeoutS?}.idempotency∈header|get-before-post|safe-put. An emptyresponseMappingreturns the raw body (howkind: apiartifact tools receive content). Ships with placeholder URLs — the fill-in procedure and per-API spec-request sheet: docs/org-apis-onboarding.md. - config/field-sources.json — where each
non-form session value comes from: Jira issue paths (with
{customfields.NAME}indirection), one-hop linked-issue reads, defaults, and the write-back plans that let a rejected payload key be corrected on the anchor Jira. The full how-to: dda-agent/docs/field-configuration.md. - config/report-catalogs.json — the per-requirement catalogs the report reads back from the org system (requirement record, Autosys schedule): which API answers, our name and label for every column, the layout, and which columns feed the charts. A catalog the org holds nothing for is absent from the report, not empty.
- config/type-mappings.json —
{version, mappings: {<name>: {<SOURCE TYPE>: <target>}}}used bykind: ddlartifacts.__default__: nullmeans unmapped types error (TYPE_MAPPING_FAILED). Shipshive-to-bigquery,teradata-to-bigquery. - Templates (config/templates-seed.json + the
templatescollection) — versioned artifact templates rendered by a strict, non-Jinja{{dotted.path|filter}}engine overrecord.*/derived.*/meta.*/table.*. Whitelisted filters:upper,lower,snake,yamlq,sqlident,default:<literal>. Every unresolved placeholder is collected and raised asTEMPLATE_RENDER_FAILED.
Database DDA_MONGO__DB (default dda_agent). All documents carry schemaVersion; datetimes are
naive UTC. No secrets are ever stored: PAT/OAuth tokens live only in the in-process credential
broker; the session capability is stored as a sha256 hash; credential_response frames are
redacted before any persistence/logging.
| Collection | Purpose |
|---|---|
sessions |
One document per onboarding session: the state machine's persisted truth (state, version == audit seq, mode, jira, cursor, pendingAction, repoPlan, buildPlan, release, credsNeeded, lock, error, capabilityHash). |
requirements |
One per parsed requirement (workbook row / mode-B issue): fields, validation, gathering, source, childJira, externalRefs, skippedArtifacts. |
records |
Registry mirror per (session, requirement): frozen plan, artifacts[], builds[], businessKey (drtId + targetSchema + targetTableName), status (VALIDATED→…→REPORTED / ABANDONED), externalRefs. |
artifacts |
Generated artifacts with full review lifecycle: content/contentGridFsId, contentSha256, inputsFingerprint, status (GENERATED→PREVIEWED→APPROVED→COMMITTED / REJECTED), action, baseBlobSha, diff, supersedes, commitSha. |
templates |
Versioned artifact templates (see §4.3); one active version per templateId. |
side_effects |
The effect journal: intent/commit rows per external side effect under a unique idempotencyKey; status INTENT/COMMITTED/FAILED/ORPHAN_CHECKED. |
audit_events |
Append-only audit trail; replayable WS envelopes ride on it (seq == session version). |
messages |
Chat transcript (user/agent/system). |
commit_audits |
Insert-only compliance read model: Jira → approval → bytes → commit → build, one doc per (session, repo, commit). |
releases (external, agent-completed) |
Org release registry, _id = {releaseDate}_{fixVersion}_{applicationName}; missing/incomplete records are created from the Jira fixVersion + user-supplied links. |
jenkins_owner_nodes (external, agent-completed) |
Repo owner → Jenkins node URL ({_id: <owner>, jenkinsNode, _class}); unmapped owners are gathered at Gate 2 and written back. |
GridFS: one bucket attachments, keyed by metadata.sha256 with content-hash dedupe; stores
downloaded Jira attachments and artifact content larger than the 256 KB inline ceiling.
Base path /api/v1. The SPA does not use HTTP for data — every read and write travels the
WebSocket (§4.6). What is left here either cannot ride a frame or is not called by the app at all:
| Method & path | Auth | Purpose |
|---|---|---|
GET /sessions/{sessionId}/runbook |
owner + X-DDA-Capability |
The agent's own onboarding runbook as a Word download (Content-Disposition: attachment), built on demand from the stores — COMPLETED sessions only (else 409 RUNBOOK_UNAVAILABLE); wrong/missing capability → 403. Never persisted or committed. |
GET /sessions/{sessionId}/org-runbook?requirementId= |
owner + X-DDA-Capability |
Proxies the runbook the org system rendered, byte for byte, under its own filename. 409 RUNBOOK_UNAVAILABLE when it has published none — the one code the client may answer by falling back to the agent's own. 409 RUNBOOK_AMBIGUOUS (with requirementIds) when the session has several and none was named. |
GET /auth/ghe/login?state=<signed> |
auth-exempt | OAuth popup entry (verifies HMAC state, sets the browser-binding cookie, 302 to GHE authorize). A provider redirect is a browser navigation; no frame can carry one. |
GET /auth/ghe/callback?code=&state= |
auth-exempt | OAuth return leg (binding-cookie check, single-use state, code exchange, token broker, self-closing popup). |
GET /health |
none | Liveness {status: "ok", version}. |
GET /health/deps |
none | Dependency reachability {mongo, jira, ghe, jenkins} (1 s/check, 30 s cache). |
GET /metrics |
principal | Prometheus text exposition, derived on demand from the stores. |
A download is a browser navigation rather than a data feed: it needs a binary body and a
Content-Disposition filename. The probes exist for a load balancer and Prometheus, which cannot
speak the app protocol. tests/unit/test_transport_surface.py fails if any other route appears, or
if the SPA calls fetch() outside its download helper.
WS /api/v1/sessions/{sessionId}/ws the session connection
WS /api/v1/ws the control connection (no session)
Connect sequence: accept (unauthenticated and useless) → the client's first frame is auth
(nothing else is read until it lands; no frame within 5 s → close 4003) → ownership check
(a session owned by someone else closes 4003, disclosing nothing) → seat (writer by default;
a new writer supersedes the previous one with 4001) → if writer and in an auto state, the
engine continues → a session_state snapshot → replay of persisted envelopes with
seq > lastSeq → live frames.
A browser WebSocket cannot set request headers, which is why a single-use ticket used to be
minted over HTTP first. The auth frame replaces it: no second transport, and no secret in a URL.
The declared headers never override the real ones: a deployment behind an SSO proxy authenticates
from the genuine header on the handshake, so a client cannot claim an identity the proxy did not
assert. (Both sides are lower-cased before the merge — HTTP header names are case-insensitive, so
an override by dict key alone would leave X-User-Id and x-user-id both standing and the
case-insensitive lookup would return the client's.) capability and orgToken are lifted into
locals before anything logs or persists the frame.
Origin is checked against cors_origins on both sockets. A WebSocket handshake is not subject
to the same-origin policy and carries the user's cookies and proxy-asserted headers whatever page
opened it, so without this any site the user visits could open an authenticated connection to their
session. An absent Origin is allowed — non-browser clients send none, and only a browser is subject
to the attack. An empty cors_origins means the deployment has not restricted origins.
Envelope (both directions, camelCase):
{ "v": 1, "id", "sessionId", "type", "ts", "replyTo", "seq", "payload" }. seq is server-assigned
per persisted event and is the replay cursor; ephemeral frames carry seq: null. sessionId is
empty on the control connection. A breaking change bumps v (mismatched clients closed 4002;
server speaks v: 1).
Close codes: 4001 superseded (another writer took the seat) · 4002 unsupported protocol
version · 4003 not authenticated, unknown session, foreign owner, or a wrong capability ·
4009 slow consumer (evicted; reconnect + replay loses nothing) · 4500 snapshot/replay failed ·
4511 unexpected handler error · 1000 (idle timeout) after 90 s of silence on the session
connection, 300 s on the control connection (send ping every ~30 s).
Reads travel the same connection as live push:
// client
{ "type": "request", "id": "<uuid>", "payload": { "op": "stages.get", "args": {} } }
// server
{ "type": "response", "replyTo": "<uuid>", "seq": null,
"payload": { "ok": true, "data": { } } }
{ "type": "response", "replyTo": "<uuid>", "seq": null,
"payload": { "ok": false, "error": { "code", "message", "userMessage", "suggestedFix" } } }A response is never persisted or replayed: a derived read is not something that happened to
the session. Every collection op is paged and the dispatcher refuses a payload over 4 MB with
RESPONSE_TOO_LARGE — an unbounded read sharing the connection would trip the 4009 eviction that
bulk was kept off the socket to avoid.
Reads are allowed for observers; the ops in WRITE_OPS are the writer's alone and are refused with
SESSION_LOCKED. Reads are answered concurrently (up to 8 in flight per connection, then
refused): one can wait on a third party — repos.list calls GitHub — and answering inline froze
every other frame on the connection meanwhile.
Session ops (api/ws/rpc.py → SESSION_OPS, all delegating to api/reads.py):
| Op | Args | Returns |
|---|---|---|
session.get |
— | The safe session view (never the capability hash or lock internals). |
stages.get |
— | {steps: [{state, status, enteredAtUtc, leftAtUtc, durationSeconds, visits, outcome}]}. |
children.list |
— | The child-Jira board rows, DISCARDED and DEFERRED included. |
messages.page |
before?, limit (≤200, default 50) |
Older chat history, oldest first. |
gate.items |
checkpointId, offset, limit (≤200, default 50), full, itemId |
Paged gate items by reference (Gate 1 → requirements; Gate 3 → artifacts, content capped at 64 KB unless full). itemId fetches ONE item by identity — an id that no longer resolves is dropped from the page, so a positional re-read would load, and then save over, a different artifact. |
repos.list |
platform |
{repos: [{owner, name}]} — one bounded call, never an org enumeration. |
auth.refresh |
orgToken |
Re-stash the org bearer (rolling TTL). The SPA calls this every 240 s; the token's own TTL is 300 s. |
Control ops (CONTROL_OPS, on WS /api/v1/ws — these are not session-scoped):
| Op | Args | Returns |
|---|---|---|
sessions.list |
status: all|resumable |
The caller's session summaries. |
sessions.create |
jiraKey? |
{sessionId, capability, status, state} — capability is returned once. The engine starts on the first writer connection. |
sessions.delete |
sessionId, capability? |
Abort: a wrong capability is refused, an absent one is fine (ownership decides), terminal is idempotent. |
Client → server: user_message {text} (writer) · input_response {checkpointId, values}
· approval_response {checkpointId, decision: approve|revise|abort, comment?, edits?} (gate
edits shapes are per-gate; capability required) · credential_response {platform, secret}
(secret stripped + redacted at ingress, validated against the platform GET /user, then brokered)
· session_control {action: pause|resume_pipeline|abort|retry|take_over, capability?} · ping.
Server → client (● = persisted for replay): session_state (connect snapshot) ·
stage_changed ● · agent_message ● / agent_message_delta / thinking · progress ●
(milestones incl. the final report) · input_request ● (payload.kind discriminates:
jira_key, mode_choice, confirm, fix_version_choice, gathering cards, release_details,
platform_choice, jenkins_node, staleness_warning,
artifact_correction_choice, build_retry) · credential_request ● {platform: GITSAAS|GHE|JIRA|ORG, kind: pat|oauth|wf_token, reason, skipOption, scopesNeeded? (repo-plan platforms), authorizeUrl? (GHE)} · approval_request ● (the gate card, items by
reference for Gates 1 & 3) · error ● · pong.
Reconnect/replay: pass lastSeq = the highest processed
seq; envelopes are persisted before send (a crash costs at most a duplicate, never a gap);
the snapshot's pending re-builds the current wait state's original *_request with its original
requestId. Any writer frame heartbeats the persisted writer lock (60 s TTL).
33 states; WAIT = the engine stops and waits for a user/webhook event; ● = journaled side-effecting stage (never auto-resumed unattended).
| Group | State(s) | What happens |
|---|---|---|
| Intake | CREATED (auto), AWAITING_JIRA_KEY (WAIT), FETCHING_JIRA (auto), DOWNLOADING_ATTACHMENT (auto ●), PARSING_WORKBOOK / PARSING_DESCRIPTION (auto) |
Fetch the issue, detect mode, download+parse (A) or codec-parse (B). |
| Validation | VALIDATING (auto), GATHERING_INPUTS (WAIT) |
excel-form-equivalent validation; release lookup; duplicate guard; conversational fix-up. |
| Gate 1 | AWAITING_JIRA_APPROVAL (WAIT gate) |
jira_preview; approve (+ field edits) → child creation; revise → summary redraft. |
| Registration | CREATING_CHILD_JIRAS (auto ●), REGISTERING_JIRA (auto ●), GATHERING_INTERNAL (WAIT, mode B) |
Journaled create/attach/link with label adoption + round-trip verification; org-system API-1; mode-B internal fields. |
| Repos & creds | RESOLVING_REPOS (auto), AWAITING_GITSAAS_PAT / AWAITING_GHE_AUTH (WAIT) |
Build the slot plan; prompt for credentials (PAT before GHE); skip_platform = documented partial onboarding. |
| Write creds | AWAITING_JIRA_PAT / AWAITING_ORG_TOKEN (WAIT) |
Lazy write-credential waits entered on first need by a side-effecting stage (registration → report): the user's Jira PAT (create/update issues as you) and the org-framework token (org-system registration); each resumes to the exact stage that needed it. |
| Gate 2 | AWAITING_REPO_APPROVAL (WAIT gate) |
repo_plan; server re-validates bindings/push permission/branch name/skips/order + CI preflights. A skipped slot can be restored (unskips), without which a repo-addressing refusal is inescapable. |
| Records | CREATING_RECORDS (auto ●), UPDATING_RECORDS (auto ●) |
Existing-system APIs 2–4; DEFERRED without config. |
| Artifacts | GENERATING_ARTIFACTS (auto) |
Template render, column-details → DDL, kind: api tools; fingerprint dedupe; oversize → GridFS. |
| Gate 3 | AWAITING_ARTIFACT_APPROVAL (WAIT gate) |
artifact_preview (items by reference, per-artifact diff); revise regenerates only commented tools (supersedes lineage). |
| Commit | COMMITTING (auto ●) |
Per repo in frozen order: staleness gate → base re-check → journaled branch create (adoption) → journaled commit (author = Gate-3 approver) → commit_audits doc; then api-external afterCommit + onCommitted transitions. |
| Gate 4 | AWAITING_BUILD_APPROVAL (WAIT gate) |
build_trigger; approve freezes buildPlan. |
| CI | TRIGGERING_CI (auto ●), POLLING_CI (auto) |
Journaled trigger (attempt-scoped keys); Jenkins snapshot wait / GHA discovery; persisted elapsed budgets. |
| Report | REPORTING (auto ●), COMPLETED (terminal) |
Marker-deduped Jira comments, tolerant transitions, statuses → REPORTED; any failed job parks the build_retry card. |
| Park / Terminal | AWAITING_USER_FIX / BLOCKED_ONBOARDING (WAIT park), FAILED / CANCELLED (terminal) |
USER_FIXABLE / ONBOARDING_REQUIRED park with resumeTo; UNFIXABLE → FAILED; abort → CANCELLED. |
Control & failure policy: pause stops after the current step (resume_pipeline continues);
abort → CANCELLED + credential wipe + ABORT_SUMMARY (created effects are listed, not rolled
back); retry re-enters resumeTo. AGENT_FIXABLE retries with backoff 2 s/8 s/30 s (escalates
after 3); USER_FIXABLE → AWAITING_USER_FIX; ONBOARDING_REQUIRED → BLOCKED_ONBOARDING;
UNFIXABLE → FAILED; credential errors park deterministically (invalid credentials dropped
first).
Resume: session state/cursor/version, pending action, requirements/records/artifacts, the
journal, audit + replay stream, chat, frozen plans (incl. persisted CI budgets) and GridFS blobs
survive a restart. Brokered credentials, OAuth nonces and the deps-health cache are in-process
and lost (the user is re-prompted). The boot recovery sweep repairs dangling intents and
releases expired locks; only Jenkins-only POLLING_CI sessions auto-resume.
A requirement that cannot proceed is taken out of the onboarding rather
than half-processed. RequirementStatus.DEFERRED sits beside DISCARDED, and
one selector — documents.active — hides both from every stage, report and
artifact. The board still shows them, and distinguishes them: this row was
wrong and not this time are different answers to someone asking why nothing
was built for it.
| Where | What it means |
|---|---|
Gate 1 (AWAITING_JIRA_APPROVAL) |
The last point at which deferring means no child Jira at all — the next stage creates them. Nothing downstream runs for the row. |
Generation (GENERATING_ARTIFACTS) |
No provider can build it. The Jira already exists 11 stages back and is kept; records, artifacts, commit and build are skipped, and Gate 3 names the requirement so approving an incomplete set is a visible choice. |
When no active requirement is left, the session can be set aside:
SessionState.DEFERRED is terminal (not CANCELLED — nothing failed) and not
resumable, and its working data is purged: requirements, artifacts, records
and chat messages. The session row and the entire audit trail survive — the
audit is the record of what was done on the user's behalf. The purge is
idempotent, because the session is marked before its data goes and a crash
midway must still be finishable.
Attachment blobs are deliberately not deleted. GridFS dedupes on the content hash, so this session's bytes may equally be another session's; nothing refcounts them, and deleting one still referenced would corrupt a session that did nothing wrong. Reclaiming them needs refcounting in the blob store.
- Auth modes (v1 — accepted risk):
dev(fixeddev-user, header honored) andheader(identity header required). The identity header is spoofable — v1 is for internal networks / behind a trusted proxy only. TheAuthProviderport keeps SSO a drop-in. - Capability token — mitigates header spoofing: a 32-byte urlsafe secret returned once at creation, only its sha256 stored. Ownership is what authorizes the sensitive surface; the capability is a tamper check on top of it — absent is fine for the owner (who must never be locked out by a lost client-held secret), a WRONG one fails loud. Comparisons are constant-time.
- Socket auth — the connection authenticates from its own first frame (§4.6). Identity comes
from the real handshake headers where a proxy asserts them, and the frame's declared headers
only fill in what the handshake did not carry, so a client cannot claim an identity the proxy
did not.
Originis checked againstcors_origins, because a WebSocket handshake is not subject to the same-origin policy. - Credential handling — four per-session secret kinds live only in the in-memory broker: the
GITSAAS PAT (TTL 24 h), the GHE OAuth token (
min(expires_in, 8 h)), the user's Jira PAT (writes as the acting user, TTL 24 h), and the org-framework token (min(expires_in, 1 h), default 5 min). Repo tokens are validated (GET /user+ PAT scope) before brokering; all are invalidated on 401s, wiped on abort; adapters fetch the token per request, never cache it. - Redaction, defense in depth — (1) ingress:
credential_response(andtake_over) secrets are replaced before validation/logging/persistence; (2) persistence: every stored envelope passesredact()again; (3) logging: a structlog processor blanks any event key containingsecret,authorization,password,token, orcapability. - GHE OAuth browser binding — the signed state (HMAC, 10-min, single-use nonce) is minted by an
authenticated step; the login hop sets a cookie HMAC-bound to the nonce and the callback requires
it, so a leaked
code+statecannot be completed from another browser. - Prompt-injection hardening — untrusted content is fenced with a "data, not instructions" preamble; outputs are structurally validated (keys filtered to known fields, values length-capped, ids re-matched) and flow through the same validator as typed input; the LLM never writes artifact bytes; LLM failure always degrades to a deterministic fallback.
- CI param hygiene — build param/input keys are linted against secret-shaped names at plan build; Jenkins parameters are a form-encoded POST body, never a query string.
Jenkins — node resolution owner → Mongo jenkins_owner_nodes (_id exact match; when unmapped
owners are the only Gate-2 blockers, a jenkins_node card gathers the URLs — domain-restricted to
the canonical controller — and writes them back; unknown at trigger time → JENKINS_NODE_UNKNOWN);
the crumb issuer is always the canonical URL; auth = user + API-token Basic (crumb-exempt since
Jenkins 2.96; on a 403 it fetches a crumb once over the same cookie jar); jobKind parameterized
(job/{org}/job/{repo}) or multibranch (branch child job, / → %2F); ensure_job_ready
snapshot wait for multibranch; every trigger carries DDA_RUN_ID=ci:<sessionId>:<org>/<repo> for
correlation + orphan checks; polling budgets queue 5 min, build 30 min.
GitHub Actions — workflow_dispatch on the session PAT; no run id at dispatch, so GitHub's Date
header is recorded as t0; run discovery matches workflow path + head_sha == committed sha +
unclaimed, adopts the earliest and best-effort cancels other matches (CI_CANCEL); a 180 s
discovery budget then exactly one journaled re-dispatch; Gate-2 preflight scans the workflow file on
the default branch.
Retry / accept-partial — builds run in parallel after Gate 4 (order constrains commits only).
Any enabled job ending non-SUCCESS parks the build_retry card: retry re-triggers only the
failed jobs under attempt+1 keys; accept_partial reports as BUILT_PARTIAL. Every terminal
build milestone is appended to the repo's commit_audits doc.
- Health:
GET /api/v1/health(liveness + version),GET /api/v1/health/deps(30 s cache) — eight checks (mongo,jira,ghe,jenkins,gitsaas,tachyon,diva,org_apis), eachok|error: …|not_configured; 503 whenever any configured dependency errors, so readiness gating can react. Every check is opt-in by configuration; setDDA_GITSAAS__API_URLexplicitly to include GitSaaS reachability (unset, the adapter targetsapi.github.comon its own and the check answersnot_configured). - Deployment readiness: the per-team checklist lives in docs/deployment.md
(org-system API onboarding: docs/org-apis-onboarding.md).
make preflight(python -m dda_agent.preflight [--env prod] [--skip-deps]) prints one report — settings, config cross-references, template presence, dependency reachability, a real LLM completion on both tiers (llm; also standalone aspython -m dda_agent.llm_check), a real DIVA token exchange (diva), seed-collection counts, placeholder-endpoint detection — and exits non-zero on any error. - Metrics:
GET /api/v1/metrics— Prometheus text (authenticated), derived on demand. Families:dda_sessions{state},dda_side_effects{kind,status},dda_requirements{status},dda_credentials_held(gauges),dda_errors_total{code}(point-in-time). Watch growth indda_side_effects{status="INTENT"}and sessions stuck in the park states. - Logs: structlog to stdout with the secret-redaction processor; level via
DDA_LOG_LEVEL(defaultINFO). Each line carries[session=<id>](bound per run/connection) — grep it to follow one session end to end: stage transitions (VALIDATING -> …), the outbound HTTP status (httpx), control ops (retry/abort/pause/resume), and one failure line per park naming why + where —VALIDATING: INTAKE_VALIDATION_FAILED [USER_FIXABLE] -> parked (AWAITING_USER_FIX): <reason + field errors>. Upstream failures with no httpx status line (transport timeouts, the Tachyon/LLM path, MongoDB) log their ownWARNING. Keep the level atINFOfor the full trace:WARNINGdrops transitions/HTTP/control context, andERRORhides the park lines (they areWARNING), so don't raise it aboveWARNINGif you want to see why sessions park. - Recovery sweep runs at every startup (orphan-check dangling intents, release expired writer
locks, auto-resume Jenkins-only
POLLING_CI). - Run → session correlation: Jenkins via the
DDA_RUN_IDbuild parameter; GitHub Actions viahead_sha== commit sha →commit_auditsbycommitSha; any session event onaudit_events. - Config changes: templates via
scripts/seed_templates.py, Jenkins owner→node rows viascripts/seed_jenkins_nodes.py, releases viascripts/seed_releases.py(all validate-first, idempotent upserts); pattern/type-mapping/org-API config changes require a restart (validated at boot; a broken file refuses to boot).
Known limitations (all visible in the code): tachyon-sdk is org-internal (lazy import, not a
pyproject.toml dependency; no config ⇒ llm=None + deterministic fallbacks everywhere) ·
single-instance design (OAuth nonces / credential broker are in-process — run one instance or
use sticky sessions and accept credential re-prompts on failover) · Jenkins multibranch
param drop (a /build fallback drops DDA_RUN_ID, so such builds can't be correlated) · GHA run
discovery is heuristic (path + head_sha + t0−60 s window + journal claim) · spoofable identity
header (v1 auth) · MongoDB < 6.0 (the three $in partial-unique indexes fall back to app-level
guards) · the Gate-2 Jenkins preflight is resolver-only (full existence check is at trigger time).
Category legend: AGENT_FIXABLE = automatic retry with backoff (escalates after 3) ·
USER_FIXABLE = park in AWAITING_USER_FIX · ONBOARDING_REQUIRED = park in
BLOCKED_ONBOARDING · UNFIXABLE → FAILED.
| Code | Category | Meaning |
|---|---|---|
AGENT_ERROR |
UNFIXABLE | Base/unspecified agent failure. |
CONFIG_INVALID |
UNFIXABLE | Broken settings or config file (usually refuses boot). |
DIVA_AUTH_FAILED |
ONBOARDING_REQUIRED | The second chat agent rejected this deployment's credentials; questions about the session itself still work. |
EXTERNAL_SERVICE |
AGENT_FIXABLE | Generic external-service failure (ambiguous; INTENT survives). |
DEPENDENCY_UNAVAILABLE |
AGENT_FIXABLE | A dependency's circuit is open after repeated faults; calls are refused until the reset window elapses. |
JIRA_ERROR / GITHUB_ERROR / CI_ERROR / MONGO_ERROR / LLM_UNAVAILABLE |
AGENT_FIXABLE | Transient dependency failures. |
LLM_AUTH_FAILED / LLM_RATE_LIMITED |
AGENT_FIXABLE | Typed LLM 401-403 / 429 — still LLMErrors, so optional-call fallbacks keep degrading deterministically. |
BRANCH_EXISTS |
AGENT_FIXABLE | POST git/refs 422 — adopt vs rename. |
COMMIT_CONFLICT |
AGENT_FIXABLE | Non-fast-forward — head re-read + one rebuild. |
RATE_LIMITED |
AGENT_FIXABLE | 429; backoff honors Retry-After. |
STATE_CONFLICT |
AGENT_FIXABLE | Optimistic-lock race — re-read and re-decide. |
DUPLICATE_EFFECT |
AGENT_FIXABLE | INTENT hit without an orphan check — run recovery. |
JENKINS_JOB_NOT_READY / GHA_RUN_NOT_FOUND |
AGENT_FIXABLE | CI discovery/readiness timeouts. |
EXTERNAL_TOOL_FAILED |
AGENT_FIXABLE | An artifact-producing org-system API failed. |
PENDING_CARD_LOST |
USER_FIXABLE | A WAIT state lost its pending card — Retry re-runs the stage that rebuilds it. |
BRANCH_DIVERGED |
USER_FIXABLE | The deterministic branch exists but a human pushed to it. |
BRANCH_PROTECTED |
USER_FIXABLE | Branch protection rejected the ref update — deterministic, never retried. |
COMMIT_FILE_TOO_LARGE |
USER_FIXABLE | A blob POST was rejected (422) — the file exceeds GitHub's blob limit. |
GITHUB_SSO_REQUIRED |
USER_FIXABLE | 403 with X-GitHub-SSO — authorize the token for the org's SSO. |
GHA_WORKFLOW_NOT_FOUND |
ONBOARDING_REQUIRED | workflow_dispatch 404 — the workflow file is absent on the target ref. |
GHA_DISPATCH_REJECTED |
USER_FIXABLE | workflow_dispatch 422 — no dispatch trigger / bad inputs / invalid ref. |
JENKINS_AUTH_FAILED |
ONBOARDING_REQUIRED | Jenkins rejected the service credentials (401 / post-crumb 403). |
API_CONFLICT |
USER_FIXABLE | Existing system reports a conflicting/duplicate record (409). |
API_ENDPOINT_NOT_FOUND |
ONBOARDING_REQUIRED | A write hit a 404 — the configured org-system endpoint does not exist. |
API_RESOURCE_NOT_FOUND |
USER_FIXABLE | A GET hit a 404 — the org system has no record for the addressed id. |
REGISTRATION_FAILED |
USER_FIXABLE | Some rows failed org-system registration while the rest completed; the decision card offers continue-partial (discard the failed rows) vs fix-and-retry. |
RECORDS_FAILED |
USER_FIXABLE | Some rows failed the org-system record calls while the rest completed; same continue-partial vs fix-and-retry decision card. |
CI_TIMEOUT |
USER_FIXABLE | A CI polling budget exhausted — parks naming the job. |
CRED_REQUIRED / CRED_INVALID |
USER_FIXABLE | Credential needed / rejected — parks in the matching credential wait. |
VALIDATION_FAILED |
USER_FIXABLE | Requirement field validation failed (per-field errors). |
TEMPLATE_RENDER_FAILED |
USER_FIXABLE | Placeholder rendering failed (all failures listed). |
TEMPLATE_NOT_FOUND |
ONBOARDING_REQUIRED | Template missing/inactive — seed it. |
TYPE_MAPPING_FAILED |
USER_FIXABLE | Source type has no target mapping (and no default). |
COLUMN_SHEET_INVALID |
USER_FIXABLE | Column-details workbook unreadable per config. |
RUNBOOK_AMBIGUOUS |
USER_FIXABLE | A runbook was requested without naming a requirement on a session that has several; the response lists the ids to choose from. |
DOCUMENT_TOO_LARGE |
UNFIXABLE | A proxied document exceeded the size the agent will hold in memory; refused deterministically rather than retried. |
REPO_ADDRESSING |
USER_FIXABLE | An ordinal-addressed pattern has a skipped or unbound repository slot; retry resumes at the repository plan. |
ARTIFACT_UNSAFE |
USER_FIXABLE | An artifact's path or content failed the pre-commit safety rules (traversal, CI-control path, credential-shaped literal). |
ARTIFACT_PATH_CONFLICT |
USER_FIXABLE | Two live artifacts render to the same repo path. |
API_VALIDATION_REJECTED |
USER_FIXABLE | Existing-system API rejected the payload (message passed through). |
INTAKE_VALIDATION_FAILED |
USER_FIXABLE | The org validation API failed intake keys not editable on the form (Jira identity / release / feature) — fix them at the source, then retry. |
INTAKE_ONBOARDING_REQUIRED |
ONBOARDING_REQUIRED | Every failed intake key is a value the org system has not onboarded — no card can change those, so the session parks as blocked on another team. Onboard the values, then Retry: the session resumes at VALIDATING and re-reads the anchor. |
JIRA_ISSUE_TYPE_UNAVAILABLE |
ONBOARDING_REQUIRED | The anchor's project does not expose the configured child issue type via createmeta (type missing there, or invisible to the service account). |
JIRA_NOT_FOUND / JIRA_NO_XLSM_ATTACHMENT / ATTACHMENT_TOO_LARGE |
USER_FIXABLE | Intake input problems. |
JIRA_PERMISSION_DENIED |
USER_FIXABLE | User write got 403 with a valid PAT — permission wall (issue state / project role), not a credential problem. |
DESC_FORMAT_UNPARSEABLE |
USER_FIXABLE | Mode-B: codec and LLM fallback both failed. |
FIXVERSION_MISSING |
USER_FIXABLE | The anchor Jira has no fixVersion. |
RELEASE_DATE_MISSING |
USER_FIXABLE | The chosen fixVersion has no release date set in Jira (retry re-reads Jira live). |
JIRA_ALREADY_ONBOARDED |
USER_FIXABLE | A registry-active record already exists for the business key. |
JIRA_VERIFY_MISMATCH |
USER_FIXABLE | Post-create round-trip verification found unfixable diffs. |
JIRA_CHILDREN_FAILED |
USER_FIXABLE | Some child-issue rows failed while the rest completed; the decision card offers continue-partial (discard the failed rows) vs fix-and-retry. |
WORKBOOK_PARSE_FAILED |
USER_FIXABLE | .xlsm parsing failed. |
ILLEGAL_EVENT / STALE_CHECKPOINT / AUTH_FAILED |
USER_FIXABLE | Protocol-level rejections. |
RUNBOOK_UNAVAILABLE |
USER_FIXABLE | The runbook was requested before the session reached COMPLETED. |
ONBOARDING_REQUIRED / JIRA_AUTH_FAILED |
ONBOARDING_REQUIRED | Generic / Jira service-account auth. |
REPO_NOT_FOUND / REPO_ACCESS_DENIED |
ONBOARDING_REQUIRED | Missing/invisible repo (404) / real permission wall (403). |
REPO_ORDER_CONFLICT / REPO_CI_CONFLICT |
ONBOARDING_REQUIRED | Contradictory cross-pattern slot order / CI config. |
JIRA_CREATEMETA_MISSING_FIELD |
ONBOARDING_REQUIRED | createmeta requires a field absent from DDA_JIRA__CUSTOMFIELDS. |
JENKINS_NODE_UNKNOWN |
ONBOARDING_REQUIRED | No Jenkins node mapped for the repo owner. |
SESSION_CORRUPT |
UNFIXABLE | The persisted session document no longer deserializes — quarantined behind a typed error. |
UNCLASSIFIED_FAILURE |
(classified at runtime) | An uncategorized non-AgentError; classified once via the LLM, else USER_FIXABLE (parks for a human). |
WebSocket transport error codes — these ride the error envelope (ephemeral, per
connection), NOT session.error; each carries catalog userMessage text:
| Code | Meaning |
|---|---|
INVALID_ENVELOPE |
Frame is not valid JSON, fails envelope validation, or the sessionId mismatches. |
UNSUPPORTED_PROTOCOL_VERSION |
Client protocol version is not supported. |
UNSUPPORTED_TYPE |
Message type is not accepted on this socket. |
SESSION_LOCKED |
Another connection holds the writer seat. |
CAPABILITY_REQUIRED |
The action needs a capability this connection lacks. |
INTERNAL |
Unexpected server error while handling the frame. |
NOT_AUTHENTICATED |
A frame arrived before the connection's opening auth frame. |
TOO_MANY_READS |
This connection already has the maximum reads in flight. |
Request/response error codes — these ride a response envelope's
payload.error (never the error envelope) and answer exactly one request:
| Code | Meaning |
|---|---|
UNKNOWN_OP |
The named op is not implemented by this server. |
BAD_REQUEST |
The request envelope carried no usable op. |
NOT_FOUND |
The read named something this session no longer has (e.g. a superseded gate checkpoint). |
RESPONSE_TOO_LARGE |
The result exceeds one frame; ask for a smaller page. |
The DDA Agent reuses this project's parser (
excel-form/build/parse_output.py/config_loader.py) for Mode A intake. The exhaustive field/control/rules reference lives in docs/add-pattern.md and docs/field-config.md; the authoritative source of truth is the code (excel-form/build/config_validate.py/config_loader.pyand the VBA), since the docs can drift.
A self-contained, macro-enabled Excel workbook (.xlsm) that renders the CDM-Next Prework
data-movement intake form. Every pattern, section, field, option list and validation rule is
authored in JSON — no code changes to add or modify fields. The workbook runs fully offline:
validation and record storage happen inside the workbook. An optional backend path (HTTP
contract, Node mock server, Spring Boot reference, distribution portal) serves live option lists and
distributes the file. It runs in Windows desktop Excel only (not Mac/web/mobile); the build is
pure Python.
The hidden Config / Lists / Settings sheets are the stable contract between the Python build and
the VBA runtime. Per-pattern output sheets (Req-*) are created lazily by VBA on first submit, then
hidden and password-protected.
All scripts are pure Python (no Excel/Office needed) unless noted. Two build env vars: CDM_ENV
(environment to bake; overridden by --env) and CDM_NOLOCK=1 (skip VBA-project locking for
debugging).
| Script | Purpose |
|---|---|
build_template.py |
Stage 1 (only when VBA changes): injects src/vba/ modules into build/template.xlsm via pyOpenVBA and zeroes the p-code cache. |
build_workbook.py [--env <name>] |
Stage 2 (main build): validates config, lays the Form + hidden sheets over template.xlsm, locks the VBA project, emits dist/CDM-Next-Prework.xlsm + dist/config.bundle.json. Env precedence: --env > CDM_ENV > settings.json "environment" > dev. |
config_loader.py / config_validate.py |
Libraries (no CLI): normalize the authoring files; fail the build on any construct the VBA validator couldn't enforce. |
lock_vba.py <path> |
Makes the VBA project "unviewable" (deterrence only; no password; reversible). Called by stage 2 unless CDM_NOLOCK=1. |
verify_xlsm.py [path] |
Structural gate: valid macro package, all module streams present, p-code cache zeroed, data sheets present, no Req* sheet pre-built. Exit 0 pass / 1 not found / 2 failed. |
parse_output.py [path] [-o out.json] [--strict] |
Reads merged-block records out of every output sheet into JSON (schema-driven from the embedded Config sheet). Multiselects → string arrays, grids → arrays of row objects, attachments inlined as {fileName, mimeType, sizeBytes, base64}. Records exist only if the workbook was saved after submitting. |
Authoring lives in excel-form/src/config/: settings.json (global
runtime settings), shared-lists.json (option lists reused across patterns), and one
patterns/<id>.json per pattern (sections, fields, grids, pattern-local lists). Rebuild after any
edit with build_workbook.py; the build validates the whole config first and fails with a bulleted
message.
settings.json— keys written to the hidden Settings sheet:UseMock(the single online/offline switch for option lists — validation/submission are always offline),DefaultPattern,WfClientId(thex-wf-client-idheader, defaultutcap),OutputPassword/FormPassword(change before distributing — they also appear indist/config.bundle.json),HideSheets, plusenvironment+ anenvironmentsmap (exactly one base URL is baked in per build, stored asApiBaseUrl).- Field object —
key(camelCase; becomes the JSON key + named rangefld_<key>),label,control(one oftext,email,number,date,time,dropdown,multiselect,readonly,grid,attachment),source(none|static|api|dependent|attachment),list/from/column/dependsOn,required,visibleWhen,dataType,default,help,optionsPath,rules. visibleWhengrammar:always·Key=Value/Key!=Value·Key in [A,B]/Key notin [A,B]·Key contains Value·Key empty/Key notempty; combine with&&(AND, binds tighter) and||(OR-of-AND-groups). No other operators.rules(enforced offline by VBA, live as you fill):pattern(+patternMsg; must beVBScript.RegExp-parseable — no lookahead/lookbehind/inline flags/named groups),min/max,integer,multiple(email only),lessThan/greaterThan(same-type cross-field),accept/maxSizeKb/content(attachment only). Rule/control fit is enforced at build time.- Grids (
section.grid) — a repeating sub-table (≤ 5 columns; columns restricted totextanddropdown); read back as an array of row objects. - Attachments —
rules.acceptis mandatory; bytes are stored base64-chunked in the hidden Attachments sheet; an optionalrules.contentobject validates the file's content at attach time (sheet, exact header row, per-row field matches, required columns, min data rows) and can feed option lists intosource: "attachment"fields.
See docs/add-pattern.md for the long-form field reference and docs/field-config.md for copy-paste recipes.
On open (macros enabled) the workbook renders the form for the current pattern: a frozen header with
a live progress indicator, a left Sections rail with per-section badges, and label/input/hint
rows. Validate & Submit runs the full offline rule engine over visible fields; on failure it
lists issues and writes nothing; on success it appends the record to the pattern's Req-* sheet as
a merged block (scalars merge vertically; each multiselect value and grid row gets its own row) and
flushes staged attachments. Submissions persist only when the workbook is saved. A floating
RECORDS panel previews/deletes submitted records.
The only network calls the form ever makes are option-list fetches
(GET {ApiBaseUrl}/options/…) for api/dependent fields when UseMock=false; each carries four
headers (x-correlation-id, x-request-id, x-wf-request-date, x-wf-client-id). The VBA is
organized into ~23 modules (ThisWorkbook.cls, modState, modConfig, modExpr, modVisibility,
modDepend, modApi, modRender, modRail, modMultiSelect, modPickPanel, modGrid,
modValidate, modSubmit, modRecords, modAttachment, modContent, …).
Worksheet protection and VBA locking are deterrence, not security. The Form sheet is
password-protected (FormPassword, UserInterfaceOnly so VBA can write); Config/Lists/Settings are
veryHidden with no password (hiding is their only shield — Settings is intentionally writable so
tooling can flip settings, which is why the stored passwords are not real secrets); Req-* and
Attachments sheets are veryHidden + fully password-protected (OutputPassword); the VBA project is
made "unviewable" by lock_vba.py. Change OutputPassword and FormPassword before
distributing — both also appear in plaintext in dist/config.bundle.json.
Files downloaded from a browser carry Mark of the Web; users Unblock the file (right-click →
Properties → Unblock, or Unblock-File) and enable macros. See
docs/customer-enable-macros.md; run
tools/Check-FormReadiness.ps1 on a customer machine first.
- Mock server (mock-server/) — zero-dependency Node (≥ 18) implementation of the
Excel-form HTTP contract against a local file store; also serves the portal.
node mock-server/server.js(ornpm start) →http://localhost:8080(API/api/v1, portal/). Requiresexcel-form/dist/config.bundle.json; reads onlyPORT(default8080). Enforces the fourx-*headers, resolves dependent lists, implements the canonical validation rules, and answers CORS*. - Portal (portal/) — a static, framework-free page listing published workbooks (name,
version, size, SHA-256, download). Calls the API at the relative path
/api/v1, so it must be served from the API origin (the mock server in dev). - HTTP contract (api/contracts.md, api/openapi.yaml) —
the canonical Excel-form backend contract (no auth):
GET /options/{listName}(+ dependent form),POST /validate, and the form-distribution endpoints (GET/POST /forms,GET /forms/{id}/metadata,GET /forms/{id}/download). - Java reference (api/java/, api/java/README-java.md) — a
copy-in Spring Boot reference implementing the contract with MongoDB GridFS storage
(docs/storage-mongodb.md). Re-copy
dist/config.bundle.jsonafter every build so backend rules stay in lock-step; upload a workbook viaPOST /api/v1/formsto populate the portal. - Readiness tool (tools/Check-FormReadiness.ps1) — a read-only PowerShell pre-flight that checks a customer machine (desktop Excel, macro policy, Mark-of-the-Web, required COM components, Defender ASR, optional API reachability). Exit 0 ready / 1 caveats / 2 not ready.
DDA Agent — all backend suites run from dda-agent/ (pytest rootdir; asyncio_mode = auto):
cd dda-agent
backend\.venv\Scripts\python -m pytest # everything
backend\.venv\Scripts\python -m pytest tests\unit # fast unit suite
backend\.venv\Scripts\python -m pytest tests\e2e # REST + WS end-to-end flows (in-memory container)
- Unit (
tests/unit/) — pure Python against in-memory adapters + port doubles;mongomock-motorcovers the Mongo query logic. - Contract (
tests/contract/) — adapters againstrespx-mocked HTTP, plustest_validator_vs_mock_server.py, which starts the real Node mock server againstexcel-form/dist/config.bundle.jsonand diffs the local validator againstPOST /validate(needsnode+ the built bundle, else skips). - Integration (
tests/integration/) — needs a real MongoDB atmongodb://localhost:27017(1 s ping; skips without it); covers GridFS + real partial-unique index enforcement. - Frontend —
cd dda-agent/frontend && npm test(Vitest, jsdom, Testing Library). - Gates —
ruff check .,flake8 backend/src(cognitive complexity CCR001 ≤ 15), andpyright(basic, 0 errors, run from the repo root);mypyis advisory. See §3.1 for themaketargets.
CDM-Next Prework Excel form — four standalone scripts under
excel-form/test/ (each prints PASS/FAIL + RESULT: n/m; not pytest):
config_validate_test.py (build gate, Python only), parser_guards_test.py (parse_output.py
guards, Python + openpyxl), and the two COM end-to-end tests com_e2e_test.py (~140 checks) and
multiblock_parse_test.py (Windows + desktop Excel + pywin32 + a built dist).
COM-test prerequisite: build the workbook with
UseMock: trueso api-sourced dropdowns have mock-seeded lists. Set it insettings.json, rebuild, run the tests, then restoreUseMock: false/HideSheets: trueand rebuild before distributing. No mock server or network is needed by any test.
| Term | Meaning |
|---|---|
| Pattern | A data-movement onboarding shape defined in pattern-behaviors.json (repo slots, artifact tools, column-details reading, description section). The requirement's dataMovementPattern selects it. |
| Slot | A pattern-defined repository role (position, label, platform, artifactTypes, optional CI). The user binds each slot to a concrete org/repo at Gate 2; identical bindings merge into one plan entry. |
| Requirement | One onboarding unit — a workbook row (mode A) or the single Jira (mode B). |
| drtId / anchor key | drtId is the requirement's business id (part of the business key drtId + targetSchema + targetTableName). The anchor key is the session's root Jira (master in A, the requirement issue in B); it scopes idempotency labels, branch names and the one-live-session-per-anchor guard. |
| Gate | A human approval checkpoint (four of them) delivered as an approval_request with a checkpointId; answers must quote the current checkpoint (else STALE_CHECKPOINT). |
| Effect journal | The side_effects collection: one row per external side effect under a deterministic idempotency key, driven through intent → call → commit — the exactly-once mechanism. |
| Fingerprint | inputsFingerprint on an artifact: a hash of everything that determines its bytes; unchanged ⇒ regeneration is skipped; a field edit changes it, forcing regeneration (supersedes lineage). |
| Capability | The per-session secret returned once at creation (sha256 stored). Possession = the right to drive the session. |
| Writer / observer | WS roles: one capability-backed writer (holds the heartbeat lock) plus any number of read-only observers; take_over re-seats. |
| Mode A / Mode B | MASTER_EXCEL (master Jira + .xlsm, many requirements, child Jiras created) vs SINGLE_JIRA (one pre-created requirement Jira, description-parsed). |
| Park | A wait state expressing "a human must act" (AWAITING_USER_FIX / BLOCKED_ONBOARDING) with resumeTo. |
| Adoption | Recognizing an external effect already exists and taking it over instead of recreating it (labeled child Jiras, existing dda/… branches, discovered GHA runs, journaled results after a crash). |
{ "type": "auth", "payload": { "headers": { "X-User-Id": "..." }, // FALLBACK only — a real handshake header wins "capability": "...", // optional; absent is fine, WRONG closes 4003 "orgToken": "...", // optional org-framework bearer; stripped at ingress "lastSeq": 12, // replay cursor "role": "observer" // optional: watch without taking the writer seat } }