persistent_stores_design

Persistent Stores Design

Status note — KB surface simplified to notes-only (2026). This doc was written when the knowledgebase had two surfaces (structured graph.json + per-topic notes). The graph surface has been removed; the knowledgebase is now notes-only — per-topic markdown files under knowledgebase/notes/ plus a notes/_index.json registry. Any section below that references graph.json, index.json, entities, relationships, kb_upsert_*, or kb_shape: "graph+notes" describes the earlier design and does not reflect current behavior. The code of truth is:

pkg/orchestrator/agents/workflow/step_based_workflow/kb_graph.go — only seeds notes/_index.json now.

pkg/orchestrator/agents/workflow/step_based_workflow/kb_update_agent.go — prompts are notes-only.

pkg/orchestrator/agents/workflow/step_based_workflow/kb_tools.go — holds BuildStepKBGuidance / BuildKBContributionReviewMessage for direct-write mode (shell + diff_patch_workspace_file under notes/).

pkg/workflowtypes/types.go — ResolveKBShape always returns "notes-only"; legacy "graph+notes" values collapse at load.

Direct-write mode is a per-step option (knowledgebase_write_method: "direct") in which the step agent writes to notes/ inline via a post-completion self-review turn. Agent mode (default) keeps using the post-step KB update agent, which now also targets only notes/.

This document captures the planned improvements to how workflows manage persistent data, report output, and knowledgebase access control.

Background: Current Stores

Workflows currently have three persistent stores at the workspace root:

Store	Location	Written by	Purpose
Learnings / Skills	`learnings/_global/`	Learning agent (automatic)	HOW to execute — patterns, selectors, auth flows
Knowledgebase	`knowledgebase/`	Any step agent	Domain facts + doubles as persistent data store
Reports	`reports/{group}/`	Report agent	Human-readable run summary

The tension: knowledgebase is doing two jobs — storing domain context to help agents make better decisions, and acting as a database for structured data that needs to survive across runs. These have different access patterns and formats and should be separated.

1. New `db/` Folder

What it is

A dedicated folder for structured persistent data in JSON format, separate from the knowledgebase. Steps use it to store state, results, or any data that needs to survive across runs and be readable by later steps or other groups.

Folder structure

workspace/
├── db/               ← new, always enabled, JSON files only
├── knowledgebase/    ← graph-structured domain knowledge (see section 4)
├── learnings/        ← procedural skill knowledge (HOW to run)
├── planning/         ← plan.json, step_config.json (read-only to builder via guard; mutated via plan-mod tools only)
├── reports/          ← report_plan.md (widget definitions; builder-writable)
├── soul/             ← new, builder's long-term memory: soul.md (see section 8)
└── runs/

Note: reports/ no longer exists — the report is a live frontend view over db/ + KB rather than a static archive (see section 2).

Behaviour

Always created on workspace init (no toggle, unlike knowledgebase)
All steps get read + write access by default
JSON format only — per-file schema is defined by the builder at design time (see Write semantics below); no runtime enforcement
Never deleted during cleanup
Injected into execution agent prompt as an absolute path

Write semantics (single-db model)

There is exactly one db/ per workspace — not partitioned by group or iteration. All groups and all reruns write into the same files. This keeps the report simple (one live view) but makes schema design a hard requirement, not a convention.

Builder responsibility (design time). When the interactive builder defines a db/ file, it must decide and record:

Primary key — what uniquely identifies a row (company_id, url, composite)
Group coexistence — do multiple groups share the file (then include a group field and key by (group, id)) or does each group get its own file (db/companies_{group}.json)?
Merge rule — upsert by key (default), append-only, or replace

These decisions go into a short schema note per file — inline in report_plan.md near the widget that consumes it, or a dedicated reports/db_schema.md. Step agents read the note at execution time so they know the contract.

Step agent responsibility (runtime). Agents MUST NOT overwrite db/ files wholesale. The execution prompt instructs them to: read the file, upsert by the builder-defined key (or append, per the schema note), then write back the merged content.

For reliability a typed helper (db_upsert(file, key, rows)) is preferable to raw shell redirection — LLMs under pressure will happily cat > file.json with only their own rows and wipe the other groups' data. Leaving writes as shell + prompt is workable short-term but will produce lost-row bugs under stress. Without explicit schemas and write discipline, any workflow with more than one group or rerun will silently clobber prior data the first time two paths share a file.

Key implementation touchpoints

File	Change
`controller_execution.go`	Add `DBFolderName = "db"` constant + `getDBPath()`
`controller_run_manager.go`	Always create `db/` on init (no `UseKnowledgebase`-style gate)
`controller_agent_factory.go:397`	Add `db/` to read+write in `setupExecutionFolderGuard`
`controller_agent_factory.go:806`	Same for orchestrator agent guard
`controller_todo_task.go:91`	Add `db/` to read+write paths
`execution_only_agent.go`	Tell agents about `db/` path and JSON convention
`todo_task_orchestrator_agent.go`	Same
`pre_validation.go:1080`	Add `"db"` to known workflow folder names
~~`evaluation_types.go`~~ (retired)	~~Add `DBWrite bool` to `EvaluationStep`~~
~~`controller_agent_factory.go:800`~~ (retired)	~~Evaluation folder guard~~
Workflow builder prompt	Recommend a dedicated subfolder for measurement outputs

Evaluation runs and `db/` (retired — eval subsystem removed; kept as history)

Evaluation steps can read and write db/ — write access is opt-in per step via a DBWrite flag. There is no hard folder-guard split between regular and eval writes; the folder guard grants read and (optionally) write on all of db/. How db/ is organized is up to the plan author.

Best practice (builder prompt guidance, not enforcement)

The workflow interactive builder prompt should recommend — but not enforce — that eval output goes to a dedicated namespace so it doesn't clobber regular-step data that the live report displays:

Suggest db/eval/ as the default subfolder for eval writes
Suggest JSON or markdown under that path (narrative score summaries, structured scores, etc.)
Suggest that regular db/ files and eval output files have disjoint names

If a plan deliberately wants evaluation output mixed into the same db/ files as regular steps (e.g. the eval step is producing the canonical data, not just scoring it), the builder lets that through. The tradeoff is on the plan author: enabling DBWrite on an evaluation step means that step's output is free to mutate whatever db/ files it touches, including ones the live report reads. The builder should flag this when enabling the toggle.

Add a db_write boolean to EvaluationStep in evaluation_types.go:

type EvaluationStep struct {
    ID              string
    Title           string
    Description     string
    PreValidation   *ValidationSchema
    SuccessCriteria string
    ContextOutput   string
    DBWrite         bool `json:"db_write,omitempty"` // If true, evaluation step can write to db/
}

Folder guard (controller_agent_factory.go:800):

Read: always add db/ to read paths
Write: add db/ to write paths only if DBWrite: true on the evaluation step

Open questions

Should db/ writes be blocked from learn-code scripts (similar to the knowledgebase/learnings/ check in controller_learn_code.go:457)?

2. Dynamic Report System

What changes from current

	Current	Proposed
Report source	Raw execution artifacts from `runs/{runFolder}/execution/`	`db/` files + `knowledgebase/graph.json` (workspace-scoped)
Generation	Report agent runs once, produces static markdown	No agent — frontend renders live from `report_plan.md`
Storage	`reports/{group}/{timestamp}.md` archive	No archive — always reflects current `db/` state
Definition	`planning/output_plan.json` (instructions for the agent)	`reports/report_plan.md` (markdown widget definitions)
If no data	Empty report with headings	Report not shown at all

`report_plan.md` format

Stored at reports/report_plan.md. Created and maintained by the workflow interactive builder.

## headings define sections. Fenced blocks define widgets. A widget:row block groups widgets side-by-side. Standalone widget blocks are full-width.

## Overview

```widget:text
source: db/summary.json
path: description
```

```widget:row
- chart | source: db/results.json | path: counts
- chart | source: db/monthly.json | path: totals
```

## Companies Found

```widget:table
source: knowledgebase/graph.json
path: entities
filter: type=company
```

Widget types

Three types only:

Block	Description	Data shape at `path`
`widget:text`	Markdown text block	String value
`widget:chart`	Bar or line chart	`[{ "label": string, "value": number }]`
`widget:table`	Data table — columns from object keys	Array of objects
`widget:row`	Layout wrapper — groups widgets side-by-side	—

Inside widget:row, each line is - {type} | source: {path} | path: {key}.

Widget fields

Field	Required	Description
`source`	yes	`db/{file}.json` or `knowledgebase/graph.json`
`path`	yes	Dot-notation key into the JSON file
`filter`	no	`key=value` — filters array items (useful for KB entities)

Data sources

Source	What it points to
`db/{file}.json`	Any JSON file in the workspace-level `db/` folder
`knowledgebase/graph.json`	KB graph — entities and relationships

Both are workspace-scoped. The report always reflects the latest state.

Widgets can point at any path under db/ — including eval outputs if the plan writes them under db/eval/ (the recommended convention, see section 1). The report does not distinguish "regular" vs "eval" data automatically; if the builder wants an Evaluation section in the report, it adds widgets pointed at the eval files explicitly.

If db/ has no data yet, the report is not shown — no empty state, no placeholder widgets.

How the frontend renders it

Fetch and parse reports/report_plan.md
Split on ## headings → sections
For each fenced block, parse type + fields
For widget:row, parse the inline widget list
For each widget, fetch source file, walk path, apply filter
If any required source file is missing or resolved data is empty → do not render the report
Render each widget using its type component

Who creates `report_plan.md`

The workflow interactive builder (chat phase). The user describes what they want to see; the builder agent writes the markdown document based on what steps write to db/ and what the KB captures. Since the builder knows the step plan, it can infer the right source paths.

Report is not triggered by workflow execution

The report is no longer a phase that runs after execution completes. It is purely a frontend view — opened on demand. db/ is populated during normal step execution and evaluation runs, so the report data is always there when the user opens it.

MaybeRunAutoFinalOutput() is removed — no post-run auto-trigger
The "Final Report" button in the toolbar no longer starts a phase — it just opens the report viewer
The report viewer checks whether db/ has data; if not, it shows nothing

What goes away

final_output.go — report agent removed entirely
planning/output_plan.json — replaced by reports/report_plan.md
reports/ folder — no more archive files
WorkflowStatusReportExecution = "report-execution" phase constant
runReportExecutionOnly() in workflow_orchestrator.go
MaybeRunAutoFinalOutput() — no post-run auto-trigger
handleGetFinalOutputs, handleGenerateFinalOutput, handleGetFinalOutputConfig, handleUpdateFinalOutputConfig HTTP handlers
FinalOutputPopup.tsx — replaced by dynamic report viewer

Key implementation touchpoints

File	Change
New: `frontend/src/components/workflow/ReportViewer.tsx`	Parses `report_plan.md`, resolves sources, renders widgets
New: `frontend/src/components/workflow/widgets/TableWidget.tsx`	Table renderer
New: `frontend/src/components/workflow/widgets/TextWidget.tsx`	Markdown text renderer
New: `frontend/src/components/workflow/widgets/ChartWidget.tsx`	Bar/line chart (reuses existing SVG chart logic from `MarkdownRenderer.tsx`)
`frontend/src/services/api.ts`	Add `getReportPlan()`, `getDBFile()` API calls
`frontend/src/services/api-types.ts`	Add `ParsedReportPlan`, `ReportSection`, `ReportWidget` types
`WorkflowToolbar.tsx`	"Report" button opens `ReportViewer` (no phase start)
`cmd/server/workflow.go`	Add `handleGetReportPlan()`, `handleGetDBFile()` handlers; remove old final-output handlers
`workflowtypes/types.go`	Remove `WorkflowStatusReportExecution`
`workflow_orchestrator.go`	Remove `runReportExecutionOnly()` routing
Delete: `final_output.go`	Report agent no longer needed

3. Step-Level Knowledgebase Access Control

What it is

Replace the binary disable_knowledgebase per-step flag with an explicit access mode: "read", "write", "read-write", or "none". The folder guard enforces the mode.

Current state

UseKnowledgebase bool at preset/workflow level
DisableKnowledgebase *bool in AgentConfigs — only disables, can't make a step read-only

Proposed: `knowledgebase_access` + `knowledgebase_contribution` fields

// planning/step_config.json
{
  "steps": [
    {
      "step_id": "step-1",
      "agent_configs": {
        "knowledgebase_access": "read"
      }
    },
    {
      "step_id": "step-2",
      "agent_configs": {
        "knowledgebase_access": "read-write",
        "knowledgebase_contribution": "Extract company names, industries, and decision-maker contacts from the scraped profiles. Focus on companies with 50-200 employees in the SaaS space."
      }
    }
  ]
}

knowledgebase_contribution is a plain string instruction that tells the KB update agent what to extract from this step's output and how to represent it in the graph. It is:

Only relevant when knowledgebase_access is "write" or "read-write"
Set by the user in the workflow interactive builder (not auto-generated)
The trigger for the KB update agent — if not set, the KB update agent does not run for that step even if write access is granted

Valid values:

Value	Read	Write	Notes
`"read-write"`	yes	yes	Default when knowledgebase is enabled
`"read"`	yes	no	Step can consume but not modify
`"write"`	no	yes	Step can append/update but not read
`"none"`	no	no	Equivalent to `disable_knowledgebase: true`

DisableKnowledgebase *bool is kept for backward compatibility and maps to "none" / "read-write".

Resolution logic

resolveKnowledgebaseAccess(stepConfig, presetEnabled):
  1. knowledgebase_access field set → use it directly
  2. disable_knowledgebase: true    → "none"
  3. disable_knowledgebase: false   → "read-write"
  4. preset enabled                 → "read-write"
  5. preset disabled                → "none"

Key implementation touchpoints

File	Change
`planning_agent.go:227`	Add `KnowledgebaseAccess string` and `KnowledgebaseContribution string` to `AgentConfigs`; keep `DisableKnowledgebase` as deprecated
`controller_execution.go:1065`	Replace bool resolution with `resolveKnowledgebaseAccess()` helper
`controller_agent_factory.go:397`	`setupExecutionFolderGuard` takes `kbAccess string` instead of `...bool`; adds to read/write paths separately
`controller_agent_factory.go:806`	Same for orchestrator agent guard
`controller_learn_code.go:642`	Use same `resolveKnowledgebaseAccess()` helper
`execution_only_agent.go`	Prompt reflects actual access mode (e.g. `READ` vs `READ/WRITE`)
`todo_task_orchestrator_agent.go`	Same

4. Knowledgebase Graph Structure

Motivation

Currently the knowledgebase has no enforced shape — agents write whatever they want. This makes it hard to query, merge across runs, or reason about what's known. A file-based graph structure gives it a fixed, queryable shape without requiring a graph database.

Inspired by graph-based agent memory research (Graphiti, GraphRAG, AGENTiGraph): knowledge is modelled as entities (nodes) and relationships (edges). The basic unit is a triple: (subject, predicate, object). Each fact has provenance (which step/run created it) and a validity window (when it was created, whether it was superseded).

File layout

knowledgebase/
  graph.json    ← source of truth: entities + relationships
  index.json    ← lightweight overview: counts, types, last updated
  raw/          ← unstructured source material if needed

Execution agents get the knowledgebase/ path injected (not the content — same as today). They read what they need via shell:

cat knowledgebase/index.json                                           # quick overview
cat knowledgebase/graph.json | jq '.entities[] | select(.type=="company")'

graph.json serves both consumers: the KB update agent (merge logic) and execution agents (on-demand reads). No separate summary file needed.

`graph.json` schema

{
  "version": "1",
  "updated_at": "2026-04-15T10:00:00Z",
  "entities": [
    {
      "id": "company-acme",
      "type": "company",
      "label": "Acme Corp",
      "properties": {
        "website": "https://acme.com",
        "industry": "SaaS"
      },
      "created_at": "2026-04-15T09:00:00Z",
      "source": { "step": "step-1", "run": "iteration-2/group-a" }
    }
  ],
  "relationships": [
    {
      "id": "rel-001",
      "from": "company-acme",
      "to": "person-john",
      "type": "has_contact",
      "properties": {
        "role": "CEO"
      },
      "created_at": "2026-04-15T09:00:00Z",
      "source": { "step": "step-2", "run": "iteration-2/group-a" }
    }
  ]
}

`index.json` schema

A lightweight summary so agents can decide whether to load the full graph:

{
  "entity_count": 42,
  "relationship_count": 87,
  "entity_types": ["company", "person", "product", "event"],
  "relationship_types": ["has_contact", "owns", "competes_with"],
  "last_updated": "2026-04-15T10:00:00Z",
  "last_updated_by": { "step": "step-3", "run": "iteration-2/group-a" }
}

Key design principles

Agents write via the KB update agent (see section 5) — step execution agents do not write directly to graph.json
Agents read freely — execution agents with knowledgebase_access: read or read-write can shell-read graph.json or index.json directly
No graph DB required — plain JSON files, readable with any shell tool (cat, jq)
Source provenance on every fact — every entity and relationship records which step/run wrote it, enabling auditability and conflict detection
Purely JSON — no markdown files in the knowledgebase; graph.json is the source of truth for both machine merging and LLM reading

5. KB Update Agent (Post-Step)

What it is

A dedicated agent that runs after each step completes (parallel to the success learning agent) and updates knowledgebase/graph.json with facts extracted from that step's output. Mirrors the learning agent pattern exactly.

When it runs

Both the success learning agent and the KB update agent run through single-writer queues — one worker goroutine per queue, draining jobs FIFO. Step completion enqueues and returns immediately; the workers process one agent at a time.

Step execution
    ↓
Validation passes
    ↓
enqueue(learningQueue, job)   → learning worker runs one agent at a time → writes learnings/_global/
enqueue(kbUpdateQueue, job)   → kb worker runs one agent at a time       → writes knowledgebase/graph.json

The two queues are independent — a learning agent can run in parallel with a KB update agent (they write different files), but two learning agents never run concurrently, and two KB update agents never run concurrently.

The KB update agent only enqueues if all three conditions are met:

Knowledgebase is enabled at the preset level
knowledgebase_access for the step is "write" or "read-write"
knowledgebase_contribution is set for the step (non-empty)

If knowledgebase_contribution is not set, the job is skipped entirely — there is no generic fallback extraction.

Serialization (learning and KB writes)

Both learnings/_global/ and knowledgebase/graph.json are workspace-shared files merged from multiple step outputs. Running merges concurrently creates lost-update races. The fix is structural: a single worker per file group, with a buffered channel for incoming jobs.

type KBUpdateJob struct {
    StepID      string
    OutputPath  string
    Contribution string
}

var kbUpdateQueue = make(chan KBUpdateJob, 100)

func startKBUpdateWorker() {
    go func() {
        for job := range kbUpdateQueue {
            runKBUpdateAgent(job) // one at a time, FIFO
        }
    }()
}

Properties this gives:

No merge races — only one goroutine ever reads-modifies-writes graph.json
FIFO by completion order — step-N's update sees the effects of step-(N-1)'s update, enabling cumulative entity/relationship linking instead of duplicate-then-reconcile
Non-blocking on step execution — the enqueue is O(1); step completion never waits for learning or KB work
Full-run KB update shares the queue — the manual kb-update phase (section 6) enqueues onto the same kbUpdateQueue, so it can't race with a live step's post-step update

Two separate queues:

learningQueue → one worker → all learning agents serialized
kbUpdateQueue → one worker → all KB agents (post-step and full-run) serialized

The learning agent today fires in a bare goroutine (controller_learning.go:432) with no serialization. It has the same latent race on learnings/_global/ that the KB agent would have on graph.json; the same queue pattern retrofits cleanly and closes both gaps at once.

File writes inside each worker use write-to-temp-then-rename for crash safety:

tmp := path + ".tmp"
os.WriteFile(tmp, data, 0o644)
os.Rename(tmp, path) // atomic on POSIX

What it reads

Step execution output (runs/{runFolder}/execution/{step-id}/)
knowledgebase_contribution from step config — the user-defined extraction instruction
Current knowledgebase/graph.json (to merge, not overwrite)
knowledgebase/index.json (for context on what's already known)

What it writes

Updated knowledgebase/graph.json — merges new entities and relationships
Updated knowledgebase/index.json — refreshed counts, types, last_updated

Agent instructions

The knowledgebase_contribution string is the primary directive. Fixed rules on top:

Extract WHAT the workflow discovered — entities, facts, relationships found in the output
Do NOT capture HOW the step ran (that's for learnings)
Merge carefully — do not duplicate existing entities, update properties if newer
Record source on every new entity and relationship

Key implementation touchpoints

File	Change
`controller_execution.go:2055`	Enqueue KB update job after validation passes (alongside learning enqueue) — no direct goroutine spawn
New file: `controller_kb_update.go`	KB update orchestration + `kbUpdateQueue` channel and worker (mirrors `controller_learning.go`)
New file: `kb_update_agent.go`	Agent definition and system prompt (mirrors `learning_agent.go`)
`controller_agent_factory.go`	Factory method for KB update agent
`controller_learning.go:432`	Retrofit: replace bare `go func()` with `learningQueue` enqueue; add `learningQueue` channel + worker at package init
Server startup (`cmd/server/server.go` or equivalent)	Start `learningQueue` and `kbUpdateQueue` workers once at process init

Folder guard for KB update agent

Read: runs/{runFolder}/execution/{step-id}/ + knowledgebase/
Write: knowledgebase/ only

6. Global KB Update Tool (Full-Run)

What it is

A manual phase (kb-update) that processes all step outputs from a completed run and rebuilds or enriches the knowledgebase from scratch. Analogous to the report generation phase — triggered from the toolbar, not automatic.

Use cases:

Knowledgebase was disabled during a run, now you want to populate it retrospectively
A run completed before the KB update agent existed
knowledgebase_contribution was updated after the run and you want to re-extract

Per-step knowledgebase_contribution from step_config.json drives the extraction — steps without a contribution defined are skipped, same as the automatic post-step flow.

Flow

User clicks "Update KB" in toolbar → selects run folder
    ↓
POST /api/workflow/knowledgebase/update
    ↓
KBUpdateOrchestrator reads all outputs from runs/{runFolder}/execution/
    ↓
For each step with a non-empty knowledgebase_contribution:
    enqueue(kbUpdateQueue, job)   ← same queue used by post-step updates
    ↓
KB worker drains jobs FIFO — serialized against any live post-step KB updates
    ↓
graph.json + index.json written/merged
    ↓
Frontend: KnowledgebaseView refreshes

Why share the queue: the full-run phase could otherwise race with a still-running workflow's post-step KB updates, producing lost merges. Enqueuing onto kbUpdateQueue means the worker processes manual-phase jobs and live-step jobs in the order they arrive, with no concurrent writers to graph.json. The manual phase does not wait for an empty queue before returning to the user — it enqueues and reports a job count; progress is visible via the usual event stream.

HTTP endpoint

POST /api/workflow/knowledgebase/update
Body: { workspace_path, run_folder, merge_strategy: "merge" | "replace" }

"merge" — adds new entities/relationships, updates existing ones if newer
"replace" — clears knowledgebase and rebuilds from this run's outputs

Key implementation touchpoints

File	Change
`workflowtypes/types.go`	Add `WorkflowStatusKBUpdate = "kb-update"`
`workflow_orchestrator.go`	Route `kb-update` status to `runKBUpdateOnly()`
New file: `kb_update_execution.go`	Full-run KB update logic (mirrors `evaluation_execution.go`)
`cmd/server/workflow.go`	Add `handleKBUpdate()` handler + register route
`cmd/server/server.go`	Register `POST /api/workflow/knowledgebase/update`

7. Interactive Builder Changes

The workflow interactive builder is a chat-based phase (workflow-builder) that lets users modify the workflow plan via natural language. The following UI additions are needed to support the features above.

New: KB Management Popup

A new popup component (KnowledgebaseView.tsx) parallel to LearningsView.tsx and FinalOutputPopup.tsx:

View: renders knowledgebase/index.json summary (entity count, types, relationship types, last updated)
Explore: expandable entity list from graph.json
Run KB Update: triggers the full-run KB update phase (section 6) for the selected run folder
Clear: deletes and recreates knowledgebase/graph.json and index.json
Export: downloads graph.json

New: KB Update toolbar button

In WorkflowToolbar.tsx alongside the existing "Final Report" button:

"Update KB" → opens KnowledgebaseView → user selects run folder → clicks "Run KB Update"
    ↓
handleRunKBUpdate() → onStartPhase('kb-update', { selected_run_folder })

Updated: Step config UI — KB access mode + contribution

In the per-step config panel, replace the existing disable_knowledgebase toggle with:

Knowledgebase access: [ read-write ▼ ]
                       read
                       write
                       read-write
                       none

KB contribution:
┌─────────────────────────────────────────────────────────┐
│ Extract company names, industries, and decision-maker   │
│ contacts from the scraped profiles. Focus on companies  │
│ with 50-200 employees in the SaaS space.                │
└─────────────────────────────────────────────────────────┘

The contribution textarea is only shown when access is "write" or "read-write"
Saving writes both knowledgebase_access and knowledgebase_contribution to step_config.json
The interactive builder (workflow-builder chat phase) can also set these fields via plan modification tools — knowledgebase_contribution should be one of the fields the builder agent can populate when it adds or configures a step

Updated: Preset modal

The existing use_knowledgebase toggle in PresetModal.tsx stays as the workflow-level gate. No change needed here.

New: `db/` schema notes owned by the builder

Because db/ is a single workspace-level store shared by all groups and reruns (see section 1), the builder is the only place where the full picture — which steps write which files, which widgets read them — exists. When the builder adds or updates a step that writes to db/, it must also declare the schema for each output file: primary key, group coexistence, merge rule.

This is part of the same builder flow that produces report_plan.md, since the write side (step) and read side (widget) reference the same db/ file. Implies either a new plan-modification tool (set_db_schema) or an extension to add_scripted_step / update_scripted_step so the schema note is persisted alongside the step's db output declaration rather than left to the step agent to infer at runtime.

Key implementation touchpoints

File	Change
New: `frontend/src/components/workflow/KnowledgebaseView.tsx`	KB management popup
`WorkflowToolbar.tsx`	Add "Update KB" button + `handleRunKBUpdate()`
Step config panel component	Replace `disable_knowledgebase` toggle with `knowledgebase_access` dropdown
`frontend/src/services/api.ts`	Add `updateKnowledgebase()`, `getKnowledgebaseIndex()`, `clearKnowledgebase()` API calls
`frontend/src/services/api-types.ts`	Add `KBIndex`, `KBUpdateRequest`, `KBUpdateResponse` types

8. `soul.md` — Stable Workflow Intent

What it is

A Markdown file containing the workflow's stable north star. Runtime, evaluation, Pulse, and builder agents read its ## Objective and ## Success Criteria sections to decide what the workflow must achieve.

What it is NOT

Not architecture or a design document — the current "how" remains revisable in planning/plan.json, step config, and implementation artifacts
Not a decision log, navigation index, reference list, or store for agent assumptions
Not workflow config — capabilities, LLM settings, schedules stay in workflow.json
Not execution knowledge — HOW to run steps stays in learnings/
Not business context, examples, or domain facts — those stay in knowledgebase/

What it contains

It contains:

## Objective — the outcome the workflow exists to produce.
## Success Criteria — observable, checkable conditions for success.
Optional ## Constraints — only boundaries explicitly stated or approved by the user as durable and non-negotiable.
Optional ## Notifications — the user's Pulse notification preference.

Architecture, providers, models, channels, thresholds, step boundaries, tactics, and other implementation choices are not stable merely because an agent wrote them down. Pulse and Goal Advisor may challenge and replace them when evidence supports a better approach.

Relationship to `objective` and `success_criteria`

soul.md is the canonical source. planning/plan.json is the current implementation attempt and no longer owns root objective/success fields. Runtime helpers parse the two required H2 sections directly from soul.md.

Format

# LinkedIn Outreach

## Objective
Build a qualified list of prospects that can support effective outreach.

## Success Criteria
- Produce at least 20 verified, relevant leads per run.
- Keep duplicate and invalid-contact rates below the user-approved threshold.

## Constraints
- Never contact a prospect without human approval. <!-- only if user-approved -->

## Notifications
- Send a summary after every scheduled run.

When the builder writes to it

When the user confirms or changes the objective or success criteria.
When the user explicitly marks a boundary as a durable workflow-wide constraint.
When the user changes Pulse notification preferences.

Do not silently promote agent-inferred assumptions or implementation decisions into soul.md. When authorship or durability is unclear and materially affects the goal, Pulse surfaces it under Assumptions challenged for user review.

Location

soul/soul.md

Dedicated top-level folder at workspace root, parallel to planning/, db/, knowledgebase/, and learnings/. It stays separate because it defines the stable outcome that the current plan is judged against.

The soul/ folder is added to the builder's writable folder-guard paths. Workshop/Pulse may update it under the stable-intent contract; execution and evaluation agents read it but treat it as read-only.

Key implementation touchpoints

File	Change
`controller_run_manager.go`	Create `soul/` folder on workspace init (alongside `db/`, `knowledgebase/`, etc.)
Builder folder-guard setup (`interactive_workshop_manager.go` / `controller_agent_factory.go`)	Add `soul/` to the builder's writable paths so shell writes to `soul/soul.md` go through
`interactive_workshop_manager.go`	Load `soul/soul.md` at session start; inject as `SoulContent` template var into all workshop mode prompts
`interactive_workshop_manager.go`	After user-confirmed objective/success/constraint changes, builder updates `soul/soul.md` via shell (guard permits it; no dedicated mutation tool needed)
`cmd/server/workflow.go`	Add `handleGetSoul()`, `handleUpdateSoul()` HTTP handlers
`pre_validation.go:1080`	Add `"soul"` to known workflow folder names
`WorkflowToolbar.tsx`	Expose `soul.md` in the workspace file browser (already visible if file exists)

Implementation notes (2026-04)

Written against the shipping code after the bulk of this design landed. Captures deltas from the sections above — read the originals first, treat this as the "what actually happened" addendum. The original sections are left as the design intent; where reality diverged, this section wins.

Phase sequencing (what shipped when)

Phase	Scope	Status
1	`db/` folder + folder guards + `DBWrite` eval flag; KB access control (`knowledgebase_access` + `knowledgebase_contribution`); `soul/soul.md`	Shipped
2	KB graph scaffolding (empty seed only); `learningQueue` + `kbUpdateQueue` single-writer queues; learning retrofit; KB update agent + post-step trigger; `reorganize_knowledgebase` builder tool	Shipped
3	Dynamic report system — `ReportViewer.tsx` + parser + toolbar wiring; removal of the old `final_output.go` report agent and everything that fed it	Shipped
4	`KnowledgebaseView.tsx` (KB viewer + clear + export)	Shipped (partial — see §Deferred)
Deferred	Full-run `kb-update` phase (§6), step-config UI panel for `knowledgebase_access` / `knowledgebase_contribution`, Phase 4 `run_full_kb_update` button	Not shipped

Major change: Go does NOT parse `graph.json` or `index.json`

Sections 4 and 5 originally envisioned Go-side types (KBGraph, KBEntity, KBRelationship), a KBDelta handshake type returned from the KB update agent, ApplyKBDelta for Go-side merging, and RebuildKBIndex for index regeneration.

Actual implementation pivoted mid-build: the LLM owns graph.json and index.json end-to-end via shell. Go does not parse, validate, or merge content. The only Go code touching these files is:

kb_graph.go — 50 lines; two string-literal seed files (emptyGraphJSON, emptyIndexJSON) and InitKBGraphFiles which writes them once on workspace init when they don't exist. No types, no parsing.
The KB update agent prompt (kb_update_agent.go) — describes the schema in natural language; the agent reads, merges, and writes via cat, jq, diff_patch_workspace_file.

Consequence: if the agent's prompt evolves the schema (e.g. adds a confidence field), Go's seed files must be updated manually to match. Documented as a constraint at the top of kb_graph.go.

Major change: per-step KB access defaults to `"none"`, not `"read-write"`

Section 3's resolution logic specified:

preset enabled + no explicit step setting → "read-write"

Actual implementation is stricter: default is "none" regardless of preset state. KB access is opt-in per step.

if !presetEnabled { return KBAccessNone }       // preset is hard gate
if explicit setting { return explicit }          // honor it
return KBAccessNone                              // default: disabled

Rationale: the prior default let any step shell-write into knowledgebase/ despite the prompt saying "only the KB update agent writes graph.json". Opt-in by default removes the accidental-shell-write path.

Breaking consequence: existing workflows that implicitly used KB read access (any step doing cat knowledgebase/graph.json without declaring knowledgebase_access) will fail folder guard after this ships. Fix per-workflow by setting "knowledgebase_access": "read" on affected steps.

Major change: deprecated fields fully deleted

The design (§3) proposed keeping DisableKnowledgebase *bool as a backward-compat shim indefinitely. Actual: the shim has been deleted outright. LearningAfterLoopIteration (already marked deprecated in the struct) is also gone. Any step_config.json with "disable_knowledgebase": true or "learning_after_loop_iteration": ..." is silently ignored after this ships.

The update_step_config tool's JSON schema no longer exposes either. The builder agent only sees knowledgebase_access + knowledgebase_contribution as the KB-control knobs.

Major change: `reorganize_knowledgebase` builder tool (new, not in original design)

Added during Phase 2. The interactive builder can now apply natural-language transformations to graph.json (dedupe, rename types, purge bad provenance) via a dedicated one-shot agent KBReorganizeAgent. Serialized through the same kbUpdateQueue as the post-step KB update agent so they can't race.

Tool name: reorganize_knowledgebase(instruction: string)
Agent: KBReorganizeAgent in kb_update_agent.go
Orchestration: runKBReorganizePhase / RunKBReorganize in controller_kb_update.go
Registration: RegisterReorganizeKnowledgebaseTool in planning_exports.go
Available in: workflow-builder mode only

Major change: Section 2 report uses existing workspace API, not new wrappers

Section 2 listed handleGetReportPlan and handleGetDBFile as new HTTP handlers to write.

Actual implementation: those wrappers were built, then deleted. ReportViewer.tsx uses the existing workspace document API (agentApi.getPlannerFileContent → /api/documents/{path}) directly. No new backend endpoints. This removes ~80 lines of Go duplication that mirrored what the workspace service already exposes.

Path whitelist deferred: the original design included a server-side whitelist restricting widget sources to db/** and knowledgebase/graph.json / index.json. With the wrappers removed, the whitelist is gone too — widgets can technically point at any workspace file. The builder is expected to keep widget sources sensible (design doc already makes this a builder responsibility).

Major change: `ChartBlock` removed from `MarkdownRenderer.tsx`

Section 2 originally planned to reuse the SVG bar/line chart in MarkdownRenderer.tsx's existing ChartBlock component for widget:chart rendering. Actual: the markdown chart rendering was removed entirely (not used anywhere else in the app), and ReportViewer.tsx ships a simple HTML/CSS bar chart inline instead.

If SVG/line charts become necessary later, lift a small Chart component to frontend/src/components/ui/Chart.tsx and have both the markdown renderer and the report viewer consume it. Don't re-add chart rendering into MarkdownRenderer.

Major change: scheduler auto-report generation removed

Section 6 described a full-run kb-update phase but didn't discuss what happens to the old scheduler behavior of auto-generating reports after workflow runs. Actual: shouldAutoGenerateWorkshopReport and generateWorkshopScheduleReport helpers in cmd/server/scheduler.go have been deleted. The dynamic report is a live frontend view — there is no artifact to generate post-run, so scheduled runs now complete without a report side-effect.

Prompt changes (touching all three agent classes)

All three agent prompts now explicitly distinguish the three persistent stores:

learnings/ → HOW to run (methods, patterns, quirks)
knowledgebase/ → decisions, facts, strategies built up over time (entities + relationships; only the post-step KB update agent writes graph.json, never a step)
db/ → workflow state and results (rows produced/consumed this run; step-owned; upsert-by-key; never overwrite wholesale)

Applied in:

Execution agent prompt (execution_only_agent.go)
Todo-task orchestrator agent prompt (todo_task_orchestrator_agent.go)
Workshop builder prompt (interactive_workshop_manager.go)

The workshop builder prompt also carries the full three-way comparison table and "when to use which" deciding questions so the builder agent can recommend the right store to users asking "where should I put X?"

Concurrency model landed as single-writer queues (not mutex)

Section 5 considered three options: per-file mutex, single-writer queue, or append-only journal. Implemented: single-writer queues, via queues.go:

learningQueue — one worker, FIFO; serializes the learning agent. Retrofitted into both the tracked (workshop) and fallback (non-workshop) launch sites. Closes a pre-existing race on learnings/_global/ files.
kbUpdateQueue — one worker, FIFO; serializes post-step KB updates and reorganize_knowledgebase tool invocations (both flows enqueue onto the same channel).

Workers lazy-start via sync.Once on first enqueue — no explicit boot wiring. Panic guard per job so a bad agent run can't kill the worker.

Files actually touched

Area	New files	Modified	Deleted
Persistent stores	`kb_graph.go`, `kb_update_agent.go`, `controller_kb_update.go`, `queues.go`, `workshop_helpers.go`	`controller_execution.go`, `controller_run_manager.go`, `controller_agent_factory.go`, `controller_todo_task.go`, `controller_learning.go`, `controller_batch_execution.go`, `controller_learn_code.go`, `evaluation_types.go`, `execution_only_agent.go`, `todo_task_orchestrator_agent.go`, `planning_agent.go`, `pre_validation.go`	—
Report system	`ReportViewer.tsx`, `reportPlanParser.ts`	`interactive_workshop_manager.go` (prompts + tool schema), `api-types.ts`, `api.ts`	`final_output.go`, `FinalOutputPopup.tsx`
KB UI	`KnowledgebaseView.tsx`	`WorkflowToolbar.tsx` (Database icon, popup wiring)	—
Workflow-level	—	`workflow.go`, `server.go`, `scheduler.go`, `workflow_orchestrator.go`, `types.go` (workflowtypes)	—
Markdown renderer	—	`MarkdownRenderer.tsx` (deleted ChartBlock + chart language branch)	—

Rough LOC impact: +~2,800 net added, ~5,200 removed (dominated by the old report-agent deletion).

Not yet shipped (§Deferred, per original phase 4)

Full-run kb-update phase (§6) — manual "rebuild KB from a completed run" trigger with its own HTTP endpoint and orchestrator routing. KnowledgebaseView reserves space for a button but doesn't call it.
Step-config UI panel — knowledgebase_access dropdown and knowledgebase_contribution textarea. Right now the builder must set these via update_step_config in chat or via hand-edited step_config.json.
Scheduler "Run KB Update" integration — the scheduler could offer a post-run KB rebuild option; out of scope until §6 ships.

References

Auto-synced from docs/ on main. Edit there, not here.

persistent_stores_design

Persistent Stores Design

Background: Current Stores

1. New db/ Folder

What it is

Folder structure

Behaviour

Write semantics (single-db model)

Key implementation touchpoints

Evaluation runs and db/ (retired — eval subsystem removed; kept as history)

Open questions

2. Dynamic Report System

What changes from current

report_plan.md format

Widget types

Widget fields

Data sources

How the frontend renders it

Who creates report_plan.md

Report is not triggered by workflow execution

What goes away

Key implementation touchpoints

3. Step-Level Knowledgebase Access Control

What it is

Current state

Proposed: knowledgebase_access + knowledgebase_contribution fields

Resolution logic

Key implementation touchpoints

4. Knowledgebase Graph Structure

Motivation

File layout

graph.json schema

index.json schema

Key design principles

5. KB Update Agent (Post-Step)

What it is

When it runs

Serialization (learning and KB writes)

What it reads

What it writes

Agent instructions

Key implementation touchpoints

Folder guard for KB update agent

6. Global KB Update Tool (Full-Run)

What it is

Flow

HTTP endpoint

Key implementation touchpoints

7. Interactive Builder Changes

New: KB Management Popup

New: KB Update toolbar button

Updated: Step config UI — KB access mode + contribution

Updated: Preset modal

New: db/ schema notes owned by the builder

Key implementation touchpoints

8. soul.md — Stable Workflow Intent

What it is

What it is NOT

What it contains

Relationship to objective and success_criteria

Format

When the builder writes to it

Location

Key implementation touchpoints

Implementation notes (2026-04)

Phase sequencing (what shipped when)

Major change: Go does NOT parse graph.json or index.json

Major change: per-step KB access defaults to "none", not "read-write"

Major change: deprecated fields fully deleted

Major change: reorganize_knowledgebase builder tool (new, not in original design)

Major change: Section 2 report uses existing workspace API, not new wrappers

Major change: ChartBlock removed from MarkdownRenderer.tsx

Major change: scheduler auto-report generation removed

Prompt changes (touching all three agent classes)

Concurrency model landed as single-writer queues (not mutex)

Files actually touched

Not yet shipped (§Deferred, per original phase 4)

References

Uh oh!

Uh oh!

1. New `db/` Folder

Evaluation runs and `db/` (retired — eval subsystem removed; kept as history)

`report_plan.md` format

Who creates `report_plan.md`

Proposed: `knowledgebase_access` + `knowledgebase_contribution` fields

`graph.json` schema

`index.json` schema

New: `db/` schema notes owned by the builder

8. `soul.md` — Stable Workflow Intent

Relationship to `objective` and `success_criteria`

Major change: Go does NOT parse `graph.json` or `index.json`

Major change: per-step KB access defaults to `"none"`, not `"read-write"`

Major change: `reorganize_knowledgebase` builder tool (new, not in original design)

Major change: `ChartBlock` removed from `MarkdownRenderer.tsx`