Context Artist

A prompt engineering workbench for local LLMs. Author, version, and test prompts through a browser UI. Integrate via WebSocket, HTTP, or Node.js library. Everything runs locally via Ollama.

---

Requirements

• Node.js v22+
• Ollama running locally (default: http://localhost:11434)

---

Setup

npm install
npm start                                        # random port, no db (UI prompts on open)
npm start -- --port 3000 --db /path/to/ctxart.db # fixed port and db path
npm start -- --ollama http://192.168.1.5:11434   # custom Ollama URL

The server prints its port on startup:

Server running on port 54321
Webapp: http://localhost:54321

Open that URL in your browser. If no --db flag was passed, the UI will prompt for a database path before you can use it — enter an absolute path and the file will be created if it doesn't exist.

---

Core Concepts

Objective

An objective is the named unit of work — what the LLM is supposed to accomplish. Each objective has a slug (summarize-ticket, classify-intent) that appears in all API and WS calls. It holds one or more template versions.

Template Version

A template defines how an objective executes: system prompt, seed chat (few-shot turns), and model selection. Versions are immutable once committed. You edit a draft, then commit it to create a new version.

Draft → Commit Flow

1. Create or select an objective
2. Edit in the Editor tab — changes are saved as a draft (auto-persisted, 500ms debounce)
3. Click Commit to lock the draft as a new version
4. Committed versions are available for runs and benchmarks

Test Cases

Saved inputs (with optional ideal outputs) used to benchmark versions. Run all test cases for an objective across any version from the Tests tab.

---

UI Tabs (per objective)

Tab	Purpose
Editor	Draft system prompt, seed chat, model selection.
Versions	List of committed versions and their run information.
Runs	Live and historical run output with token confidence heatmap and test case management.
Metrics	Aggregated analytics: mean logprob, latency, token cost per version.
Config	Objective-level settings: API info, tags.

---

Running a Prompt

WebSocket (primary runtime interface)

Connect to /socket. All LLM output flows through WebSockets.

Subscribe to channels:

{ "type": "sub", "channels": ["firehose"] }

Channel options: firehose, objective:{slug}, tag:{tag}, run:{run_id}

Trigger a run (committed version):

{
  "type": "run",
  "objective": "summarize-ticket",
  "input": "ticket content here",
  "caller_id": "my-app"
}

Optional fields: versionId (defaults to latest committed version), testCaseId (uses a saved test case as input instead of input).

Events emitted:

// Run started
{ "type": "run.start", "run_id": "...", "objective": "slug", "versionId": "...",
  "input": "...", "context": { "system": "...", "seedChat": [...], "userInput": "..." },
  "model": "...", "draft": false, "channels": [...] }

// Token (streaming)
{ "type": "run.token", "run_id": "...", "delta": "...", "draft": false, "channels": [...] }

// Completed
{ "type": "run.done", "run_id": "...", "output": "...",
  "metrics": { "tokens_in": 312, "tokens_out": 87, "mean_logprob": -0.31, "latency_ms": 1240 },
  "logprobs": [...], "draft": false, "channels": [...] }

Subscribe to run:{run_id} (from run.start) and unsubscribe after run.done to wait on a single result without polling.

Headless Draft Testing

Draft runs use the objective's current draft instead of any committed version. Their output is published only to objective:{slug}:draft and the initiating socket's run:{run_id} — they never appear on firehose, objective:{slug}, or tag:{tag} channels, and are never written to the database.

Workflow:

1. Create the objective and write its draft via the HTTP API (see Drafts below).
2. Subscribe to the draft channel so you receive run events:

   { "type": "sub", "channels": ["objective:summarize-ticket:draft"] }

3. Trigger a draft run:

   {
     "type": "run",
     "objective": "summarize-ticket",
     "input": "test input",
     "draft": true
   }

4. Receive the same run.start / run.token / run.done event sequence as a normal run, with "draft": true on each event.

Multiple subscribers can monitor the same draft channel simultaneously — for example a CI script and the browser UI watching the same objective will both receive the output.

Once the draft is satisfactory, commit it to a version via the HTTP API (POST /api/objectives/:slug/versions) and run against the version normally.

HTTP REST (management only)

Objectives

POST /api/objectives — create an objective

{
  "name": "Summarize ticket", // required — human label
  "slug": "summarize-ticket", // optional — auto-derived from name if omitted
  "tags": ["production", "support"] // optional
}

GET /api/objectives — list all objectives

GET /api/objectives/:slug — get one objective

PATCH /api/objectives/:slug — update tags or default version

{
  "tags": ["production"], // optional
  "defaultVersionId": "uuid" // optional
}

Drafts

Each objective has one draft — the in-progress edit before committing. The UI auto-saves the draft on every change.

GET /api/objectives/:slug/draft — get current draft

PATCH /api/objectives/:slug/draft — save draft (full replace)

{
  "model": "qwen3:8b",
  "systemBlocks": [{ "type": "text", "content": "You are a..." }],
  "seedChat": [
    {
      "role": "user",
      "blocks": [{ "type": "text", "content": "Example input" }]
    },
    {
      "role": "assistant",
      "blocks": [{ "type": "text", "content": "Example output" }]
    }
  ],
  "think": false,
  "options": { "temperature": 0.7, "top_k": null, "top_p": null, "min_p": null }
}

DELETE /api/objectives/:slug/draft — clear draft

Versions

Versions are created by committing a draft. They are immutable after creation.

GET /api/objectives/:slug/versions — list committed versions

POST /api/objectives/:slug/versions — commit a new version (body has the same shape as the draft PATCH above)

Test Cases

GET /api/objectives/:slug/tests — list test cases

POST /api/objectives/:slug/tests — create a test case

{
  "name": "Short ticket",
  "inputBlocks": [{ "type": "text", "content": "Printer not working" }]
}

PATCH /api/objectives/:slug/tests/:id — update a test case (same fields, all optional)

DELETE /api/objectives/:slug/tests/:id — delete a test case

Runs

GET /api/objectives/:slug/runs — last 20 runs for an objective

GET /api/objectives/:slug/versions/:v/runs — last 20 runs for a specific version number

Tags

GET /api/tags — list all tags in use across all objectives

GET /api/tags/:tag/objectives — list objectives with a given tag

System

GET /api/status — server readiness: { "db": true, "ollama": true }

POST /api/config/db — set database path at runtime (no db required to call this)

{ "path": "/absolute/path/to/ctxart.db" }

GET /api/ollama/models — list model names available in Ollama

---

Data Storage

All data is stored in /path/to/ctxart.db (SQLite, better-sqlite3). You can select the path on load, or by passing --db /path/to/ctxart.db. Back it up by copying the file.

---

TypeScript / UI Development

The server compiles .ts files on-the-fly with esbuild — no build step needed. Edit files in src/ui/app/ and reload the browser. The UI is built with Lit web components.

To add a new component, create src/ui/app/components/ctx-my-thing.ts.