Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions .claude/skills/swamp-model/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,19 +12,19 @@ machine-readable output.

Swamp uses a dual-layer architecture:

- **Data directory (`/data/`)** - Internal storage organized by entity type
- **Data directory (`/.data/`)** - Internal storage organized by entity type
- **Logical views (`/models/`)** - Human-friendly symlinked directories

The `/models/` directory provides convenient exploration of each model:

```
/models/{model-name}/
input.yaml → ../data/inputs/{type}/{id}.yaml
resource.yaml → ../data/resources/{type}/{id}.yaml
data.yaml → ../data/data/{type}/{id}.yaml
outputs/ → ../data/outputs/{type}/{id}/
logs/ → ../data/logs/{type}/{id}/
files/ → ../data/files/{type}/{id}/
input.yaml → ../.data/inputs/{type}/{id}.yaml
resource.yaml → ../.data/resources/{type}/{id}.yaml
data.yaml → ../.data/data/{type}/{id}.yaml
outputs/ → ../.data/outputs/{type}/{id}/
logs/ → ../.data/logs/{type}/{id}/
files/ → ../.data/files/{type}/{id}/
```

This structure is maintained automatically. Use `swamp repo index` to rebuild if
Expand Down
6 changes: 3 additions & 3 deletions .claude/skills/swamp-workflow/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,18 @@ machine-readable output.

Swamp uses a dual-layer architecture:

- **Data directory (`/data/`)** - Internal storage organized by entity type
- **Data directory (`/.data/`)** - Internal storage organized by entity type
- **Logical views (`/workflows/`)** - Human-friendly symlinked directories

The `/workflows/` directory provides convenient exploration of each workflow:

```
/workflows/{workflow-name}/
workflow.yaml → ../data/workflows/{id}.yaml
workflow.yaml → ../.data/workflows/{id}.yaml
runs/
latest → {most-recent-run}/
{timestamp}/
run.yaml → ../data/workflow-runs/{id}/{run-id}.yaml
run.yaml → ../.data/workflow-runs/{id}/{run-id}.yaml
```

This structure is maintained automatically. Use `swamp repo index` to rebuild if
Expand Down
14 changes: 7 additions & 7 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@ dev-logs/*
swamp

# Generated/runtime data (ignored)
data/data/
data/outputs/
data/workflow-runs/
data/inputs-evaluated/
data/workflows-evaluated/
data/logs/
data/files/
.data/data/
.data/outputs/
.data/workflow-runs/
.data/inputs-evaluated/
.data/workflows-evaluated/
.data/logs/
.data/files/

test-repo/
experiments/webapp/frontend/node_modules/
Expand Down
2 changes: 1 addition & 1 deletion design/agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ are the recommended way to explore and understand the repository structure.

### Data Directory (Direct Access)

The `/data/` directory contains the internal storage format. Agents can access
The `/.data/` directory contains the internal storage format. Agents can access
it directly when needed, but the layout reflects swamp's internal architecture
rather than user-facing concerns.

Expand Down
6 changes: 3 additions & 3 deletions design/expressions.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,11 +99,11 @@ registering custom types, functions, etc in their swamp repo.

When loading the YAML, first parse the CEL expressions. Then take the data
structures they emit and embed them in the data structure. Write those to a
directory in the repository called `/data/inputs-evaluated/` whose structure is
the same as `/data/inputs/`. This directory should be in a swamp repo's
directory in the repository called `/.data/inputs-evaluated/` whose structure is
the same as `/.data/inputs/`. This directory should be in a swamp repo's
.gitignore file.

The same is true for `/data/workflows-evaluated/`, and it should also be in
The same is true for `/.data/workflows-evaluated/`, and it should also be in
.gitignore.

These evaluated directories are internal working directories in the data layer,
Expand Down
42 changes: 21 additions & 21 deletions design/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,10 @@ A model can migrate its inputs and resources from one version to the next.

## Inputs

Inputs are specified as YAML files that live in the `/data/inputs/` directory of
a repository, underneath the normalized type as a directory. The file name is
Inputs are specified as YAML files that live in the `/.data/inputs/` directory
of a repository, underneath the normalized type as a directory. The file name is
`${id}.yaml`. For example,
`data/inputs/aws/ec2/vpc/fc7fd41e-ae16-4b31-b57a-86de716e3ece.yaml`.
`.data/inputs/aws/ec2/vpc/fc7fd41e-ae16-4b31-b57a-86de716e3ece.yaml`.

The valid shape of an input is specified with a Zod 4 schema.

Expand Down Expand Up @@ -79,7 +79,7 @@ method.
### Logs

A method may produce 0..\* log artifacts. They have a name, can have lines
streamed to them, and by default are stored in `/data/logs/` in the repository
streamed to them, and by default are stored in `/.data/logs/` in the repository
underneath the normalized model type as a directory. Logs are named like
`{model-id}-{method name}-{log name}-{timestamp}.log`.

Expand All @@ -91,27 +91,27 @@ artifact named 'k8slog'.
The stream of log output should have an event emitter attached to it, so we can
stream logs in real time.

By default, the `/data/logs/` directory is not stored in git.
By default, the `/.data/logs/` directory is not stored in git.

### Files

A method may store 0..\* file artifacts. They have a name, and can be written to
directly. By default they will be stored in the `/data/files/` directory of the
directly. By default they will be stored in the `/.data/files/` directory of the
repository underneath the normalized model type as a directory plus the model ID
and method name. For example
`data/files/aws/s3/bucket/{model-id}/{method-name}/{filename}`.
`.data/files/aws/s3/bucket/{model-id}/{method-name}/{filename}`.

By default, the `/data/files/` directory is not stored in git.
By default, the `/.data/files/` directory is not stored in git.

## Resource

Resource artifacts are used to track data about an external resource that should
be persisted over time (for example, the data about an AWS cloud resource).

A method may produce 0..1 resource as specified as YAML files that live in the
`/data/resources/` directory of a repository, underneath the normalized type as
`/.data/resources/` directory of a repository, underneath the normalized type as
a directory. The file name is `${id}.yaml`. For example,
`data/resources/aws/ec2/vpc/fc7fd41e-ae16-4b31-b57a-86de716e3ece.yaml`.
`.data/resources/aws/ec2/vpc/fc7fd41e-ae16-4b31-b57a-86de716e3ece.yaml`.

The valid shape of a resource is specified with a Zod 4 schema.

Expand All @@ -122,7 +122,7 @@ Resources are tracked in git.
Data artifacts are pure data objects that are not persisted over time in git.

A method may produce 0..1 data artifacts, stored as YAML files in the
`/data/data/` directory of a repository, underneath the normalized type as a
`/.data/data/` directory of a repository, underneath the normalized type as a
directory. The file name is `${id}.yaml`.

The valid shape of data is specified with a Zod 4 schema.
Expand Down Expand Up @@ -151,11 +151,11 @@ Use **data artifacts** when:
## Output

Each method invocation produces an output record, which gets tracked in the
`/data/outputs/` directory of a repository (which should not be tracked in git).
The output record should track the state of the method execution, and the list
of artifacts produced by the method. It should track state as the method
`/.data/outputs/` directory of a repository (which should not be tracked in
git). The output record should track the state of the method execution, and the
list of artifacts produced by the method. It should track state as the method
executes. It should be structured as
`/data/outputs/{normalized-type}/{method}/{model-id}-{timestamp}.yaml`.
`/.data/outputs/{normalized-type}/{method}/{model-id}-{timestamp}.yaml`.

## Logical Views

Expand All @@ -166,13 +166,13 @@ provides human/agent-friendly exploration of models by name.

```
/models/{model-name}/
input.yaml → symlink to /data/inputs/{type}/{id}.yaml
resource.yaml → symlink to /data/resources/{type}/{id}.yaml
data.yaml → symlink to /data/data/{type}/{id}.yaml
logs/ → symlink to /data/logs/{type}/{id}/
files/ → symlink to /data/files/{type}/{id}/
input.yaml → symlink to /.data/inputs/{type}/{id}.yaml
resource.yaml → symlink to /.data/resources/{type}/{id}.yaml
data.yaml → symlink to /.data/data/{type}/{id}.yaml
logs/ → symlink to /.data/logs/{type}/{id}/
files/ → symlink to /.data/files/{type}/{id}/
outputs/
{method}/ → symlinks to /data/outputs/{type}/{method}/{id}-*.yaml
{method}/ → symlinks to /.data/outputs/{type}/{method}/{id}-*.yaml
```

This structure allows exploring all artifacts for a model in one place, using
Expand Down
16 changes: 8 additions & 8 deletions design/repo.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,24 +115,24 @@ The RepoIndexService maintains two primary logical views:

```
/models/{model-name}/
input.yaml → /data/inputs/{type}/{id}.yaml
resource.yaml → /data/resources/{type}/{id}.yaml
data.yaml → /data/data/{type}/{id}.yaml
logs/ → /data/logs/{type}/{id}/
files/ → /data/files/{type}/{id}/
input.yaml → /.data/inputs/{type}/{id}.yaml
resource.yaml → /.data/resources/{type}/{id}.yaml
data.yaml → /.data/data/{type}/{id}.yaml
logs/ → /.data/logs/{type}/{id}/
files/ → /.data/files/{type}/{id}/
outputs/
{method}/ → /data/outputs/{type}/{method}/{id}-{timestamp}.yaml
{method}/ → /.data/outputs/{type}/{method}/{id}-{timestamp}.yaml
```

**Workflow View (`/workflows/`):**

```
/workflows/{workflow-name}/
workflow.yaml → /data/workflows/{id}.yaml
workflow.yaml → /.data/workflows/{id}.yaml
runs/
latest/ -> (points to latest timestamp)
{timestamp}/
run.yaml → /data/workflow-runs/{workflow-id}/{run-id}.yaml
run.yaml → /.data/workflow-runs/{workflow-id}/{run-id}.yaml
steps/
{step-name}/
output.yaml → symlink to step output
Expand Down
16 changes: 8 additions & 8 deletions design/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,14 @@ weighted topological sort, so thtat htye have maximum paralleism through the
workflow. Like steps, jobs also have conditions that trigger them.

Workflows are specified in YAML files, that are validated with Zod, in the
`/data/workflows/` directory of the repository, with their `{uuid}.yaml`.
Workflow run output is stored in `/data/workflow-runs/` at
`/data/workflow-runs/{workflow-uuid}/{run-uuid}.yaml`.
`/.data/workflows/` directory of the repository, with their `{uuid}.yaml`.
Workflow run output is stored in `/.data/workflow-runs/` at
`/.data/workflow-runs/{workflow-uuid}/{run-uuid}.yaml`.

## Workflow Definition

Workflows are specified in `/data/workflows/{uuid}.yaml`. They have a unique id,
a globally unique name, and a set of jobs.
Workflows are specified in `/.data/workflows/{uuid}.yaml`. They have a unique
id, a globally unique name, and a set of jobs.

## Jobs

Expand All @@ -49,7 +49,7 @@ not vary between identical inputs. (If the inputs are identical, the run order
should be deterministic.)

The output of the run will be written to a workflow run log, kept in
`/data/workflow-runs/{workflow-uuid}/{run-uuid}.yaml`.
`/.data/workflow-runs/{workflow-uuid}/{run-uuid}.yaml`.

## Logical Views

Expand All @@ -60,10 +60,10 @@ that provides human/agent-friendly exploration of workflows by name.

```
/workflows/{workflow-name}/
workflow.yaml → symlink to /data/workflows/{uuid}.yaml
workflow.yaml → symlink to /.data/workflows/{uuid}.yaml
runs/
{run-id}/
run.yaml → symlink to /data/workflow-runs/{workflow-uuid}/{run-uuid}.yaml
run.yaml → symlink to /.data/workflow-runs/{workflow-uuid}/{run-uuid}.yaml
steps/
{step-name}/
output.yaml → symlink to step output
Expand Down
4 changes: 2 additions & 2 deletions integration/echo_model_test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -122,8 +122,8 @@ Deno.test("Echo model: directory structure is correct", async () => {
await dataRepo.save(modelType, result.data!);

// Verify directory structure
const inputDir = join(repoDir, "data", "inputs", "swamp/echo");
const dataDir = join(repoDir, "data", "data", "swamp/echo");
const inputDir = join(repoDir, ".data", "inputs", "swamp/echo");
const dataDir = join(repoDir, ".data", "data", "swamp/echo");

assertEquals(existsSync(inputDir), true, "Input directory should exist");
assertEquals(
Expand Down
24 changes: 12 additions & 12 deletions integration/repo_index_test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,14 @@ async function withTempDir(fn: (dir: string) => Promise<void>): Promise<void> {

async function setupRepoDir(dir: string): Promise<void> {
const subdirs = [
"data/inputs",
"data/resources",
"data/data",
"data/outputs",
"data/workflows",
"data/workflow-runs",
"data/logs",
"data/files",
".data/inputs",
".data/resources",
".data/data",
".data/outputs",
".data/workflows",
".data/workflow-runs",
".data/logs",
".data/files",
"models",
"workflows",
];
Expand Down Expand Up @@ -318,7 +318,7 @@ Deno.test("Integration: repo index verify detects broken symlinks", async () =>
const modelDir = join(repoDir, "models", "broken-model");
await ensureDir(modelDir);
await Deno.symlink(
"../data/inputs/nonexistent/file.yaml",
"../.data/inputs/nonexistent/file.yaml",
join(modelDir, "input.yaml"),
);

Expand All @@ -341,7 +341,7 @@ Deno.test("Integration: repo index prune removes broken symlinks", async () => {
await ensureDir(modelDir);
const brokenLink = join(modelDir, "input.yaml");
await Deno.symlink(
"../data/inputs/nonexistent/file.yaml",
"../.data/inputs/nonexistent/file.yaml",
brokenLink,
);

Expand Down Expand Up @@ -453,7 +453,7 @@ Deno.test("CLI: repo index --verify fails on broken symlinks", async () => {
const modelDir = join(repoDir, "models", "broken-verify-test");
await ensureDir(modelDir);
await Deno.symlink(
"../data/inputs/nonexistent/file.yaml",
"../.data/inputs/nonexistent/file.yaml",
join(modelDir, "input.yaml"),
);

Expand Down Expand Up @@ -481,7 +481,7 @@ Deno.test("CLI: repo index --prune removes broken symlinks", async () => {
await ensureDir(modelDir);
const brokenLink = join(modelDir, "input.yaml");
await Deno.symlink(
"../data/inputs/nonexistent/file.yaml",
"../.data/inputs/nonexistent/file.yaml",
brokenLink,
);

Expand Down
2 changes: 1 addition & 1 deletion integration/workflow_test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1121,7 +1121,7 @@ Deno.test("CLI: model delete cleans up empty type directories", async () => {
assertEquals(createResult.code, 0, "Model create should succeed");

// Verify the directory structure was created
const inputsDir = `${repoDir}/data/inputs`;
const inputsDir = `${repoDir}/.data/inputs`;
const echoDir = `${inputsDir}/swamp/echo`;
const swampDir = `${inputsDir}/swamp`;

Expand Down
18 changes: 9 additions & 9 deletions src/domain/repo/repo_index_service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,22 +50,22 @@ export interface RebuildResult {
*
* Model View (`/models/{model-name}/`):
* ```
* input.yaml → ../data/inputs/{type}/{id}.yaml
* resource.yaml → ../data/resources/{type}/{id}.yaml
* data.yaml → ../data/data/{type}/{id}.yaml
* logs/ → ../data/logs/{type}/{id}/
* files/ → ../data/files/{type}/{id}/
* input.yaml → ../.data/inputs/{type}/{id}.yaml
* resource.yaml → ../.data/resources/{type}/{id}.yaml
* data.yaml → ../.data/data/{type}/{id}.yaml
* logs/ → ../.data/logs/{type}/{id}/
* files/ → ../.data/files/{type}/{id}/
* outputs/
* {method}/ → ../data/outputs/{type}/{method}/
* {method}/ → ../.data/outputs/{type}/{method}/
* ```
*
* Workflow View (`/workflows/{workflow-name}/`):
* ```
* workflow.yaml → ../data/workflows/workflow-{id}.yaml
* workflow.yaml → ../.data/workflows/workflow-{id}.yaml
* runs/
* latest/ → {latest-timestamp}/
* {timestamp}/
* run.yaml → ../data/workflow-runs/{workflow-id}/workflow-run-{run-id}.yaml
* run.yaml → ../.data/workflow-runs/{workflow-id}/workflow-run-{run-id}.yaml
* steps/
* {step-name}/
* output.yaml → symlink to step output
Expand Down Expand Up @@ -161,7 +161,7 @@ export interface RepoIndexService {
* Rebuilds all logical views from scratch.
*
* Deletes existing /models/ and /workflows/ directories
* and recreates them from the /data/ directory.
* and recreates them from the /.data/ directory.
*
* @returns Rebuild result with counts
*/
Expand Down
2 changes: 1 addition & 1 deletion src/domain/repo/repo_service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ Use \`swamp --help\` to see available commands.
private async createDataDirectoryStructure(
repoPath: RepoPath,
): Promise<void> {
const dataDir = join(repoPath.value, "data");
const dataDir = join(repoPath.value, ".data");
const subdirs = [
"inputs",
"resources",
Expand Down
Loading