Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions docs/ce/local-development/backend.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: Backend (orchestrator) local setup
---

The backend serves orchestration APIs, GitHub app endpoints, and internal APIs the UI relies on.

## Quick start

1. Create `backend/.env` with the essentials (adjust DB URL/ports):
```bash
DATABASE_URL=postgres://postgres:root@localhost:5432/postgres?sslmode=disable
DIGGER_INTERNAL_SECRET=orchestrator-secret # must match UI ORCHESTRATOR_BACKEND_SECRET
DIGGER_ENABLE_INTERNAL_ENDPOINTS=true # enables /_internal/*
DIGGER_ENABLE_API_ENDPOINTS=true # enables /api/*
HOSTNAME=http://localhost:3000 # used for GitHub app callbacks
```
Optional but useful:
- `GITHUB_APP_ID`, `GITHUB_APP_PEM`, `GITHUB_APP_WEBHOOK_SECRET` if you already have an app.
- `GITHUB_ORG` if you want `/github/setup` to target an org.
2. Start the service (from `backend/`):
```bash
set -a; source .env; set +a
go run main.go # or: make start
```
Default port: `3000`.

## Make the UI happy

- The UI calls `/api/*` and `/github/*` with `Authorization: Bearer $DIGGER_INTERNAL_SECRET` and `DIGGER_ORG_ID`/`DIGGER_USER_ID` headers.
- You must upsert the WorkOS org + user the UI is authenticated as:
```bash
SECRET=$DIGGER_INTERNAL_SECRET
ORG_ID=org_xxx # WorkOS org id
ORG_NAME=my-org # slug shown in backend
ADMIN_EMAIL=you@example.com
USER_ID=user_xxx # WorkOS user id
USER_EMAIL=$ADMIN_EMAIL

# org
curl -s -X POST http://localhost:3000/_internal/api/upsert_org \
-H "Authorization: Bearer $SECRET" \
-H "Content-Type: application/json" \
-d '{"org_name":"'"$ORG_NAME"'","external_source":"workos","external_id":"'"$ORG_ID"'","admin_email":"'"$ADMIN_EMAIL"'"}'

# user
curl -s -X POST http://localhost:3000/_internal/api/create_user \
-H "Authorization: Bearer $SECRET" \
-H "Content-Type: application/json" \
-d '{"external_source":"workos","external_id":"'"$USER_ID"'","email":"'"$USER_EMAIL"'","external_org_id":"'"$ORG_ID"'"}'
```

## GitHub app integration

- For a quick install link, set `ORCHESTRATOR_GITHUB_APP_URL` in `ui/.env.local` to your app’s install URL (`https://github.com/apps/<app>/installations/new`).
- To create a new app via the backend, open `http://localhost:3000/github/setup` (requires `HOSTNAME` set to a reachable URL for callbacks).

## Troubleshooting

- **404 on /api/repos**: ensure `DIGGER_ENABLE_API_ENDPOINTS=true` and the org/user above are created.
- **401/403**: verify `Authorization` header uses `DIGGER_INTERNAL_SECRET`.
- **GitHub connect 404**: set `ORCHESTRATOR_GITHUB_APP_URL` as described.
39 changes: 39 additions & 0 deletions docs/ce/local-development/overview.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
title: Local development overview
---

This section explains how to run the three core services locally:

- **Backend** (`backend/`, port 3000 by default) – orchestrator + REST APIs for repos/orgs/jobs.
- **Statesman** (`taco/cmd/statesman`, port 8080) – state storage API and Terraform Cloud-compatible endpoints.
- **UI** (`ui/`, port 3030) – TanStack Start frontend that talks to both services and WorkOS.

## Prerequisites

- Go toolchain for backend + statesman, Node 18+ for UI (`pnpm` or `npm`).
- A WorkOS project with User Management enabled and at least one organization + member (needed for UI auth and org IDs).
- Optional: GitHub App for repo onboarding (the backend can help you create one via `/github/setup`).

## Shared secrets and ports

- Pick two secrets and reuse them across components:
- `ORCHESTRATOR_BACKEND_SECRET` ≡ `DIGGER_INTERNAL_SECRET` (backend) ≡ UI env.
- `STATESMAN_BACKEND_WEBHOOK_SECRET` ≡ `OPENTACO_ENABLE_INTERNAL_ENDPOINTS` (statesman) ≡ UI env.
- Default ports: backend `3000`, statesman `8080`, UI `3030`.

## High-level workflow

1) **Start backend** with internal + API endpoints enabled (so UI can call `/api/*` and `/github/*`).
2) **Start statesman** with internal endpoints enabled; use memory storage for quick start.
3) **Configure UI** `.env.local` with URLs + secrets + WorkOS creds; run `pnpm dev --host --port 3030`.
4) **Sync org/user** into backend and statesman (WorkOS org id and user id/email) via the provided curl snippets in each page.
5) (Optional) **GitHub App**: set `ORCHESTRATOR_GITHUB_APP_URL` to your install URL or `http://localhost:3000/github/setup` to create one via the backend.

## Troubleshooting cheatsheet

- **Backend /api/* returns 404**: `DIGGER_ENABLE_API_ENDPOINTS` not `true` or org not upserted.
- **Statesman 403**: webhook secret mismatch. **Statesman 404/500 resolving org**: org not synced (missing `external_org_id`).
- **UI WorkOS auth succeeds but org is empty**: add membership in WorkOS and resync org/user to services.
- **GitHub connect opens 404**: set `ORCHESTRATOR_GITHUB_APP_URL` to a valid install/setup URL.

Continue with the per-service pages for commands and env examples.
54 changes: 54 additions & 0 deletions docs/ce/local-development/statesman.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
title: Statesman local setup
---

Statesman serves state storage + Terraform Cloud-compatible APIs. The UI uses its internal endpoints, so enable webhook auth and sync your org/user.

## Quick start

1. Set env vars:
```bash
OPENTACO_ENABLE_INTERNAL_ENDPOINTS=statesman-secret # must match UI STATESMAN_BACKEND_WEBHOOK_SECRET
OPENTACO_AUTH_DISABLE=true # skips OIDC for local
OPENTACO_STORAGE=memory # default; uses SQLite query backend automatically
# Optional: OPENTACO_SECRET_KEY for signed URLs; OPENTACO_PORT=8080
```
2. Run the service (from repo root):
```bash
cd taco
go run cmd/statesman/main.go -storage memory -auth-disable # or ./statesman with the same flags
```
Default port: `8080`.

## Sync org and user (required for UI)

Statesman resolves orgs by `external_org_id` (your WorkOS org id). If it cannot resolve, `/internal/api/units` will 500.

```bash
SECRET=$OPENTACO_ENABLE_INTERNAL_ENDPOINTS
ORG_ID=org_xxx # WorkOS org id
ORG_NAME=digger-org # slug to store
ORG_DISPLAY="Digger Org"
USER_ID=user_xxx # WorkOS user id
USER_EMAIL=you@example.com

# create/sync org
curl -s -X POST http://localhost:8080/internal/api/orgs/sync \
-H "Authorization: Bearer $SECRET" \
-H "X-User-ID: $USER_ID" -H "X-Email: $USER_EMAIL" \
-H "Content-Type: application/json" \
-d '{"name":"'"$ORG_NAME"'","display_name":"'"$ORG_DISPLAY"'","external_org_id":"'"$ORG_ID"'"}'

# ensure user exists in that org
curl -s -X POST http://localhost:8080/internal/api/users \
-H "Authorization: Bearer $SECRET" \
-H "X-Org-ID: '$ORG_ID'" -H "X-User-ID: $USER_ID" -H "X-Email: $USER_EMAIL" \
-H "Content-Type: application/json" \
-d '{"subject":"'"$USER_ID"'","email":"'"$USER_EMAIL"'"}'
```

## Troubleshooting

- **403**: webhook secret mismatch (`OPENTACO_ENABLE_INTERNAL_ENDPOINTS` vs UI `STATESMAN_BACKEND_WEBHOOK_SECRET`).
- **404/500 resolving org**: org not synced; rerun the `orgs/sync` call above.
- **SQLite quirks**: defaults to SQLite in-process; no config needed for local. For Postgres/MySQL, set `TACO_QUERY_BACKEND` and related envs (see `docs/ce/state-management/query-backend`).
51 changes: 51 additions & 0 deletions docs/ce/local-development/ui.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
title: UI local setup
---

The UI is a TanStack Start app that authenticates via WorkOS and calls both backend and statesman.

## Quick start

1. Copy `.env.example` to `.env.local` in `ui/` and fill the essentials:
```bash
# URLs
PUBLIC_URL=http://localhost:3030
ALLOWED_HOSTS=localhost,127.0.0.1

# WorkOS (User Management)
WORKOS_CLIENT_ID=<your_workos_client_id>
WORKOS_API_KEY=<your_workos_api_key>
WORKOS_COOKIE_PASSWORD=<32+ random chars>
WORKOS_REDIRECT_URI=http://localhost:3030/api/auth/callback
WORKOS_WEBHOOK_SECRET=<if using webhooks locally via tunnel>

# Backend
ORCHESTRATOR_BACKEND_URL=http://localhost:3000
ORCHESTRATOR_BACKEND_SECRET=orchestrator-secret # matches backend DIGGER_INTERNAL_SECRET
ORCHESTRATOR_GITHUB_APP_URL=http://localhost:3000/github/setup # or your app install URL

# Statesman
STATESMAN_BACKEND_URL=http://localhost:8080
STATESMAN_BACKEND_WEBHOOK_SECRET=statesman-secret # matches OPENTACO_ENABLE_INTERNAL_ENDPOINTS

# Optional
DRIFT_REPORTING_BACKEND_URL=http://localhost:3004
DRIFT_REPORTING_BACKEND_WEBHOOK_SECRET=...
POSTHOG_KEY=...
```
In WorkOS, add `http://localhost:3030/api/auth/callback` as a redirect.
2. Install deps and run:
```bash
cd ui
pnpm install # or npm install
pnpm dev --host --port 3030
```
Open `http://localhost:3030` and sign in with a WorkOS user that belongs to at least one org.
3. Ensure backend + statesman were started and the same secrets are in place (see [Backend](./backend) and [Statesman](./statesman)).
4. Sync the WorkOS org/user to both services using the curl snippets on those pages (required for repos/units to load).

## Common errors

- **NotFound/Forbidden listing units**: statesman org/user not synced or webhook secret mismatch.
- **404 on repos or GitHub connect**: backend missing org/user, `DIGGER_ENABLE_API_ENDPOINTS` not set, or `ORCHESTRATOR_GITHUB_APP_URL` points to a non-existent path.
- **WorkOS login succeeds but dashboard redirects to / or errors**: the signed-in user has no WorkOS org membership; add to an org and resync to services.
11 changes: 10 additions & 1 deletion docs/docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,15 @@
"ce/azure-specific/azure-devops-locking-connection-methods"
]
},
{
"group": "Local Development",
"pages": [
"ce/local-development/overview",
"ce/local-development/backend",
"ce/local-development/statesman",
"ce/local-development/ui"
]
},
{
"group": "Contributing",
"pages": [
Expand Down Expand Up @@ -213,4 +222,4 @@
"linkedin": "https://www.linkedin.com/company/diggerhq/"
}
}
}
}
Loading