diff --git a/docs/ce/local-development/backend.mdx b/docs/ce/local-development/backend.mdx new file mode 100644 index 000000000..21895892b --- /dev/null +++ b/docs/ce/local-development/backend.mdx @@ -0,0 +1,61 @@ +--- +title: Backend (orchestrator) local setup +--- + +The backend serves orchestration APIs, GitHub app endpoints, and internal APIs the UI relies on. + +## Quick start + +1. Create `backend/.env` with the essentials (adjust DB URL/ports): + ```bash + DATABASE_URL=postgres://postgres:root@localhost:5432/postgres?sslmode=disable + DIGGER_INTERNAL_SECRET=orchestrator-secret # must match UI ORCHESTRATOR_BACKEND_SECRET + DIGGER_ENABLE_INTERNAL_ENDPOINTS=true # enables /_internal/* + DIGGER_ENABLE_API_ENDPOINTS=true # enables /api/* + HOSTNAME=http://localhost:3000 # used for GitHub app callbacks + ``` + Optional but useful: + - `GITHUB_APP_ID`, `GITHUB_APP_PEM`, `GITHUB_APP_WEBHOOK_SECRET` if you already have an app. + - `GITHUB_ORG` if you want `/github/setup` to target an org. +2. Start the service (from `backend/`): + ```bash + set -a; source .env; set +a + go run main.go # or: make start + ``` + Default port: `3000`. + +## Make the UI happy + +- The UI calls `/api/*` and `/github/*` with `Authorization: Bearer $DIGGER_INTERNAL_SECRET` and `DIGGER_ORG_ID`/`DIGGER_USER_ID` headers. +- You must upsert the WorkOS org + user the UI is authenticated as: + ```bash + SECRET=$DIGGER_INTERNAL_SECRET + ORG_ID=org_xxx # WorkOS org id + ORG_NAME=my-org # slug shown in backend + ADMIN_EMAIL=you@example.com + USER_ID=user_xxx # WorkOS user id + USER_EMAIL=$ADMIN_EMAIL + + # org + curl -s -X POST http://localhost:3000/_internal/api/upsert_org \ + -H "Authorization: Bearer $SECRET" \ + -H "Content-Type: application/json" \ + -d '{"org_name":"'"$ORG_NAME"'","external_source":"workos","external_id":"'"$ORG_ID"'","admin_email":"'"$ADMIN_EMAIL"'"}' + + # user + curl -s -X POST http://localhost:3000/_internal/api/create_user \ + -H "Authorization: Bearer $SECRET" \ + -H "Content-Type: application/json" \ + -d '{"external_source":"workos","external_id":"'"$USER_ID"'","email":"'"$USER_EMAIL"'","external_org_id":"'"$ORG_ID"'"}' + ``` + +## GitHub app integration + +- For a quick install link, set `ORCHESTRATOR_GITHUB_APP_URL` in `ui/.env.local` to your app’s install URL (`https://github.com/apps//installations/new`). +- To create a new app via the backend, open `http://localhost:3000/github/setup` (requires `HOSTNAME` set to a reachable URL for callbacks). + +## Troubleshooting + +- **404 on /api/repos**: ensure `DIGGER_ENABLE_API_ENDPOINTS=true` and the org/user above are created. +- **401/403**: verify `Authorization` header uses `DIGGER_INTERNAL_SECRET`. +- **GitHub connect 404**: set `ORCHESTRATOR_GITHUB_APP_URL` as described. diff --git a/docs/ce/local-development/overview.mdx b/docs/ce/local-development/overview.mdx new file mode 100644 index 000000000..d1ab2912a --- /dev/null +++ b/docs/ce/local-development/overview.mdx @@ -0,0 +1,39 @@ +--- +title: Local development overview +--- + +This section explains how to run the three core services locally: + +- **Backend** (`backend/`, port 3000 by default) – orchestrator + REST APIs for repos/orgs/jobs. +- **Statesman** (`taco/cmd/statesman`, port 8080) – state storage API and Terraform Cloud-compatible endpoints. +- **UI** (`ui/`, port 3030) – TanStack Start frontend that talks to both services and WorkOS. + +## Prerequisites + +- Go toolchain for backend + statesman, Node 18+ for UI (`pnpm` or `npm`). +- A WorkOS project with User Management enabled and at least one organization + member (needed for UI auth and org IDs). +- Optional: GitHub App for repo onboarding (the backend can help you create one via `/github/setup`). + +## Shared secrets and ports + +- Pick two secrets and reuse them across components: + - `ORCHESTRATOR_BACKEND_SECRET` ≡ `DIGGER_INTERNAL_SECRET` (backend) ≡ UI env. + - `STATESMAN_BACKEND_WEBHOOK_SECRET` ≡ `OPENTACO_ENABLE_INTERNAL_ENDPOINTS` (statesman) ≡ UI env. +- Default ports: backend `3000`, statesman `8080`, UI `3030`. + +## High-level workflow + +1) **Start backend** with internal + API endpoints enabled (so UI can call `/api/*` and `/github/*`). +2) **Start statesman** with internal endpoints enabled; use memory storage for quick start. +3) **Configure UI** `.env.local` with URLs + secrets + WorkOS creds; run `pnpm dev --host --port 3030`. +4) **Sync org/user** into backend and statesman (WorkOS org id and user id/email) via the provided curl snippets in each page. +5) (Optional) **GitHub App**: set `ORCHESTRATOR_GITHUB_APP_URL` to your install URL or `http://localhost:3000/github/setup` to create one via the backend. + +## Troubleshooting cheatsheet + +- **Backend /api/* returns 404**: `DIGGER_ENABLE_API_ENDPOINTS` not `true` or org not upserted. +- **Statesman 403**: webhook secret mismatch. **Statesman 404/500 resolving org**: org not synced (missing `external_org_id`). +- **UI WorkOS auth succeeds but org is empty**: add membership in WorkOS and resync org/user to services. +- **GitHub connect opens 404**: set `ORCHESTRATOR_GITHUB_APP_URL` to a valid install/setup URL. + +Continue with the per-service pages for commands and env examples. diff --git a/docs/ce/local-development/statesman.mdx b/docs/ce/local-development/statesman.mdx new file mode 100644 index 000000000..ac426b1e3 --- /dev/null +++ b/docs/ce/local-development/statesman.mdx @@ -0,0 +1,54 @@ +--- +title: Statesman local setup +--- + +Statesman serves state storage + Terraform Cloud-compatible APIs. The UI uses its internal endpoints, so enable webhook auth and sync your org/user. + +## Quick start + +1. Set env vars: + ```bash + OPENTACO_ENABLE_INTERNAL_ENDPOINTS=statesman-secret # must match UI STATESMAN_BACKEND_WEBHOOK_SECRET + OPENTACO_AUTH_DISABLE=true # skips OIDC for local + OPENTACO_STORAGE=memory # default; uses SQLite query backend automatically + # Optional: OPENTACO_SECRET_KEY for signed URLs; OPENTACO_PORT=8080 + ``` +2. Run the service (from repo root): + ```bash + cd taco + go run cmd/statesman/main.go -storage memory -auth-disable # or ./statesman with the same flags + ``` + Default port: `8080`. + +## Sync org and user (required for UI) + +Statesman resolves orgs by `external_org_id` (your WorkOS org id). If it cannot resolve, `/internal/api/units` will 500. + +```bash +SECRET=$OPENTACO_ENABLE_INTERNAL_ENDPOINTS +ORG_ID=org_xxx # WorkOS org id +ORG_NAME=digger-org # slug to store +ORG_DISPLAY="Digger Org" +USER_ID=user_xxx # WorkOS user id +USER_EMAIL=you@example.com + +# create/sync org +curl -s -X POST http://localhost:8080/internal/api/orgs/sync \ + -H "Authorization: Bearer $SECRET" \ + -H "X-User-ID: $USER_ID" -H "X-Email: $USER_EMAIL" \ + -H "Content-Type: application/json" \ + -d '{"name":"'"$ORG_NAME"'","display_name":"'"$ORG_DISPLAY"'","external_org_id":"'"$ORG_ID"'"}' + +# ensure user exists in that org +curl -s -X POST http://localhost:8080/internal/api/users \ + -H "Authorization: Bearer $SECRET" \ + -H "X-Org-ID: '$ORG_ID'" -H "X-User-ID: $USER_ID" -H "X-Email: $USER_EMAIL" \ + -H "Content-Type: application/json" \ + -d '{"subject":"'"$USER_ID"'","email":"'"$USER_EMAIL"'"}' +``` + +## Troubleshooting + +- **403**: webhook secret mismatch (`OPENTACO_ENABLE_INTERNAL_ENDPOINTS` vs UI `STATESMAN_BACKEND_WEBHOOK_SECRET`). +- **404/500 resolving org**: org not synced; rerun the `orgs/sync` call above. +- **SQLite quirks**: defaults to SQLite in-process; no config needed for local. For Postgres/MySQL, set `TACO_QUERY_BACKEND` and related envs (see `docs/ce/state-management/query-backend`). diff --git a/docs/ce/local-development/ui.mdx b/docs/ce/local-development/ui.mdx new file mode 100644 index 000000000..b5e5768e8 --- /dev/null +++ b/docs/ce/local-development/ui.mdx @@ -0,0 +1,51 @@ +--- +title: UI local setup +--- + +The UI is a TanStack Start app that authenticates via WorkOS and calls both backend and statesman. + +## Quick start + +1. Copy `.env.example` to `.env.local` in `ui/` and fill the essentials: + ```bash + # URLs + PUBLIC_URL=http://localhost:3030 + ALLOWED_HOSTS=localhost,127.0.0.1 + + # WorkOS (User Management) + WORKOS_CLIENT_ID= + WORKOS_API_KEY= + WORKOS_COOKIE_PASSWORD=<32+ random chars> + WORKOS_REDIRECT_URI=http://localhost:3030/api/auth/callback + WORKOS_WEBHOOK_SECRET= + + # Backend + ORCHESTRATOR_BACKEND_URL=http://localhost:3000 + ORCHESTRATOR_BACKEND_SECRET=orchestrator-secret # matches backend DIGGER_INTERNAL_SECRET + ORCHESTRATOR_GITHUB_APP_URL=http://localhost:3000/github/setup # or your app install URL + + # Statesman + STATESMAN_BACKEND_URL=http://localhost:8080 + STATESMAN_BACKEND_WEBHOOK_SECRET=statesman-secret # matches OPENTACO_ENABLE_INTERNAL_ENDPOINTS + + # Optional + DRIFT_REPORTING_BACKEND_URL=http://localhost:3004 + DRIFT_REPORTING_BACKEND_WEBHOOK_SECRET=... + POSTHOG_KEY=... + ``` + In WorkOS, add `http://localhost:3030/api/auth/callback` as a redirect. +2. Install deps and run: + ```bash + cd ui + pnpm install # or npm install + pnpm dev --host --port 3030 + ``` + Open `http://localhost:3030` and sign in with a WorkOS user that belongs to at least one org. +3. Ensure backend + statesman were started and the same secrets are in place (see [Backend](./backend) and [Statesman](./statesman)). +4. Sync the WorkOS org/user to both services using the curl snippets on those pages (required for repos/units to load). + +## Common errors + +- **NotFound/Forbidden listing units**: statesman org/user not synced or webhook secret mismatch. +- **404 on repos or GitHub connect**: backend missing org/user, `DIGGER_ENABLE_API_ENDPOINTS` not set, or `ORCHESTRATOR_GITHUB_APP_URL` points to a non-existent path. +- **WorkOS login succeeds but dashboard redirects to / or errors**: the signed-in user has no WorkOS org membership; add to an org and resync to services. diff --git a/docs/docs.json b/docs/docs.json index 3ecc2bbdf..73c5aafc4 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -165,6 +165,15 @@ "ce/azure-specific/azure-devops-locking-connection-methods" ] }, + { + "group": "Local Development", + "pages": [ + "ce/local-development/overview", + "ce/local-development/backend", + "ce/local-development/statesman", + "ce/local-development/ui" + ] + }, { "group": "Contributing", "pages": [ @@ -213,4 +222,4 @@ "linkedin": "https://www.linkedin.com/company/diggerhq/" } } -} \ No newline at end of file +}