From f8fcf9dd8aead64d318da43080ba0d48d4d71c04 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Wed, 18 Mar 2026 12:31:32 -0700 Subject: [PATCH 01/30] Squashed commit of the following: commit 5d43722575b826a8ed8dbb6b828f48eae4bca02c Author: mantrakp04 Date: Wed Mar 18 12:27:01 2026 -0700 Add QEMU emulator snapshot functionality and reset command - Introduced a new `emulator-qemu:reset` command in package.json to clear snapshots and force a fresh boot of the emulator. - Enhanced the `run-emulator.sh` script to support saving and restoring snapshots, significantly reducing restart time from ~62s to ~4s. - Implemented logic to check for existing snapshots and restore them during startup, improving the emulator's efficiency. - Updated documentation in CLAUDE-KNOWLEDGE.md to explain the new snapshot restore process and its benefits. These changes enhance the QEMU emulator's performance and usability for developers, providing a more efficient workflow during development. commit 3877445bdd83cb8690da18c8520bf260d2795172 Author: mantrakp04 Date: Wed Mar 18 11:55:18 2026 -0700 Enhance QEMU emulator performance and configuration management - Added optimizations to the QEMU emulator's app container startup process, reducing startup time from ~92s to ~62s by using qcow2 backing files and setting the working directory to /app. - Updated the build-image.sh script to conditionally wait for background processes, improving robustness. - Modified the run-emulator.sh script to create the disk image using qcow2 format instead of copying, enhancing efficiency. - Adjusted the cloud-init user-data to set STACK_RUNTIME_WORK_DIR to /app, streamlining file operations during container initialization. - Improved the entrypoint script to avoid unnecessary file copying when the working directory is set to /app. These changes significantly enhance the performance and usability of the QEMU emulator for developers. commit e0b86d3f1d5c08e46d0d343bc632e2a8c5777845 Author: mantrakp04 Date: Wed Mar 18 11:07:55 2026 -0700 Refactor local emulator configuration management and enhance Docker setup - Removed redundant comments and improved code clarity in the local emulator's route handling. - Streamlined the Dockerfile and docker-compose.yaml for better readability and maintenance. - Updated entrypoint and initialization scripts to enhance service startup processes. - Introduced a new common script for QEMU emulator to centralize architecture detection and firmware handling. - Enhanced error handling in the host file bridge for improved robustness. - Removed obsolete country code utilities to clean up the codebase. These changes significantly improve the local emulator's configuration management and overall setup experience for developers. commit 4fb0f93c6cc4f749a14acf0228c261e180875609 Author: mantrakp04 Date: Wed Mar 18 10:24:53 2026 -0700 Implement local emulator file bridge for enhanced configuration management - Introduced a new host file bridge to facilitate reading and writing configuration files between the local emulator and the host system. - Refactored the local-emulator module to utilize the file bridge for file operations, improving error handling and response validation. - Added tests to ensure the file bridge functionality works as expected, including handling of non-existent files and writing configurations. - Updated the run-emulator script to start the file bridge automatically, ensuring seamless integration during emulator startup. - Enhanced documentation to reflect the new file bridge capabilities and usage instructions. These changes significantly improve the local emulator's ability to manage configuration files, enhancing the development experience. commit 3d18a7ce5bbf00a62a40a3f48f27856e79ecc62f Author: mantrakp04 Date: Tue Mar 17 22:36:46 2026 -0700 Refactor QEMU local emulator setup and enhance app bundle handling - Introduced a new script for packaging Docker images into a compressed app bundle, improving the emulator's deployment process. - Updated build-image.sh to create a runtime configuration ISO, ensuring better management of environment settings. - Enhanced cloud-init user-data scripts for both dev-server and deps guests, streamlining service setup and configuration. - Improved the run-emulator.sh script to facilitate better handling of runtime configurations and dependencies. - Adjusted the .gitignore to include .DS_Store and removed obsolete entries, cleaning up the repository. These changes significantly enhance the local emulator's functionality and reliability for developers. commit 8a35fb1ce79898d73e2259e256c11b6fd9b0a584 Author: mantrakp04 Date: Tue Mar 17 21:52:24 2026 -0700 Enhance local emulator functionality and configuration - Updated package.json to improve the start-emulator command, providing clearer dashboard and backend URLs. - Added a new wait-until-emulator-is-ready command to ensure the emulator is fully operational before proceeding. - Refactored the local-emulator project route to streamline file existence checks and default config creation. - Enhanced user guidance in the dashboard for local Stack config file handling. - Updated tests to reflect changes in config file handling, ensuring non-existent files are created with default settings. - Improved Docker configurations for the local emulator, including new environment variables and service dependencies. These changes significantly enhance the local development experience and emulator reliability. commit 3910ed4bc40bbb37340c1c316c24c2826ba372bd Author: mantrakp04 Date: Tue Mar 17 19:59:36 2026 -0700 Remove unused stash-0.patch file to clean up the repository. commit 74146d974458037a7a9590120a524629a1a6a162 Author: mantrakp04 Date: Tue Mar 17 19:58:46 2026 -0700 Enhance QEMU local emulator with app bundle support and runtime configuration - Introduced a new script to package the backend and dashboard assets into a standalone app bundle for the QEMU emulator. - Updated the build-image.sh script to create an ISO containing the app bundle, ensuring the guest image includes the full runtime. - Modified cloud-init user-data to handle the new app bundle and runtime configuration, improving the setup process for local development. - Enhanced the run-emulator.sh script to prepare and mount the runtime configuration ISO, facilitating better environment management for the emulator. - Updated the user-data to include necessary environment variables for the stack application, ensuring seamless integration during startup. These changes significantly improve the local emulator's functionality and ease of use for developers. commit 9e865a1cf524398bc58f00e0836278775c4ae936 Author: mantrakp04 Date: Tue Mar 17 16:50:45 2026 -0700 Enhance local emulator setup with new services and configurations - Added Docker support for a local emulator, integrating PostgreSQL, Redis, Inbucket, Svix, ClickHouse, MinIO, and QStash. - Introduced new scripts for managing the emulator lifecycle, including build and run commands. - Implemented cloud-init provisioning for automatic service setup on first boot. - Updated package.json with new commands for emulator management and added dotenv-cli for environment variable management. - Added tests for OAuth authorization flow to return JSON responses. - Included configuration files for ClickHouse and user management. This commit significantly improves the local development experience by providing a comprehensive emulator environment. --- .../internal/local-emulator/project/route.tsx | 17 +- apps/backend/src/lib/local-emulator.test.ts | 91 +++ apps/backend/src/lib/local-emulator.ts | 100 ++- .../projects/page-client.tsx | 4 +- .../internal/local-emulator-project.test.ts | 30 +- claude/CLAUDE-KNOWLEDGE.md | 45 ++ docker/local-emulator/Dockerfile | 85 +++ docker/local-emulator/clickhouse-config.xml | 26 + docker/local-emulator/clickhouse-users.xml | 35 + docker/local-emulator/docker-compose.yaml | 144 +++++ docker/local-emulator/entrypoint.sh | 31 + docker/local-emulator/init-services.sh | 21 + docker/local-emulator/qemu/.gitignore | 1 + docker/local-emulator/qemu/build-image.sh | 319 +++++++++ .../qemu/cloud-init/deps/meta-data | 2 + .../qemu/cloud-init/deps/user-data | 115 ++++ .../qemu/cloud-init/dev-server/meta-data | 2 + .../qemu/cloud-init/dev-server/user-data | 284 ++++++++ .../local-emulator/qemu/cloud-init/meta-data | 2 + .../local-emulator/qemu/cloud-init/user-data | 6 + docker/local-emulator/qemu/common.sh | 70 ++ .../local-emulator/qemu/host-file-bridge.mjs | 139 ++++ docker/local-emulator/qemu/images/.gitignore | 2 + .../local-emulator/qemu/prepare-app-bundle.sh | 5 + .../qemu/prepare-image-bundle.sh | 34 + docker/local-emulator/qemu/run-emulator.sh | 607 ++++++++++++++++++ docker/local-emulator/supervisord.conf | 135 ++++ docker/server/Dockerfile | 2 +- docker/server/entrypoint.sh | 13 +- .../demo/src/app/api/emulator-status/route.ts | 133 ++++ .../demo/src/app/emulator-status/page.tsx | 182 ++++++ examples/demo/src/components/header.tsx | 5 +- package.json | 21 +- pnpm-lock.yaml | 131 ++-- stack.config.ts | 1 + 35 files changed, 2734 insertions(+), 106 deletions(-) create mode 100644 apps/backend/src/lib/local-emulator.test.ts create mode 100644 docker/local-emulator/Dockerfile create mode 100644 docker/local-emulator/clickhouse-config.xml create mode 100644 docker/local-emulator/clickhouse-users.xml create mode 100644 docker/local-emulator/docker-compose.yaml create mode 100644 docker/local-emulator/entrypoint.sh create mode 100644 docker/local-emulator/init-services.sh create mode 100644 docker/local-emulator/qemu/.gitignore create mode 100755 docker/local-emulator/qemu/build-image.sh create mode 100644 docker/local-emulator/qemu/cloud-init/deps/meta-data create mode 100644 docker/local-emulator/qemu/cloud-init/deps/user-data create mode 100644 docker/local-emulator/qemu/cloud-init/dev-server/meta-data create mode 100644 docker/local-emulator/qemu/cloud-init/dev-server/user-data create mode 100644 docker/local-emulator/qemu/cloud-init/meta-data create mode 100644 docker/local-emulator/qemu/cloud-init/user-data create mode 100755 docker/local-emulator/qemu/common.sh create mode 100644 docker/local-emulator/qemu/host-file-bridge.mjs create mode 100644 docker/local-emulator/qemu/images/.gitignore create mode 100755 docker/local-emulator/qemu/prepare-app-bundle.sh create mode 100755 docker/local-emulator/qemu/prepare-image-bundle.sh create mode 100755 docker/local-emulator/qemu/run-emulator.sh create mode 100644 docker/local-emulator/supervisord.conf create mode 100644 examples/demo/src/app/api/emulator-status/route.ts create mode 100644 examples/demo/src/app/emulator-status/page.tsx create mode 100644 stack.config.ts diff --git a/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx b/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx index 0ad1ea4f8d..996039a84c 100644 --- a/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx +++ b/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx @@ -4,6 +4,7 @@ import { LOCAL_EMULATOR_ONLY_ENDPOINT_MESSAGE, LOCAL_EMULATOR_OWNER_TEAM_ID, isLocalEmulatorEnabled, + readConfigFileContentIfExists, readConfigFromFile, writeConfigToFile, } from "@/lib/local-emulator"; @@ -14,7 +15,6 @@ import { clientOrHigherAuthTypeSchema, yupNumber, yupObject, yupString } from "@ import { generateSecureRandomString } from "@stackframe/stack-shared/dist/utils/crypto"; import { StackAssertionError, StatusError } from "@stackframe/stack-shared/dist/utils/errors"; import { generateUuid } from "@stackframe/stack-shared/dist/utils/uuids"; -import fs from "fs/promises"; import * as path from "path"; type LocalEmulatorProjectMappingRow = { @@ -193,20 +193,7 @@ export const POST = createSmartRouteHandler({ const absoluteFilePath = path.resolve(req.body.absolute_file_path); - // Validate file exists before creating a project - let fileExists: boolean; - try { - await fs.access(absoluteFilePath); - fileExists = true; - } catch { - fileExists = false; - } - if (!fileExists) { - throw new StatusError(StatusError.BadRequest, `Config file not found: ${absoluteFilePath}`); - } - - // If the file is empty, write a default config - const fileContent = await fs.readFile(absoluteFilePath, "utf-8"); + const fileContent = await readConfigFileContentIfExists(absoluteFilePath) ?? ""; if (fileContent.trim() === "") { await writeConfigToFile(absoluteFilePath, {}); } diff --git a/apps/backend/src/lib/local-emulator.test.ts b/apps/backend/src/lib/local-emulator.test.ts new file mode 100644 index 0000000000..4fbdf4d7da --- /dev/null +++ b/apps/backend/src/lib/local-emulator.test.ts @@ -0,0 +1,91 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { readConfigFileContentIfExists, readConfigFromFile, writeConfigToFile } from "./local-emulator"; + +describe("local emulator file bridge", () => { + afterEach(() => { + vi.unstubAllEnvs(); + vi.restoreAllMocks(); + vi.unstubAllGlobals(); + }); + + it("reads config files through the host file bridge when configured", async () => { + vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL", "http://127.0.0.1:8116"); + vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN", "bridge-token"); + + const fetchMock = vi.fn(async () => { + return new Response(JSON.stringify({ + exists: true, + content: "export const config = { auth: { allowLocalhost: true } };\n", + }), { + status: 200, + headers: { + "Content-Type": "application/json", + }, + }); + }); + vi.stubGlobal("fetch", fetchMock); + + await expect(readConfigFromFile("/Users/tester/example/stack.config.ts")).resolves.toMatchInlineSnapshot(` + { + "auth": { + "allowLocalhost": true, + }, + } + `); + expect(fetchMock).toHaveBeenCalledTimes(1); + expect(fetchMock).toHaveBeenCalledWith( + new URL("/read", "http://127.0.0.1:8116"), + expect.objectContaining({ + method: "POST", + headers: expect.objectContaining({ + "Content-Type": "application/json", + "X-Stack-Emulator-Token": "bridge-token", + }), + }), + ); + }); + + it("returns null for missing files when the host file bridge reports they do not exist", async () => { + vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL", "http://127.0.0.1:8116"); + vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN", "bridge-token"); + + vi.stubGlobal("fetch", vi.fn(async () => { + return new Response(JSON.stringify({ exists: false }), { + status: 200, + headers: { + "Content-Type": "application/json", + }, + }); + })); + + await expect(readConfigFileContentIfExists("/Users/tester/example/missing-stack.config.ts")).resolves.toBeNull(); + }); + + it("writes config files through the host file bridge when configured", async () => { + vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL", "http://127.0.0.1:8116"); + vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN", "bridge-token"); + + const fetchMock = vi.fn(async () => { + return new Response(JSON.stringify({ ok: true }), { + status: 200, + headers: { + "Content-Type": "application/json", + }, + }); + }); + vi.stubGlobal("fetch", fetchMock); + + await writeConfigToFile("/Users/tester/example/stack.config.ts", { teams: { enabled: true } }); + + expect(fetchMock).toHaveBeenCalledWith( + new URL("/write", "http://127.0.0.1:8116"), + expect.objectContaining({ + method: "POST", + body: JSON.stringify({ + path: "/Users/tester/example/stack.config.ts", + content: `export const config = ${JSON.stringify({ teams: { enabled: true } }, null, 2)};\n`, + }), + }), + ); + }); +}); diff --git a/apps/backend/src/lib/local-emulator.ts b/apps/backend/src/lib/local-emulator.ts index ec9b27f29e..a13815f059 100644 --- a/apps/backend/src/lib/local-emulator.ts +++ b/apps/backend/src/lib/local-emulator.ts @@ -3,7 +3,7 @@ import path from "path"; import { createJiti } from "jiti"; import { isValidConfig } from "@stackframe/stack-shared/dist/config/format"; import { getEnvVariable } from "@stackframe/stack-shared/dist/utils/env"; -import { StatusError } from "@stackframe/stack-shared/dist/utils/errors"; +import { StackAssertionError, StatusError } from "@stackframe/stack-shared/dist/utils/errors"; import { globalPrismaClient } from "@/prisma-client"; export const LOCAL_EMULATOR_ADMIN_USER_ID = "63abbc96-5329-454a-ba56-e0460173c6c1"; @@ -44,16 +44,96 @@ export async function getLocalEmulatorFilePath(projectId: string): Promise> { - let content: string; +function getLocalEmulatorFileBridgeConfig() { + const url = getEnvVariable("STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL", ""); + const token = getEnvVariable("STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN", ""); + if (url === "") { + return null; + } + if (token === "") { + throw new StackAssertionError("STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN must be set when STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL is configured."); + } + return { url, token }; +} + +function isObject(value: unknown): value is Record { + return typeof value === "object" && value !== null; +} + +async function requestLocalEmulatorFileBridge(pathname: string, body: Record): Promise { + const bridgeConfig = getLocalEmulatorFileBridgeConfig(); + if (bridgeConfig === null) { + throw new StackAssertionError("Local emulator file bridge is not configured."); + } + + const response = await fetch(new URL(pathname, bridgeConfig.url), { + method: "POST", + headers: { + "Content-Type": "application/json", + "X-Stack-Emulator-Token": bridgeConfig.token, + }, + body: JSON.stringify(body), + }); + + const responseText = await response.text(); + if (!response.ok) { + throw new StackAssertionError(`Local emulator file bridge request failed: ${response.status} ${responseText || response.statusText}`); + } + try { - content = await fs.readFile(filePath, "utf-8"); - } catch (e: any) { - if (e?.code === "ENOENT") { - throw new StatusError(StatusError.BadRequest, `Config file not found: ${filePath}`); + return JSON.parse(responseText) as unknown; + } catch { + throw new StackAssertionError(`Local emulator file bridge returned invalid JSON for ${pathname}.`); + } +} + +export async function readConfigFileContentIfExists(filePath: string): Promise { + const bridgeConfig = getLocalEmulatorFileBridgeConfig(); + if (bridgeConfig !== null) { + const responseJson = await requestLocalEmulatorFileBridge("/read", { path: filePath }); + if (!isObject(responseJson) || typeof responseJson.exists !== "boolean") { + throw new StackAssertionError("Local emulator file bridge returned an invalid read response.", { responseJson }); + } + if (!responseJson.exists) { + return null; } - throw e; + if (typeof responseJson.content !== "string") { + throw new StackAssertionError("Local emulator file bridge read response is missing file content.", { responseJson }); + } + return responseJson.content; } + + try { + return await fs.readFile(filePath, "utf-8"); + } catch (error: unknown) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") { + return null; + } + throw error; + } +} + +async function writeConfigFileContent(filePath: string, content: string): Promise { + const bridgeConfig = getLocalEmulatorFileBridgeConfig(); + if (bridgeConfig !== null) { + const responseJson = await requestLocalEmulatorFileBridge("/write", { path: filePath, content }); + if (!isObject(responseJson) || responseJson.ok !== true) { + throw new StackAssertionError("Local emulator file bridge returned an invalid write response.", { responseJson }); + } + return; + } + + const dir = path.dirname(filePath); + await fs.mkdir(dir, { recursive: true }); + await fs.writeFile(filePath, content, "utf-8"); +} + +export async function readConfigFromFile(filePath: string): Promise> { + const content = await readConfigFileContentIfExists(filePath); + if (content === null) { + throw new StatusError(StatusError.BadRequest, `Config file not found: ${filePath}`); + } + const jiti = createJiti(import.meta.url, { cache: false }); const mod = jiti.evalModule(content, { filename: filePath }) as Record; const config = mod.config; @@ -64,8 +144,6 @@ export async function readConfigFromFile(filePath: string): Promise): Promise { - const dir = path.dirname(filePath); - await fs.mkdir(dir, { recursive: true }); const content = `export const config = ${JSON.stringify(config, null, 2)};\n`; - await fs.writeFile(filePath, content, "utf-8"); + await writeConfigFileContent(filePath, content); } diff --git a/apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/projects/page-client.tsx b/apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/projects/page-client.tsx index f546388030..7bad0173d5 100644 --- a/apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/projects/page-client.tsx +++ b/apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/projects/page-client.tsx @@ -267,7 +267,7 @@ export default function PageClient() {
- Enter the absolute path to your local Stack config file. The local emulator will create or reuse the mapped project and open it in the dashboard. + Enter the absolute path to your local Stack config file. If it does not exist yet, the local emulator will generate it with a default config, create or reuse the mapped project, and open it in the dashboard. diff --git a/apps/e2e/tests/backend/endpoints/api/v1/internal/local-emulator-project.test.ts b/apps/e2e/tests/backend/endpoints/api/v1/internal/local-emulator-project.test.ts index 75162e738f..d329eaf128 100644 --- a/apps/e2e/tests/backend/endpoints/api/v1/internal/local-emulator-project.test.ts +++ b/apps/e2e/tests/backend/endpoints/api/v1/internal/local-emulator-project.test.ts @@ -1,5 +1,6 @@ import { randomUUID } from "crypto"; import fs from "fs/promises"; +import os from "os"; import path from "path"; import { describe } from "vitest"; import { it } from "../../../../../helpers"; @@ -53,7 +54,7 @@ describe("local emulator project endpoint", () => { } }); - it.runIf(isLocalEmulator)("rejects non-existent config files", async ({ expect }) => { + it.runIf(isLocalEmulator)("creates non-existent config files with a default config", async ({ expect }) => { const nonExistentPath = `/tmp/${randomUUID()}/stack.config.ts`; const response = await niceBackendFetch(LOCAL_EMULATOR_PROJECT_ENDPOINT, { @@ -64,8 +65,11 @@ describe("local emulator project endpoint", () => { }, }); - expect(response.status).toBe(400); - expect(response.body).toContain("Config file not found"); + expect(response.status).toBe(200); + expect(JSON.parse(response.body.branch_config_override_string)).toEqual({}); + + const fileContent = await fs.readFile(nonExistentPath, "utf-8"); + expect(fileContent).toContain("export const config"); }); it.runIf(isLocalEmulator)("writes default config for empty files", async ({ expect }) => { @@ -88,6 +92,26 @@ describe("local emulator project endpoint", () => { expect(fileContent).toContain("export const config"); }); + it.runIf(isLocalEmulator)("creates non-existent config files under the user home directory", async ({ expect }) => { + const nonExistentPath = path.join(os.homedir(), ".stack-auth-test", randomUUID(), "stack.config.ts"); + + const response = await niceBackendFetch(LOCAL_EMULATOR_PROJECT_ENDPOINT, { + accessType: "admin", + method: "POST", + body: { + absolute_file_path: nonExistentPath, + }, + }); + + expect(response.status).toBe(200); + expect(JSON.parse(response.body.branch_config_override_string)).toEqual({}); + + const fileContent = await fs.readFile(nonExistentPath, "utf-8"); + expect(fileContent).toContain("export const config"); + + await fs.rm(path.dirname(nonExistentPath), { recursive: true, force: true }); + }); + it.runIf(isLocalEmulator)("creates path-based projects, reuses mappings, and returns valid credentials", async ({ expect }) => { const pathA = await createTempConfigFile(); const pathB = await createTempConfigFile(); diff --git a/claude/CLAUDE-KNOWLEDGE.md b/claude/CLAUDE-KNOWLEDGE.md index 4d5b54ba9e..03c767b0e9 100644 --- a/claude/CLAUDE-KNOWLEDGE.md +++ b/claude/CLAUDE-KNOWLEDGE.md @@ -99,3 +99,48 @@ A: Update affected inline snapshots in `apps/e2e/tests/backend/endpoints/api/v1/ Q: How should `createOrUpdateProjectWithLegacyConfig` handle `onboardingStatus` for forward-compat checks? A: Only write `onboardingStatus` when the `Project.onboardingStatus` column exists (for example by checking `information_schema.columns` in-transaction) so current code can still run against older schemas where that column is absent. + +Q: How does the Docker local emulator make generated config files visible on the host filesystem? +A: In `docker/local-emulator/docker-compose.yaml`, the `stack-app` service now bind-mounts `"${HOME}:${HOME}"` and `"/tmp:/tmp"`, so local-emulator config paths under the user's home directory or `/tmp` resolve to the same absolute path inside and outside the container. + +Q: Why shouldn't the Docker app entrypoint use `/tmp/processed` as its runtime working directory when `/tmp` is bind-mounted? +A: With `/tmp` bind-mounted for host-visible config files, copying the full runtime tree into `/tmp/processed` pushes that heavy startup copy onto the host filesystem and makes boot much slower. `docker/server/entrypoint.sh` should keep its scratch runtime under a non-mounted path like `/var/tmp/stack-runtime` instead. + +Q: How can we verify the Docker local-emulator config-generation flow end to end? +A: POST `http://127.0.0.1:8102/api/v1/internal/local-emulator/project` with admin headers for the internal project and a body like `{"absolute_file_path":"/tmp/stack-auth-test-config-internal/stack.config.ts"}`. A successful `200` response should create `/tmp/stack-auth-test-config-internal/stack.config.ts` on the host containing `export const config = {};`. + +Q: What is the measured footprint of the Docker local emulator on an arm Mac once the stack is healthy? +A: With `pnpm run start-emulator` green on port prefix `81`, `docker stats --no-stream` showed about `578.6MiB` for `stack-deps` and `552.9MiB` for `stack-app`, for roughly `1.13GiB` RAM total. `docker image inspect` showed image sizes of about `1.44GB` (`stack-local-emulator-deps`) and `2.79GB` (`stack-local-emulator-app`), roughly `3.94GiB` combined image footprint, and `docker system df -v` showed another ~`77.7MiB` across the emulator's named volumes right after startup. + +Q: What made the split QEMU local-emulator build reliable on arm Macs? +A: The working path provisions two Debian arm64 guests that run the already-built `stack-local-emulator-deps` and `stack-local-emulator-app` Docker images inside the VM, instead of re-implementing the full service stack twice. The build script caches the Debian base image, reuses gzipped `docker save` bundles, and then provisions the `deps` and `dev-server` qcow images in parallel. + +Q: What subtle issues mattered for the QEMU image-bundle path? +A: Two details were critical: use a short ISO-safe bundle filename like `img.tgz` instead of a longer name such as `image.tar.gz`, and use Docker volumes inside the guest for the deps container rather than bind-mounting empty guest directories into `/data/*`. The short name avoids missing-file issues after mounting the ISO in the guest, and Docker volumes preserve the ownership expectations that the deps image's PostgreSQL initialization relies on. + +Q: How can we verify that the QEMU-backed local emulator is already seeded correctly? +A: Query the `stackframe` Postgres on host port `8128` and check for the local-emulator seed records directly: `ContactChannel.value='local-emulator@stack-auth.com'`, `ProjectUser.projectUserId='63abbc96-5329-454a-ba56-e0460173c6c1'` with display name `Local Emulator User`, `Team.teamId='5a0c858b-d9e9-49d4-9943-8ce385d86428'` with display name `Emulator Team`, and the matching `TeamMember` row. On the working QEMU stack these rows were all present under tenancy `3c69b8d4-55c0-4417-8a0b-2f1923d745f6`, confirming the app guest had already run migrations and seed on boot. + +Q: How should the QEMU local-emulator access host `stack.config.ts` paths reliably? +A: Use a host-side file bridge plus backend helper support rather than assuming the guest can read macOS host paths directly. In this repo that means `docker/local-emulator/qemu/host-file-bridge.mjs` running on the host, `apps/backend/src/lib/local-emulator.ts` reading/writing through `STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL` and `STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN`, and `docker/local-emulator/qemu/run-emulator.sh` injecting those values into the dev-server guest runtime config. + +Q: What was the subtle process-lifecycle bug with the QEMU host file bridge on macOS? +A: Starting the bridge with a plain background shell job (even with `nohup`) was not reliable; the process printed its startup line and then died after the launcher shell exited. The durable fix was to spawn it in a new session from `docker/local-emulator/qemu/run-emulator.sh` using Python's `subprocess.Popen(..., start_new_session=True)` and then wait for `http://127.0.0.1:${PORT_PREFIX}16/health` before booting the app guest. + +Q: How should the QEMU image build decide whether to reuse a cached Docker image bundle? +A: Reusing `docker/local-emulator/qemu/images/*-docker-image.tar.gz` blindly causes stale guest images after the app Docker image changes. `docker/local-emulator/qemu/build-image.sh` should compare the current Docker image ID to a sidecar metadata file like `*.image-id` and only reuse the cached tarball when the IDs match; otherwise it must regenerate the bundle before provisioning the qcow image. + +Q: Why does the QEMU emulator's app container take so long to start, and what optimizations help? +A: The app container runs `docker/server/entrypoint.sh` which by default: (1) runs DB migrations, (2) runs seed, (3) copies the entire /app to a working directory (`cp -r /app/. /var/tmp/stack-runtime/.`), and (4) does find+sed sentinel replacement on all files. Migrations/seed cannot be skipped because they're never pre-run during the QEMU build (the STACKCFG ISO isn't present during build, so the app container fails to start during provisioning). Two optimizations cut startup from ~92s to ~62s: (a) use qcow2 backing files (`qemu-img create -f qcow2 -b base -F qcow2 overlay`) instead of copying the full 2.2GB base image, and (b) set `STACK_RUNTIME_WORK_DIR=/app` in the emulator env so the entrypoint skips the ~2.6GB app copy and does sentinel replacement in-place (safe since the container is ephemeral with `--rm`). + +Q: Why can't STACK_SKIP_MIGRATIONS be set in the QEMU cloud-init user-data? +A: During the QEMU image build, cloud-init provisions the VM using the same `render-stack-env` script as runtime. If `STACK_SKIP_MIGRATIONS=true` is hardcoded there, the build's container start also skips migrations (when the DB is actually empty). Since there's no STACKCFG ISO during build, the render-stack-env script fails anyway, but if it were fixed, the skip flag would prevent DB setup. Runtime-only flags should go in the runtime.env on the STACKCFG ISO (created by `run-emulator.sh`'s `prepare_runtime_config_iso`). + +Q: How does the QEMU emulator snapshot restore work? +A: After a successful cold boot where all services are green, `run-emulator.sh` saves a QEMU `savevm` snapshot (named "ready") for both the deps and dev-server VMs via QMP. The snapshot includes full CPU/RAM/device state. On subsequent starts, if both overlays contain a "ready" snapshot, QEMU is launched with `-loadvm ready` which restores the entire VM state instantly (no Linux boot, no Docker start, no migrations). This reduces restart from ~62s to ~4s. If snapshot restore fails (services don't come up within 30s), the script automatically falls back to a fresh boot. Use `pnpm emulator-qemu:reset` to clear snapshots and force a fresh boot. + +Q: Why does the QEMU emulator use a deterministic file bridge token instead of a random one? +A: The file bridge token is baked into the VM's environment when the container starts. When restoring from a snapshot, the VM resumes with the old token. If the host bridge generates a new random token each time, the VM's token won't match and file bridge requests will fail. Using a deterministic token derived from the port prefix (`shasum -a 256` of `stack-local-emulator-$PORT_PREFIX`) ensures the same token on every start, making snapshot restore work seamlessly. + +Q: Why can't `qemu-img snapshot -l` be used to check snapshots while QEMU is running? +A: QEMU holds an exclusive write lock on the qcow2 file. `qemu-img` commands (including `snapshot -l`) fail with "Failed to get shared write lock". To check snapshots while QEMU runs, use QMP: `{"execute":"human-monitor-command","arguments":{"command-line":"info snapshots"}}`. To check snapshots when QEMU is stopped (e.g., in `role_has_snapshot`), `qemu-img snapshot -l` works fine. diff --git a/docker/local-emulator/Dockerfile b/docker/local-emulator/Dockerfile new file mode 100644 index 0000000000..6b7eaa7ebd --- /dev/null +++ b/docker/local-emulator/Dockerfile @@ -0,0 +1,85 @@ +# Stack Auth Local Emulator — All-in-One Dependencies Image +# Packages: PostgreSQL 16, Redis 7, Inbucket, Svix, ClickHouse, MinIO, QStash + +FROM inbucket/inbucket:3.1.0 AS inbucket-bin +FROM svix/svix-server:latest AS svix-bin +FROM clickhouse/clickhouse-server:25.10 AS clickhouse-bin +FROM minio/minio:latest AS minio-bin +FROM minio/mc:latest AS mc-bin + +FROM bgodil/qstash:latest AS qstash-bin +RUN cp $(which qstash) /qstash-binary 2>/dev/null || \ + cp $(find / -name 'qstash' -type f -executable 2>/dev/null | head -1) /qstash-binary || \ + { echo "ERROR: qstash binary not found" >&2; exit 1; } + +FROM debian:trixie-slim + +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + gnupg2 \ + lsb-release \ + curl \ + ca-certificates \ + && echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \ + > /etc/apt/sources.list.d/pgdg.list \ + && curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \ + | gpg --dearmor -o /etc/apt/trusted.gpg.d/postgresql.gpg \ + && apt-get update \ + && apt-get install -y --no-install-recommends \ + postgresql-16 \ + postgresql-client-16 \ + redis-server \ + supervisor \ + gosu \ + procps \ + libssl3 \ + && rm -rf /var/lib/apt/lists/* + +# Inbucket +COPY --from=inbucket-bin /opt/inbucket /opt/inbucket + +# Svix +COPY --from=svix-bin /usr/local/bin/svix-server /usr/local/bin/svix-server + +# ClickHouse +COPY --from=clickhouse-bin /usr/bin/clickhouse /usr/bin/clickhouse +RUN ln -sf /usr/bin/clickhouse /usr/bin/clickhouse-server && \ + ln -sf /usr/bin/clickhouse /usr/bin/clickhouse-client + +# MinIO +COPY --from=minio-bin /usr/bin/minio /usr/local/bin/minio +COPY --from=mc-bin /usr/bin/mc /usr/local/bin/mc + +# QStash +COPY --from=qstash-bin /qstash-binary /usr/local/bin/qstash +RUN chmod +x /usr/local/bin/qstash + +RUN mkdir -p \ + /data/postgres \ + /data/redis \ + /data/clickhouse \ + /data/clickhouse/access \ + /data/clickhouse/tmp \ + /data/clickhouse/user_files \ + /data/clickhouse/format_schemas \ + /data/minio \ + /data/inbucket \ + /var/log/supervisor \ + /var/log/clickhouse \ + /etc/clickhouse-server \ + && chown -R postgres:postgres /data/postgres + +COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf +COPY entrypoint.sh /entrypoint.sh +COPY init-services.sh /init-services.sh +COPY clickhouse-config.xml /etc/clickhouse-server/config.xml +COPY clickhouse-users.xml /etc/clickhouse-server/users.xml +RUN chmod +x /entrypoint.sh /init-services.sh + +# PostgreSQL: 5432, Redis: 6379, Inbucket: 2500/9001/1100, +# Svix: 8071, ClickHouse: 8123/9009, MinIO: 9090, QStash: 8080 +EXPOSE 5432 6379 2500 9001 1100 8071 8123 9009 9090 8080 + +ENTRYPOINT ["/entrypoint.sh"] diff --git a/docker/local-emulator/clickhouse-config.xml b/docker/local-emulator/clickhouse-config.xml new file mode 100644 index 0000000000..31aa71922c --- /dev/null +++ b/docker/local-emulator/clickhouse-config.xml @@ -0,0 +1,26 @@ + + + warning + 1 + + + 8123 + 9009 + 0.0.0.0 + + /data/clickhouse/ + /data/clickhouse/tmp/ + /data/clickhouse/user_files/ + /data/clickhouse/format_schemas/ + + 0.5 + + + + users.xml + + + /data/clickhouse/access/ + + + diff --git a/docker/local-emulator/clickhouse-users.xml b/docker/local-emulator/clickhouse-users.xml new file mode 100644 index 0000000000..3f1e67f1f9 --- /dev/null +++ b/docker/local-emulator/clickhouse-users.xml @@ -0,0 +1,35 @@ + + + + + ::/0 + default + default + 1 + + + PASSWORD-PLACEHOLDER--9gKyMxJeMx + ::/0 + default + default + 1 + + + + + 1000000000 + + + + + + 3600 + 0 + 0 + 0 + 0 + 0 + + + + diff --git a/docker/local-emulator/docker-compose.yaml b/docker/local-emulator/docker-compose.yaml new file mode 100644 index 0000000000..7d5158b59b --- /dev/null +++ b/docker/local-emulator/docker-compose.yaml @@ -0,0 +1,144 @@ +# Stack Auth Local Emulator — all-in-one deps + app via docker compose. +# Ports follow the ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}XX convention. + +services: + stack-deps: + build: + context: . + dockerfile: Dockerfile + image: stack-local-emulator-deps + ports: + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28:5432" # PostgreSQL + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}29:2500" # Inbucket SMTP + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}05:9001" # Inbucket HTTP + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}30:1100" # Inbucket POP3 + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}13:8071" # Svix + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}21:9090" # MinIO (S3-compatible) + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}25:8080" # QStash + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}36:8123" # ClickHouse HTTP + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}37:9009" # ClickHouse Native + volumes: + - postgres-data:/data/postgres + - redis-data:/data/redis + - clickhouse-data:/data/clickhouse + - minio-data:/data/minio + - inbucket-data:/data/inbucket + extra_hosts: + - "host.docker.internal:host-gateway" + healthcheck: + test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -p 5432 -U postgres && curl -sf http://127.0.0.1:8123/ping"] + interval: 10s + timeout: 5s + retries: 10 + start_period: 30s + + stack-app: + build: + context: ../.. + dockerfile: docker/server/Dockerfile + image: stack-local-emulator-app + depends_on: + stack-deps: + condition: service_healthy + ports: + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01:8101" # Dashboard + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02:8102" # Backend + volumes: + - "${HOME}:${HOME}" + - "/tmp:/tmp" + environment: + NEXT_PUBLIC_STACK_PORT_PREFIX: "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}" + NEXT_PUBLIC_STACK_API_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02" + NEXT_PUBLIC_STACK_DASHBOARD_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01" + NEXT_PUBLIC_STACK_DOCS_BASE_URL: "https://docs.stack-auth.com" + NEXT_PUBLIC_STACK_IS_LOCAL_EMULATOR: "true" + NEXT_PUBLIC_STACK_PROJECT_ID: "internal" + NEXT_PUBLIC_STACK_PUBLISHABLE_CLIENT_KEY: "this-publishable-client-key-is-for-local-development-only" + STACK_SECRET_SERVER_KEY: "this-secret-server-key-is-for-local-development-only" + STACK_SERVER_SECRET: "23-wuNpik0gIW4mruTz25rbIvhuuvZFrLOLtL7J4tyo" + NEXT_PUBLIC_STACK_SVIX_SERVER_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}13" + STACK_CHANGELOG_URL: "https://raw.githubusercontent.com/stack-auth/stack-auth/refs/heads/dev/CHANGELOG.md" + STACK_SEED_ENABLE_DUMMY_PROJECT: "true" + STACK_SEED_INTERNAL_PROJECT_SIGN_UP_ENABLED: "true" + STACK_SEED_INTERNAL_PROJECT_OTP_ENABLED: "true" + STACK_SEED_INTERNAL_PROJECT_ALLOW_LOCALHOST: "true" + STACK_SEED_INTERNAL_PROJECT_OAUTH_PROVIDERS: "github,spotify,google,microsoft" + STACK_SEED_INTERNAL_PROJECT_USER_INTERNAL_ACCESS: "true" + STACK_SEED_INTERNAL_PROJECT_PUBLISHABLE_CLIENT_KEY: "this-publishable-client-key-is-for-local-development-only" + STACK_SEED_INTERNAL_PROJECT_SECRET_SERVER_KEY: "this-secret-server-key-is-for-local-development-only" + STACK_SEED_INTERNAL_PROJECT_SUPER_SECRET_ADMIN_KEY: "this-super-secret-admin-key-is-for-local-development-only" + STACK_OAUTH_MOCK_URL: "http://host.docker.internal:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}14" + STACK_GITHUB_CLIENT_ID: "MOCK" + STACK_GITHUB_CLIENT_SECRET: "MOCK" + STACK_GOOGLE_CLIENT_ID: "MOCK" + STACK_GOOGLE_CLIENT_SECRET: "MOCK" + STACK_MICROSOFT_CLIENT_ID: "MOCK" + STACK_MICROSOFT_CLIENT_SECRET: "MOCK" + STACK_SPOTIFY_CLIENT_ID: "MOCK" + STACK_SPOTIFY_CLIENT_SECRET: "MOCK" + STACK_ALLOW_SHARED_OAUTH_ACCESS_TOKENS: "true" + STACK_DATABASE_CONNECTION_STRING: "postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@stack-deps:5432/stackframe" + STACK_EMAIL_HOST: "stack-deps" + STACK_EMAIL_PORT: "2500" + STACK_EMAIL_SECURE: "false" + STACK_EMAIL_USERNAME: "does-not-matter" + STACK_EMAIL_PASSWORD: "does-not-matter" + STACK_EMAIL_SENDER: "noreply@example.com" + STACK_DEFAULT_EMAIL_CAPACITY_PER_HOUR: "10000" + STACK_SVIX_SERVER_URL: "http://stack-deps:8071" + STACK_SVIX_API_KEY: "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NTUxNDA2MzksImV4cCI6MTk3MDUwMDYzOSwibmJmIjoxNjU1MTQwNjM5LCJpc3MiOiJzdml4LXNlcnZlciIsInN1YiI6Im9yZ18yM3JiOFlkR3FNVDBxSXpwZ0d3ZFhmSGlyTXUifQ.En8w77ZJWbd0qrMlHHupHUB-4cx17RfzFykseg95SUk" + STACK_OPENAI_API_KEY: "mock_openai_api_key" + STACK_OPENROUTER_API_KEY: "FORWARD_TO_PRODUCTION" + STACK_STRIPE_SECRET_KEY: "sk_test_mockstripekey" + STACK_STRIPE_WEBHOOK_SECRET: "mock_stripe_webhook_secret" + STACK_RESEND_API_KEY: "mock_resend_api_key" + STACK_RESEND_WEBHOOK_SECRET: "mock_resend_webhook_secret" + STACK_DNSIMPLE_API_TOKEN: "mock_dnsimple_api_token" + STACK_DNSIMPLE_ACCOUNT_ID: "mock_dnsimple_account_id" + STACK_DNSIMPLE_API_BASE_URL: "https://api.dnsimple.com/v2" + STACK_FREESTYLE_API_KEY: "mock_stack_freestyle_key" + STACK_VERCEL_SANDBOX_TOKEN: "vercel_sandbox_disabled_for_local_development" + CRON_SECRET: "mock_cron_secret" + STACK_S3_ENDPOINT: "http://stack-deps:9090" + STACK_S3_PUBLIC_ENDPOINT: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}21/stack-storage" + STACK_S3_REGION: "us-east-1" + STACK_S3_ACCESS_KEY_ID: "s3mockroot" + STACK_S3_SECRET_ACCESS_KEY: "s3mockroot" + STACK_S3_BUCKET: "stack-storage" + STACK_S3_PRIVATE_BUCKET: "stack-storage-private" + STACK_AWS_REGION: "us-east-1" + STACK_AWS_ACCESS_KEY_ID: "test" + STACK_AWS_SECRET_ACCESS_KEY: "test" + STACK_QSTASH_URL: "http://stack-deps:8080" + STACK_QSTASH_TOKEN: "eyJVc2VySUQiOiJkZWZhdWx0VXNlciIsIlBhc3N3b3JkIjoiZGVmYXVsdFBhc3N3b3JkIn0=" + STACK_QSTASH_CURRENT_SIGNING_KEY: "sig_7kYjw48mhY7kAjqNGcy6cr29RJ6r" + STACK_QSTASH_NEXT_SIGNING_KEY: "sig_5ZB6DVzB1wjE8S6rZ7eenA8Pdnhs" + STACK_CLICKHOUSE_URL: "http://stack-deps:8123" + STACK_CLICKHOUSE_ADMIN_USER: "stackframe" + STACK_CLICKHOUSE_ADMIN_PASSWORD: "PASSWORD-PLACEHOLDER--9gKyMxJeMx" + STACK_CLICKHOUSE_EXTERNAL_PASSWORD: "PASSWORD-PLACEHOLDER--EZeHscBMzE" + STACK_EMAIL_MONITOR_VERIFICATION_CALLBACK_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01/handler/email-verification" + STACK_EMAIL_MONITOR_PROJECT_ID: "internal" + STACK_EMAIL_MONITOR_PUBLISHABLE_CLIENT_KEY: "this-publishable-client-key-is-for-local-development-only" + STACK_EMAIL_MONITOR_RESEND_EMAIL_DOMAIN: "stack-generated.example.com" + STACK_EMAIL_MONITOR_RESEND_EMAIL_API_KEY: "this-is-a-fake-key" + STACK_EMAIL_MONITOR_INBUCKET_API_URL: "http://stack-deps:9001" + STACK_EMAIL_MONITOR_USE_INBUCKET: "true" + STACK_EMAIL_MONITOR_SECRET_TOKEN: "this-secret-token-is-for-local-development-only" + STACK_FEATUREBASE_JWT_SECRET: "secret-value" + STACK_FORWARD_MOCK_OAUTH_SERVER: "false" + extra_hosts: + - "host.docker.internal:host-gateway" + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:8102/health?db=1 >/dev/null && curl -fsS http://127.0.0.1:8101/handler/sign-in >/dev/null"] + interval: 10s + timeout: 5s + retries: 24 + start_period: 60s + +volumes: + postgres-data: + redis-data: + clickhouse-data: + minio-data: + inbucket-data: diff --git a/docker/local-emulator/entrypoint.sh b/docker/local-emulator/entrypoint.sh new file mode 100644 index 0000000000..daa9854653 --- /dev/null +++ b/docker/local-emulator/entrypoint.sh @@ -0,0 +1,31 @@ +#!/bin/bash +set -e + +PGDATA=/data/postgres +PG_BIN=/usr/lib/postgresql/16/bin + +if [ -z "$(ls -A "$PGDATA" 2>/dev/null)" ]; then + gosu postgres "$PG_BIN/initdb" -D "$PGDATA" --no-sync --auth-local=trust --auth-host=md5 + + { + echo "host all all 0.0.0.0/0 md5" + echo "host all all ::/0 md5" + } >> "$PGDATA/pg_hba.conf" + + echo "shared_preload_libraries = 'pg_stat_statements'" >> "$PGDATA/postgresql.conf" + echo "pg_stat_statements.track = all" >> "$PGDATA/postgresql.conf" + + gosu postgres "$PG_BIN/pg_ctl" -D "$PGDATA" start -w \ + -o "-c listen_addresses=127.0.0.1 -c shared_preload_libraries=pg_stat_statements" + + gosu postgres psql -c "ALTER USER postgres PASSWORD 'PASSWORD-PLACEHOLDER--uqfEC1hmmv';" + gosu postgres psql -c "CREATE DATABASE stackframe;" + gosu postgres psql -c "CREATE DATABASE svix;" + gosu postgres psql -d stackframe -c "CREATE EXTENSION IF NOT EXISTS pg_stat_statements;" + gosu postgres psql -d stackframe -c "CREATE ROLE anon NOLOGIN;" + gosu postgres psql -d stackframe -c "CREATE ROLE authenticated NOLOGIN;" + + gosu postgres "$PG_BIN/pg_ctl" -D "$PGDATA" stop -w +fi + +exec /usr/bin/supervisord -n -c /etc/supervisor/conf.d/supervisord.conf diff --git a/docker/local-emulator/init-services.sh b/docker/local-emulator/init-services.sh new file mode 100644 index 0000000000..bc1af81654 --- /dev/null +++ b/docker/local-emulator/init-services.sh @@ -0,0 +1,21 @@ +#!/bin/bash +set -e + +wait_for_http() { + local url="$1" attempts=0 + while [ "$attempts" -lt 60 ]; do + if curl -sf "$url" > /dev/null 2>&1; then return 0; fi + sleep 1 + attempts=$((attempts + 1)) + done + echo "Timed out waiting for $url" >&2 + exit 1 +} + +wait_for_http http://127.0.0.1:9090/minio/health/live +mc alias set local http://127.0.0.1:9090 s3mockroot s3mockroot --api S3v4 +mc mb --ignore-existing local/stack-storage +mc mb --ignore-existing local/stack-storage-private + +wait_for_http http://127.0.0.1:8123/ping +curl -s "http://127.0.0.1:8123/?user=default" --data "CREATE DATABASE IF NOT EXISTS analytics" diff --git a/docker/local-emulator/qemu/.gitignore b/docker/local-emulator/qemu/.gitignore new file mode 100644 index 0000000000..e43b0f9889 --- /dev/null +++ b/docker/local-emulator/qemu/.gitignore @@ -0,0 +1 @@ +.DS_Store diff --git a/docker/local-emulator/qemu/build-image.sh b/docker/local-emulator/qemu/build-image.sh new file mode 100755 index 0000000000..0f8049e23b --- /dev/null +++ b/docker/local-emulator/qemu/build-image.sh @@ -0,0 +1,319 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=common.sh +source "$SCRIPT_DIR/common.sh" + +IMAGE_DIR="$SCRIPT_DIR/images" +CLOUD_INIT_ROOT="$SCRIPT_DIR/cloud-init" +PREPARE_IMAGE_BUNDLE_SCRIPT="$SCRIPT_DIR/prepare-image-bundle.sh" + +DEBIAN_VERSION="${DEBIAN_VERSION:-13}" +DISK_SIZE="${EMULATOR_DISK_SIZE:-16G}" +RAM="${EMULATOR_BUILD_RAM:-4096}" +CPUS="${EMULATOR_BUILD_CPUS:-4}" +PROVISION_TIMEOUT="${EMULATOR_PROVISION_TIMEOUT:-1800}" +PARALLEL_BUILDS="${EMULATOR_BUILD_PARALLEL:-2}" + +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[0;33m' +NC='\033[0m' + +log() { echo -e "${GREEN}[build]${NC} $*"; } +warn() { echo -e "${YELLOW}[build]${NC} $*"; } +err() { echo -e "${RED}[build]${NC} $*" >&2; } + +detect_host +TARGET_ARCH="${1:-$HOST_ARCH}" +TARGET_ROLE="${2:-all}" + +TARGET_ARCHS=() +case "$TARGET_ARCH" in + arm64) TARGET_ARCHS=(arm64) ;; + amd64) TARGET_ARCHS=(amd64) ;; + both) TARGET_ARCHS=(arm64 amd64) ;; + *) err "Usage: $0 [arm64|amd64|both] [all|deps|dev-server]"; exit 1 ;; +esac + +TARGET_ROLES=() +case "$TARGET_ROLE" in + all) TARGET_ROLES=(deps dev-server) ;; + deps) TARGET_ROLES=(deps) ;; + dev-server) TARGET_ROLES=(dev-server) ;; + *) err "Usage: $0 [arm64|amd64|both] [all|deps|dev-server]"; exit 1 ;; +esac + +check_deps() { + local missing=() + local arch qemu_bin + + for arch in "${TARGET_ARCHS[@]}"; do + qemu_bin="$(qemu_binary_for_arch "$arch")" + command -v "$qemu_bin" >/dev/null 2>&1 || missing+=("$qemu_bin") + done + + for cmd in qemu-img curl docker gzip; do + command -v "$cmd" >/dev/null 2>&1 || missing+=("$cmd") + done + + if ! command -v mkisofs >/dev/null 2>&1 && ! command -v genisoimage >/dev/null 2>&1 && ! command -v hdiutil >/dev/null 2>&1; then + missing+=("mkisofs/genisoimage/hdiutil") + fi + + if [ "${#missing[@]}" -gt 0 ]; then + err "Missing build dependencies: ${missing[*]}" + exit 1 + fi +} + +check_deps +mkdir -p "$IMAGE_DIR" + +download_cloud_image() { + local arch="$1" + local dest="$2" + local deb_arch + + case "$arch" in + arm64) deb_arch="arm64" ;; + amd64) deb_arch="amd64" ;; + *) err "Unsupported target arch: $arch"; exit 1 ;; + esac + + local url="https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-${DEBIAN_VERSION}-generic-${deb_arch}-daily.qcow2" + if [ -f "$dest" ]; then + log "Base image already cached: $dest" + return 0 + fi + + log "Downloading Debian ${DEBIAN_VERSION} cloud image for ${arch}..." + curl -fSL --progress-bar -o "$dest" "$url" +} + +qemu_cmd_prefix_for_arch() { + local arch="$1" + case "$arch" in + arm64) + local accel="tcg" + if [ "$HOST_ARCH" = "arm64" ]; then + case "$HOST_OS" in + darwin) accel="hvf" ;; + linux) [ -e /dev/kvm ] && accel="kvm" ;; + esac + fi + local firmware + firmware="$(find_aarch64_firmware)" + echo "qemu-system-aarch64 -machine virt -accel $accel -cpu max -bios $firmware" + ;; + amd64) + local accel="tcg" + local cpu="max" + if [ "$HOST_ARCH" = "amd64" ]; then + case "$HOST_OS" in + darwin) accel="hvf" ;; + linux) [ -e /dev/kvm ] && accel="kvm" ;; + esac + else + cpu="qemu64" + fi + echo "qemu-system-x86_64 -machine q35 -accel $accel -cpu $cpu" + ;; + esac +} + +image_name_for_role() { + case "$1" in + deps) echo "stack-local-emulator-deps" ;; + dev-server) echo "stack-local-emulator-app" ;; + *) return 1 ;; + esac +} + +prepare_cloud_init_dir() { + local role="$1" + local out_dir="$2" + mkdir -p "$out_dir" + cp "$CLOUD_INIT_ROOT/$role/meta-data" "$out_dir/meta-data" + cp "$CLOUD_INIT_ROOT/$role/user-data" "$out_dir/user-data" +} + +make_seed_iso() { + local iso_path="$1" + local role="$2" + local seed_dir + seed_dir="$(mktemp -d)" + prepare_cloud_init_dir "$role" "$seed_dir" + make_iso_from_dir "$iso_path" "cidata" "$seed_dir" + rm -rf "$seed_dir" +} + +prepare_role_bundle() { + local role="$1" + local out_path="$2" + local image_name + image_name="$(image_name_for_role "$role")" + "$PREPARE_IMAGE_BUNDLE_SCRIPT" "$out_path" "$image_name" +} + +final_image_name() { + echo "$IMAGE_DIR/stack-emulator-$1-$2.qcow2" +} + +prepare_bundle_artifacts() { + local arch="$1" + local role="$2" + local bundle_tgz="$IMAGE_DIR/${role}-${arch}-docker-image.tar.gz" + local bundle_meta="$bundle_tgz.image-id" + local image_name image_id cached_image_id + image_name="$(image_name_for_role "$role")" + image_id="$(docker image inspect --format '{{.ID}}' "$image_name")" + cached_image_id="" + if [ -f "$bundle_meta" ]; then + cached_image_id="$(cat "$bundle_meta")" + fi + + if [ -f "$bundle_tgz" ] && [ "$cached_image_id" = "$image_id" ]; then + log "Reusing bundle: $bundle_tgz" + return 0 + fi + + log "Creating Docker image bundle for ${role} (${arch})..." + prepare_role_bundle "$role" "$bundle_tgz" + printf "%s" "$image_id" > "$bundle_meta" +} + +build_one() { + local role="$1" + local arch="$2" + local base_img="$IMAGE_DIR/debian-${DEBIAN_VERSION}-base-${arch}.qcow2" + local final_img + final_img="$(final_image_name "$role" "$arch")" + local bundle_tgz="$IMAGE_DIR/${role}-${arch}-docker-image.tar.gz" + + log "━━━ Building ${role} image (${arch}) ━━━" + + local tmp_dir + tmp_dir="$(mktemp -d /tmp/stack-qemu-build-${role}-${arch}-XXXXXX)" + local tmp_img="$tmp_dir/disk.qcow2" + local seed_iso="$tmp_dir/seed.iso" + local bundle_iso="$tmp_dir/bundle.iso" + local bundle_dir="$tmp_dir/bundle" + local serial_log="$tmp_dir/serial.log" + local pidfile="$tmp_dir/qemu.pid" + local qemu_base pid elapsed + local start_time=$SECONDS + + cp "$base_img" "$tmp_img" + qemu-img resize "$tmp_img" "$DISK_SIZE" >/dev/null 2>&1 || true + + make_seed_iso "$seed_iso" "$role" + + mkdir -p "$bundle_dir" + cp "$bundle_tgz" "$bundle_dir/img.tgz" + make_iso_from_dir "$bundle_iso" "STACKBUNDLE" "$bundle_dir" + + : > "$serial_log" + qemu_base="$(qemu_cmd_prefix_for_arch "$arch")" + + # shellcheck disable=SC2086 + $qemu_base \ + -boot order=c \ + -m "$RAM" \ + -smp "$CPUS" \ + -drive "file=$tmp_img,format=qcow2,if=virtio" \ + -drive "file=$seed_iso,format=raw,if=virtio,readonly=on" \ + -drive "file=$bundle_iso,format=raw,if=virtio,readonly=on" \ + -netdev user,id=net0 \ + -device virtio-net-pci,netdev=net0 \ + -serial "file:$serial_log" \ + -display none \ + -daemonize \ + -pidfile "$pidfile" + + pid="$(cat "$pidfile")" + elapsed=0 + while [ "$elapsed" -lt "$PROVISION_TIMEOUT" ]; do + if grep -q "STACK_CLOUD_INIT_DONE" "$serial_log" 2>/dev/null; then + break + fi + sleep 5 + elapsed=$((SECONDS - start_time)) + printf "\r [%3ds / %ds] provisioning %s..." "$elapsed" "$PROVISION_TIMEOUT" "$role" + done + echo "" + + if ! grep -q "STACK_CLOUD_INIT_DONE" "$serial_log" 2>/dev/null; then + err "Provisioning timed out for ${role} (${arch})" + tail -50 "$serial_log" >&2 || true + if kill -0 "$pid" 2>/dev/null; then + kill "$pid" 2>/dev/null || true + sleep 1 + kill -9 "$pid" 2>/dev/null || true + fi + rm -rf "$tmp_dir" + exit 1 + fi + + local shutdown_wait=0 + while [ "$shutdown_wait" -lt 90 ] && kill -0 "$pid" 2>/dev/null; do + sleep 1 + shutdown_wait=$((shutdown_wait + 1)) + done + + if kill -0 "$pid" 2>/dev/null; then + warn "Guest did not power off cleanly for ${role}; forcing shutdown." + kill "$pid" 2>/dev/null || true + sleep 2 + kill -9 "$pid" 2>/dev/null || true + fi + + cp "$tmp_img" "$final_img" + cp "$serial_log" "$IMAGE_DIR/provision-${role}-${arch}.log" + rm -rf "$tmp_dir" + + qemu-img convert -O qcow2 -c "$final_img" "$final_img.tmp" + mv "$final_img.tmp" "$final_img" + + local size + size="$(du -h "$final_img" | cut -f1)" + log "━━━ ${role} image ready: $final_img (${size}) ━━━" +} + +build_all_for_arch() { + local arch="$1" + local base_img="$IMAGE_DIR/debian-${DEBIAN_VERSION}-base-${arch}.qcow2" + download_cloud_image "$arch" "$base_img" + + local pids=() + local role + for role in "${TARGET_ROLES[@]}"; do + prepare_bundle_artifacts "$arch" "$role" & + pids+=("$!") + done + for pid in "${pids[@]}"; do + wait "$pid" + done + + pids=() + for role in "${TARGET_ROLES[@]}"; do + if [ "${#TARGET_ROLES[@]}" -gt 1 ] && [ "$PARALLEL_BUILDS" -gt 1 ]; then + build_one "$role" "$arch" & + pids+=("$!") + else + build_one "$role" "$arch" + fi + done + if [ "${#pids[@]}" -gt 0 ]; then + for pid in "${pids[@]}"; do + wait "$pid" + done + fi +} + +for arch in "${TARGET_ARCHS[@]}"; do + build_all_for_arch "$arch" +done + +log "Done. Start with: docker/local-emulator/qemu/run-emulator.sh start" diff --git a/docker/local-emulator/qemu/cloud-init/deps/meta-data b/docker/local-emulator/qemu/cloud-init/deps/meta-data new file mode 100644 index 0000000000..56b7d7e91e --- /dev/null +++ b/docker/local-emulator/qemu/cloud-init/deps/meta-data @@ -0,0 +1,2 @@ +instance-id: stack-emulator-deps-001 +local-hostname: stack-emulator-deps diff --git a/docker/local-emulator/qemu/cloud-init/deps/user-data b/docker/local-emulator/qemu/cloud-init/deps/user-data new file mode 100644 index 0000000000..d449758250 --- /dev/null +++ b/docker/local-emulator/qemu/cloud-init/deps/user-data @@ -0,0 +1,115 @@ +#cloud-config + +hostname: stack-emulator-deps +manage_etc_hosts: true + +users: + - name: stack + shell: /bin/bash + sudo: ALL=(ALL) NOPASSWD:ALL + lock_passwd: false + +chpasswd: + list: | + root:stack-emulator + stack:stack-emulator + expire: false + +package_update: true +package_upgrade: false + +packages: + - docker.io + - ca-certificates + - curl + - netcat-openbsd + - qemu-guest-agent + +write_files: + - path: /usr/local/bin/install-deps-container + permissions: '0755' + content: | + #!/bin/bash + set -euo pipefail + + mkdir -p /mnt/stack-bundle + bundle_device="$(readlink -f /dev/disk/by-label/STACKBUNDLE)" + mount -o ro "$bundle_device" /mnt/stack-bundle + + systemctl enable --now docker + until docker info >/dev/null 2>&1; do sleep 1; done + + gzip -dc /mnt/stack-bundle/img.tgz | docker load + + - path: /usr/local/bin/run-stack-deps-container + permissions: '0755' + content: | + #!/bin/bash + set -euo pipefail + + docker rm -f stack-deps >/dev/null 2>&1 || true + exec docker run \ + --rm \ + --name stack-deps \ + --network host \ + -v stack-postgres-data:/data/postgres \ + -v stack-redis-data:/data/redis \ + -v stack-clickhouse-data:/data/clickhouse \ + -v stack-minio-data:/data/minio \ + -v stack-inbucket-data:/data/inbucket \ + stack-local-emulator-deps + + - path: /usr/local/bin/stack-ready-signal + permissions: '0755' + content: | + #!/bin/bash + set -euo pipefail + + until nc -z 127.0.0.1 5432 >/dev/null 2>&1; do sleep 1; done + until curl -sf http://127.0.0.1:8123/ping >/dev/null 2>&1; do sleep 1; done + until curl -sf http://127.0.0.1:8071/api/v1/health/ >/dev/null 2>&1; do sleep 1; done + until curl -sf http://127.0.0.1:9090/minio/health/live >/dev/null 2>&1; do sleep 1; done + until [ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/ 2>/dev/null || true)" = "401" ]; do sleep 1; done + echo "STACK_EMULATOR_READY" > /dev/console 2>/dev/null || true + echo "STACK_EMULATOR_READY" > /dev/ttyAMA0 2>/dev/null || true + echo "STACK_EMULATOR_READY" > /dev/ttyS0 2>/dev/null || true + + - path: /etc/systemd/system/stack-deps.service + content: | + [Unit] + Description=Stack Auth local emulator deps container + Wants=network-online.target docker.service + After=network-online.target docker.service + + [Service] + Restart=always + RestartSec=5 + TimeoutStartSec=0 + ExecStart=/usr/local/bin/run-stack-deps-container + ExecStop=/usr/bin/docker stop stack-deps + + [Install] + WantedBy=multi-user.target + + - path: /etc/systemd/system/stack-ready.service + content: | + [Unit] + Description=Stack Auth deps readiness signal + After=stack-deps.service + + [Service] + Type=oneshot + ExecStart=/usr/local/bin/stack-ready-signal + + [Install] + WantedBy=multi-user.target + +runcmd: + - bash /usr/local/bin/install-deps-container + - systemctl daemon-reload + - systemctl enable --now stack-deps.service + - systemctl start stack-ready.service + - echo "STACK_CLOUD_INIT_DONE" > /dev/console 2>/dev/null || true + - echo "STACK_CLOUD_INIT_DONE" > /dev/ttyAMA0 2>/dev/null || true + - echo "STACK_CLOUD_INIT_DONE" > /dev/ttyS0 2>/dev/null || true + - shutdown -P now diff --git a/docker/local-emulator/qemu/cloud-init/dev-server/meta-data b/docker/local-emulator/qemu/cloud-init/dev-server/meta-data new file mode 100644 index 0000000000..10f433d4bf --- /dev/null +++ b/docker/local-emulator/qemu/cloud-init/dev-server/meta-data @@ -0,0 +1,2 @@ +instance-id: stack-emulator-dev-server-001 +local-hostname: stack-emulator-dev-server diff --git a/docker/local-emulator/qemu/cloud-init/dev-server/user-data b/docker/local-emulator/qemu/cloud-init/dev-server/user-data new file mode 100644 index 0000000000..17d1a62196 --- /dev/null +++ b/docker/local-emulator/qemu/cloud-init/dev-server/user-data @@ -0,0 +1,284 @@ +#cloud-config + +hostname: stack-emulator-dev-server +manage_etc_hosts: true + +users: + - name: stack + shell: /bin/bash + sudo: ALL=(ALL) NOPASSWD:ALL + lock_passwd: false + +chpasswd: + list: | + root:stack-emulator + stack:stack-emulator + expire: false + +package_update: true +package_upgrade: false + +packages: + - docker.io + - ca-certificates + - curl + - qemu-guest-agent + +write_files: + - path: /etc/stack-auth/local-emulator.base.env + content: | + NEXT_PUBLIC_STACK_DOCS_BASE_URL=https://docs.stack-auth.com + NEXT_PUBLIC_STACK_IS_LOCAL_EMULATOR=true + NEXT_PUBLIC_STACK_PROJECT_ID=internal + NEXT_PUBLIC_STACK_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only + STACK_SECRET_SERVER_KEY=this-secret-server-key-is-for-local-development-only + STACK_SERVER_SECRET=23-wuNpik0gIW4mruTz25rbIvhuuvZFrLOLtL7J4tyo + STACK_CHANGELOG_URL=https://raw.githubusercontent.com/stack-auth/stack-auth/refs/heads/dev/CHANGELOG.md + STACK_SEED_ENABLE_DUMMY_PROJECT=true + STACK_SEED_INTERNAL_PROJECT_SIGN_UP_ENABLED=true + STACK_SEED_INTERNAL_PROJECT_OTP_ENABLED=true + STACK_SEED_INTERNAL_PROJECT_ALLOW_LOCALHOST=true + STACK_SEED_INTERNAL_PROJECT_OAUTH_PROVIDERS=github,spotify,google,microsoft + STACK_SEED_INTERNAL_PROJECT_USER_INTERNAL_ACCESS=true + STACK_SEED_INTERNAL_PROJECT_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only + STACK_SEED_INTERNAL_PROJECT_SECRET_SERVER_KEY=this-secret-server-key-is-for-local-development-only + STACK_SEED_INTERNAL_PROJECT_SUPER_SECRET_ADMIN_KEY=this-super-secret-admin-key-is-for-local-development-only + STACK_SVIX_API_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NTUxNDA2MzksImV4cCI6MTk3MDUwMDYzOSwibmJmIjoxNjU1MTQwNjM5LCJpc3MiOiJzdml4LXNlcnZlciIsInN1YiI6Im9yZ18yM3JiOFlkR3FNVDBxSXpwZ0d3ZFhmSGlyTXUifQ.En8w77ZJWbd0qrMlHHupHUB-4cx17RfzFykseg95SUk + STACK_OPENAI_API_KEY=mock_openai_api_key + STACK_OPENROUTER_API_KEY=FORWARD_TO_PRODUCTION + STACK_STRIPE_SECRET_KEY=sk_test_mockstripekey + STACK_STRIPE_WEBHOOK_SECRET=mock_stripe_webhook_secret + STACK_RESEND_API_KEY=mock_resend_api_key + STACK_RESEND_WEBHOOK_SECRET=mock_resend_webhook_secret + STACK_DNSIMPLE_API_TOKEN=mock_dnsimple_api_token + STACK_DNSIMPLE_ACCOUNT_ID=mock_dnsimple_account_id + STACK_DNSIMPLE_API_BASE_URL=https://api.dnsimple.com/v2 + STACK_FREESTYLE_API_KEY=mock_stack_freestyle_key + STACK_VERCEL_SANDBOX_TOKEN=vercel_sandbox_disabled_for_local_development + CRON_SECRET=mock_cron_secret + STACK_S3_REGION=us-east-1 + STACK_S3_ACCESS_KEY_ID=s3mockroot + STACK_S3_SECRET_ACCESS_KEY=s3mockroot + STACK_S3_BUCKET=stack-storage + STACK_S3_PRIVATE_BUCKET=stack-storage-private + STACK_AWS_REGION=us-east-1 + STACK_AWS_ACCESS_KEY_ID=test + STACK_AWS_SECRET_ACCESS_KEY=test + STACK_QSTASH_TOKEN=eyJVc2VySUQiOiJkZWZhdWx0VXNlciIsIlBhc3N3b3JkIjoiZGVmYXVsdFBhc3N3b3JkIn0= + STACK_QSTASH_CURRENT_SIGNING_KEY=sig_7kYjw48mhY7kAjqNGcy6cr29RJ6r + STACK_QSTASH_NEXT_SIGNING_KEY=sig_5ZB6DVzB1wjE8S6rZ7eenA8Pdnhs + STACK_CLICKHOUSE_ADMIN_USER=stackframe + STACK_CLICKHOUSE_ADMIN_PASSWORD=PASSWORD-PLACEHOLDER--9gKyMxJeMx + STACK_CLICKHOUSE_EXTERNAL_PASSWORD=PASSWORD-PLACEHOLDER--EZeHscBMzE + STACK_EMAIL_MONITOR_PROJECT_ID=internal + STACK_EMAIL_MONITOR_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only + STACK_EMAIL_MONITOR_RESEND_EMAIL_DOMAIN=stack-generated.example.com + STACK_EMAIL_MONITOR_RESEND_EMAIL_API_KEY=this-is-a-fake-key + STACK_EMAIL_MONITOR_SECRET_TOKEN=this-secret-token-is-for-local-development-only + STACK_FEATUREBASE_JWT_SECRET=secret-value + STACK_FORWARD_MOCK_OAUTH_SERVER=false + STACK_GITHUB_CLIENT_ID=MOCK + STACK_GITHUB_CLIENT_SECRET=MOCK + STACK_GOOGLE_CLIENT_ID=MOCK + STACK_GOOGLE_CLIENT_SECRET=MOCK + STACK_MICROSOFT_CLIENT_ID=MOCK + STACK_MICROSOFT_CLIENT_SECRET=MOCK + STACK_SPOTIFY_CLIENT_ID=MOCK + STACK_SPOTIFY_CLIENT_SECRET=MOCK + STACK_ALLOW_SHARED_OAUTH_ACCESS_TOKENS=true + + - path: /usr/local/bin/install-stack-app-container + permissions: '0755' + content: | + #!/bin/bash + set -euo pipefail + + mkdir -p /mnt/stack-bundle + bundle_device="$(readlink -f /dev/disk/by-label/STACKBUNDLE)" + mount -o ro "$bundle_device" /mnt/stack-bundle + + systemctl enable --now docker + until docker info >/dev/null 2>&1; do sleep 1; done + + gzip -dc /mnt/stack-bundle/img.tgz | docker load + + - path: /usr/local/bin/render-stack-env + permissions: '0755' + content: | + #!/bin/bash + set -euo pipefail + + mkdir -p /mnt/stack-runtime /run/stack-auth + runtime_device="$(readlink -f /dev/disk/by-label/STACKCFG)" + mountpoint -q /mnt/stack-runtime || mount -o ro "$runtime_device" /mnt/stack-runtime + + set -a + source /mnt/stack-runtime/runtime.env + source /etc/stack-auth/local-emulator.base.env + set +a + + cat > /run/stack-auth/local-emulator.env </dev/null 2>&1 || true + exec docker run \ + --rm \ + --name stack-app \ + --network host \ + --add-host host.docker.internal:host-gateway \ + --env-file /run/stack-auth/local-emulator.env \ + stack-local-emulator-app + + - path: /usr/local/bin/stack-ready-signal + permissions: '0755' + content: | + #!/bin/bash + set -euo pipefail + + mkdir -p /mnt/stack-runtime + runtime_device="$(readlink -f /dev/disk/by-label/STACKCFG)" + mountpoint -q /mnt/stack-runtime || mount -o ro "$runtime_device" /mnt/stack-runtime + source /mnt/stack-runtime/runtime.env + + until curl -sf "http://127.0.0.1:${STACK_EMULATOR_PORT_PREFIX}02/health?db=1" >/dev/null 2>&1; do sleep 1; done + until curl -sf "http://127.0.0.1:${STACK_EMULATOR_PORT_PREFIX}01/handler/sign-in" >/dev/null 2>&1; do sleep 1; done + echo "STACK_EMULATOR_READY" > /dev/console 2>/dev/null || true + echo "STACK_EMULATOR_READY" > /dev/ttyAMA0 2>/dev/null || true + echo "STACK_EMULATOR_READY" > /dev/ttyS0 2>/dev/null || true + + - path: /etc/systemd/system/stack-app.service + content: | + [Unit] + Description=Stack Auth local emulator app container + Wants=network-online.target docker.service + After=network-online.target docker.service + + [Service] + Restart=always + RestartSec=5 + TimeoutStartSec=0 + ExecStart=/usr/local/bin/run-stack-app-container + ExecStop=/usr/bin/docker stop stack-app + + [Install] + WantedBy=multi-user.target + + - path: /etc/systemd/system/stack-ready.service + content: | + [Unit] + Description=Stack Auth app readiness signal + After=stack-app.service + + [Service] + Type=oneshot + ExecStart=/usr/local/bin/stack-ready-signal + + [Install] + WantedBy=multi-user.target + +runcmd: + - mkdir -p /etc/stack-auth + - bash /usr/local/bin/install-stack-app-container + - systemctl daemon-reload + - systemctl enable --now stack-app.service + - systemctl start stack-ready.service + - echo "STACK_CLOUD_INIT_DONE" > /dev/console 2>/dev/null || true + - echo "STACK_CLOUD_INIT_DONE" > /dev/ttyAMA0 2>/dev/null || true + - echo "STACK_CLOUD_INIT_DONE" > /dev/ttyS0 2>/dev/null || true + - shutdown -P now diff --git a/docker/local-emulator/qemu/cloud-init/meta-data b/docker/local-emulator/qemu/cloud-init/meta-data new file mode 100644 index 0000000000..15c4b5b8e0 --- /dev/null +++ b/docker/local-emulator/qemu/cloud-init/meta-data @@ -0,0 +1,2 @@ +instance-id: stack-emulator-split +local-hostname: stack-emulator diff --git a/docker/local-emulator/qemu/cloud-init/user-data b/docker/local-emulator/qemu/cloud-init/user-data new file mode 100644 index 0000000000..8fe08d594f --- /dev/null +++ b/docker/local-emulator/qemu/cloud-init/user-data @@ -0,0 +1,6 @@ +#cloud-config +write_files: + - path: /etc/motd + content: | + Stack Auth local emulator now uses the split role-specific cloud-init + definitions in docker/local-emulator/qemu/cloud-init/{deps,dev-server}. diff --git a/docker/local-emulator/qemu/common.sh b/docker/local-emulator/qemu/common.sh new file mode 100755 index 0000000000..1e3374dad4 --- /dev/null +++ b/docker/local-emulator/qemu/common.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# Shared helpers for QEMU emulator scripts. +# Source this file; do not execute it directly. + +AARCH64_FIRMWARE_PATHS=( + /opt/homebrew/share/qemu/edk2-aarch64-code.fd + /usr/share/qemu/edk2-aarch64-code.fd + /usr/share/AAVMF/AAVMF_CODE.fd + /usr/share/qemu-efi-aarch64/QEMU_EFI.fd +) + +detect_host() { + case "$(uname -m)" in + arm64|aarch64) HOST_ARCH="arm64" ;; + x86_64|amd64) HOST_ARCH="amd64" ;; + *) echo "Unsupported host architecture: $(uname -m)" >&2; exit 1 ;; + esac + + case "$(uname -s)" in + Darwin) HOST_OS="darwin" ;; + Linux) HOST_OS="linux" ;; + MINGW*|MSYS*|CYGWIN*) HOST_OS="windows" ;; + *) HOST_OS="unknown" ;; + esac +} + +qemu_binary_for_arch() { + case "$1" in + arm64) echo "qemu-system-aarch64" ;; + amd64) echo "qemu-system-x86_64" ;; + *) return 1 ;; + esac +} + +find_aarch64_firmware() { + local p + for p in "${AARCH64_FIRMWARE_PATHS[@]}"; do + if [ -f "$p" ]; then + echo "$p" + return 0 + fi + done + echo "No aarch64 UEFI firmware found." >&2 + return 1 +} + +make_iso_from_dir() { + local iso_path="$1" + local volume_name="$2" + local source_dir="$3" + + rm -f "$iso_path" "${iso_path}.iso" + if command -v hdiutil >/dev/null 2>&1; then + local tmp_dir + tmp_dir="$(mktemp -d /tmp/stack-emulator-iso-XXXXXX)" + cp -R "$source_dir/." "$tmp_dir/" + hdiutil makehybrid -o "$iso_path" "$tmp_dir" -joliet -iso -default-volume-name "$volume_name" 2>/dev/null + if [ -f "${iso_path}.iso" ]; then + mv "${iso_path}.iso" "$iso_path" + fi + rm -rf "$tmp_dir" + elif command -v mkisofs >/dev/null 2>&1; then + mkisofs -output "$iso_path" -volid "$volume_name" -joliet -rock "$source_dir" >/dev/null 2>&1 + elif command -v genisoimage >/dev/null 2>&1; then + genisoimage -output "$iso_path" -volid "$volume_name" -joliet -rock "$source_dir" >/dev/null 2>&1 + else + echo "Missing ISO creation tool (need hdiutil, mkisofs, or genisoimage)" >&2 + exit 1 + fi +} diff --git a/docker/local-emulator/qemu/host-file-bridge.mjs b/docker/local-emulator/qemu/host-file-bridge.mjs new file mode 100644 index 0000000000..22d14c5c37 --- /dev/null +++ b/docker/local-emulator/qemu/host-file-bridge.mjs @@ -0,0 +1,139 @@ +#!/usr/bin/env node + +import http from "node:http"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; + +const port = Number.parseInt(process.env.STACK_QEMU_FILE_BRIDGE_PORT ?? "", 10) || 8116; +const host = process.env.STACK_QEMU_FILE_BRIDGE_HOST ?? "0.0.0.0"; +const token = process.env.STACK_QEMU_FILE_BRIDGE_TOKEN ?? ""; + +if (token === "") { + console.error("STACK_QEMU_FILE_BRIDGE_TOKEN is required"); + process.exit(1); +} + +const allowedRoots = [os.homedir(), "/tmp"].map((root) => path.resolve(root)); + +function isWithinRoot(filePath, rootPath) { + return filePath === rootPath || filePath.startsWith(`${rootPath}${path.sep}`); +} + +function isAllowedPath(filePath) { + return allowedRoots.some((rootPath) => isWithinRoot(filePath, rootPath)); +} + +function sendJson(res, statusCode, body) { + res.writeHead(statusCode, { "Content-Type": "application/json" }); + res.end(JSON.stringify(body)); +} + +function sendText(res, statusCode, message) { + res.writeHead(statusCode, { "Content-Type": "text/plain; charset=utf-8" }); + res.end(message); +} + +async function readBody(req) { + let body = ""; + for await (const chunk of req) { + body += chunk.toString(); + } + return body; +} + +function parseRequestBody(bodyText) { + try { + const body = JSON.parse(bodyText); + return typeof body === "object" && body !== null ? body : null; + } catch { + return null; + } +} + +async function handleRead(res, body) { + const requestedPath = body.path; + if (typeof requestedPath !== "string") { + sendText(res, 400, "Body must include a string 'path'."); + return; + } + + const filePath = path.resolve(requestedPath); + if (!isAllowedPath(filePath)) { + sendText(res, 403, `Path is not allowed: ${filePath}`); + return; + } + + try { + const content = await fs.readFile(filePath, "utf-8"); + sendJson(res, 200, { exists: true, content }); + } catch (error) { + if (error?.code === "ENOENT") { + sendJson(res, 200, { exists: false }); + return; + } + throw error; + } +} + +async function handleWrite(res, body) { + const requestedPath = body.path; + const content = body.content; + if (typeof requestedPath !== "string" || typeof content !== "string") { + sendText(res, 400, "Body must include string 'path' and 'content' fields."); + return; + } + + const filePath = path.resolve(requestedPath); + if (!isAllowedPath(filePath)) { + sendText(res, 403, `Path is not allowed: ${filePath}`); + return; + } + + await fs.mkdir(path.dirname(filePath), { recursive: true }); + await fs.writeFile(filePath, content, "utf-8"); + sendJson(res, 200, { ok: true }); +} + +const server = http.createServer(async (req, res) => { + try { + if (req.url === "/health") { + sendText(res, 200, "ok"); + return; + } + + if (req.method !== "POST") { + sendText(res, 405, "Method not allowed"); + return; + } + + if (req.headers["x-stack-emulator-token"] !== token) { + sendText(res, 401, "Unauthorized"); + return; + } + + const requestBody = parseRequestBody(await readBody(req)); + if (requestBody === null) { + sendText(res, 400, "Invalid JSON body."); + return; + } + + if (req.url === "/read") { + await handleRead(res, requestBody); + return; + } + + if (req.url === "/write") { + await handleWrite(res, requestBody); + return; + } + + sendText(res, 404, "Not found"); + } catch (error) { + sendText(res, 500, error instanceof Error ? error.message : String(error)); + } +}); + +server.listen(port, host, () => { + console.log(`stack-qemu-file-bridge listening on ${host}:${port}`); +}); diff --git a/docker/local-emulator/qemu/images/.gitignore b/docker/local-emulator/qemu/images/.gitignore new file mode 100644 index 0000000000..d6b7ef32c8 --- /dev/null +++ b/docker/local-emulator/qemu/images/.gitignore @@ -0,0 +1,2 @@ +* +!.gitignore diff --git a/docker/local-emulator/qemu/prepare-app-bundle.sh b/docker/local-emulator/qemu/prepare-app-bundle.sh new file mode 100755 index 0000000000..6f89115e2c --- /dev/null +++ b/docker/local-emulator/qemu/prepare-app-bundle.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +exec "$SCRIPT_DIR/prepare-image-bundle.sh" "${1:-}" stack-local-emulator-app diff --git a/docker/local-emulator/qemu/prepare-image-bundle.sh b/docker/local-emulator/qemu/prepare-image-bundle.sh new file mode 100755 index 0000000000..40dcd01b96 --- /dev/null +++ b/docker/local-emulator/qemu/prepare-image-bundle.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +set -euo pipefail + +OUTPUT_PATH="${1:-}" +IMAGE_NAME="${2:-}" + +if [ -z "$OUTPUT_PATH" ] || [ -z "$IMAGE_NAME" ]; then + echo "Usage: $0 " >&2 + exit 1 +fi + +if ! command -v docker >/dev/null 2>&1; then + echo "docker is required to package emulator images" >&2 + exit 1 +fi + +if ! docker image inspect "$IMAGE_NAME" >/dev/null 2>&1; then + cat >&2 < "$tmp_output" +mv "$tmp_output" "$OUTPUT_PATH" diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh new file mode 100755 index 0000000000..5688aad010 --- /dev/null +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -0,0 +1,607 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=common.sh +source "$SCRIPT_DIR/common.sh" + +IMAGE_DIR="$SCRIPT_DIR/images" +RUN_DIR="/tmp/stack-emulator-run" +FILE_BRIDGE_SCRIPT="$SCRIPT_DIR/host-file-bridge.mjs" + +DEPS_RAM="${EMULATOR_DEPS_RAM:-4096}" +DEPS_CPUS="${EMULATOR_DEPS_CPUS:-4}" +APP_RAM="${EMULATOR_APP_RAM:-6144}" +APP_CPUS="${EMULATOR_APP_CPUS:-4}" +PORT_PREFIX="${PORT_PREFIX:-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}}" +FILE_BRIDGE_PORT="${EMULATOR_FILE_BRIDGE_PORT:-${PORT_PREFIX}16}" +READY_TIMEOUT="${EMULATOR_READY_TIMEOUT:-240}" +DEPS_HOST_ALIAS="10.0.2.2" + +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[0;33m' +CYAN='\033[0;36m' +NC='\033[0m' + +log() { echo -e "${GREEN}[emulator]${NC} $*"; } +warn() { echo -e "${YELLOW}[emulator]${NC} $*"; } +err() { echo -e "${RED}[emulator]${NC} $*" >&2; } +info() { echo -e "${CYAN}[emulator]${NC} $*"; } + +file_bridge_pidfile() { + echo "$RUN_DIR/host-file-bridge.pid" +} + +file_bridge_logfile() { + echo "$RUN_DIR/host-file-bridge.log" +} + +file_bridge_tokenfile() { + echo "$RUN_DIR/host-file-bridge.token" +} + +ensure_file_bridge_token() { + local token_file + token_file="$(file_bridge_tokenfile)" + # Deterministic token so snapshots can reuse the same value across restarts + FILE_BRIDGE_TOKEN="$(printf 'stack-local-emulator-%s' "$PORT_PREFIX" | shasum -a 256 | head -c 48)" + mkdir -p "$RUN_DIR" + printf "%s" "$FILE_BRIDGE_TOKEN" > "$token_file" +} + +is_file_bridge_running() { + local pidfile + pidfile="$(file_bridge_pidfile)" + if [ ! -f "$pidfile" ]; then + return 1 + fi + local pid + pid="$(cat "$pidfile")" + kill -0 "$pid" 2>/dev/null +} + +start_file_bridge() { + ensure_file_bridge_token + if is_file_bridge_running; then + return 0 + fi + + if [ ! -f "$FILE_BRIDGE_SCRIPT" ]; then + err "Missing host file bridge script: $FILE_BRIDGE_SCRIPT" + exit 1 + fi + + local pid + pid="$( + STACK_QEMU_FILE_BRIDGE_PORT="$FILE_BRIDGE_PORT" \ + STACK_QEMU_FILE_BRIDGE_HOST="0.0.0.0" \ + STACK_QEMU_FILE_BRIDGE_TOKEN="$FILE_BRIDGE_TOKEN" \ + python3 - "$FILE_BRIDGE_SCRIPT" "$(file_bridge_logfile)" <<'PY' +import os +import subprocess +import sys + +script_path = sys.argv[1] +log_path = sys.argv[2] + +with open(log_path, "ab", buffering=0) as log_file: + process = subprocess.Popen( + ["node", script_path], + stdin=subprocess.DEVNULL, + stdout=log_file, + stderr=log_file, + start_new_session=True, + env=os.environ.copy(), + close_fds=True, + ) + +print(process.pid) +PY + )" + echo "$pid" > "$(file_bridge_pidfile)" + + local elapsed=0 + while [ "$elapsed" -lt 15 ]; do + if curl -sf "http://127.0.0.1:${FILE_BRIDGE_PORT}/health" >/dev/null 2>&1; then + return 0 + fi + if ! kill -0 "$pid" 2>/dev/null; then + err "Host file bridge exited unexpectedly." + tail -40 "$(file_bridge_logfile)" 2>/dev/null || true + exit 1 + fi + sleep 1 + elapsed=$((elapsed + 1)) + done + + err "Timed out waiting for host file bridge on port ${FILE_BRIDGE_PORT}." + tail -40 "$(file_bridge_logfile)" 2>/dev/null || true + exit 1 +} + +stop_file_bridge() { + local pidfile + pidfile="$(file_bridge_pidfile)" + if [ ! -f "$pidfile" ]; then + return 0 + fi + + local pid + pid="$(cat "$pidfile")" + if kill -0 "$pid" 2>/dev/null; then + kill "$pid" 2>/dev/null || true + sleep 1 + kill -9 "$pid" 2>/dev/null || true + fi + rm -f "$pidfile" "$(file_bridge_logfile)" "$(file_bridge_tokenfile)" +} + +detect_host +ARCH="${EMULATOR_ARCH:-$HOST_ARCH}" + +select_accelerator() { + local accel="tcg" + if [ "$ARCH" = "$HOST_ARCH" ]; then + case "$HOST_OS" in + darwin) + if "$(qemu_binary_for_arch "$ARCH")" -accel help 2>&1 | grep -q hvf; then + accel="hvf" + fi + ;; + linux) + if [ -e /dev/kvm ]; then + accel="kvm" + fi + ;; + esac + fi + ACCEL="$accel" +} + +select_accelerator + +instance_dir() { + echo "$RUN_DIR/$1" +} + +image_path_for_role() { + echo "$IMAGE_DIR/stack-emulator-$1-$ARCH.qcow2" +} + +runtime_iso_path() { + echo "$(instance_dir "$1")/runtime-config.iso" +} + +SNAPSHOT_NAME="ready" +IS_SNAPSHOT_RESTORE=false + +role_has_snapshot() { + local role_dir + role_dir="$(instance_dir "$1")" + [ -f "$role_dir/disk.qcow2" ] && + qemu-img snapshot -l "$role_dir/disk.qcow2" 2>/dev/null | grep -q "$SNAPSHOT_NAME" +} + +can_restore_snapshots() { + role_has_snapshot deps && role_has_snapshot dev-server +} + +save_snapshot() { + local role="$1" + local role_dir + role_dir="$(instance_dir "$role")" + if [ ! -S "$role_dir/monitor.sock" ]; then + warn "No monitor socket for ${role}, skipping snapshot" + return 1 + fi + + log "Saving ${role} snapshot..." + local out_file="$role_dir/savevm.out" + rm -f "$out_file" + + ( + printf '{"execute":"qmp_capabilities"}\n' + sleep 0.3 + printf '{"execute":"human-monitor-command","arguments":{"command-line":"savevm %s"}}\n' "$SNAPSHOT_NAME" + sleep 180 + ) | socat -t 180 - "UNIX-CONNECT:$role_dir/monitor.sock" > "$out_file" 2>/dev/null & + local pid=$! + + local elapsed=0 + while [ "$elapsed" -lt 120 ]; do + sleep 2 + elapsed=$((elapsed + 2)) + if [ -f "$out_file" ] && [ "$(grep -c '"return"' "$out_file" 2>/dev/null)" -ge 2 ]; then + kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null || true + rm -f "$out_file" + return 0 + fi + if ! kill -0 "$pid" 2>/dev/null; then + break + fi + done + + kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null || true + rm -f "$out_file" + warn "Snapshot save timed out for ${role}" + return 1 +} + +prepare_runtime_config_iso() { + local role="$1" + local role_dir + role_dir="$(instance_dir "$role")" + local cfg_dir="$role_dir/runtime-config" + local cfg_iso + cfg_iso="$(runtime_iso_path "$role")" + rm -rf "$cfg_dir" + mkdir -p "$cfg_dir" + cat > "$cfg_dir/runtime.env" </dev/null + return $? + fi + + local code + code="$(curl -s -o /dev/null -w "%{http_code}" --max-time 2 "http://127.0.0.1:${port}${path}" 2>/dev/null || true)" + local expected + for expected in ${expected_codes//,/ }; do + if [ "$code" = "$expected" ]; then + return 0 + fi + done + return 1 +} + +deps_ready() { + service_is_up "${PORT_PREFIX}28" tcp && + service_is_up "${PORT_PREFIX}05" http / && + service_is_up "${PORT_PREFIX}29" tcp && + service_is_up "${PORT_PREFIX}13" http /api/v1/health/ && + service_is_up "${PORT_PREFIX}36" http /ping && + service_is_up "${PORT_PREFIX}21" http /minio/health/live && + service_is_up "${PORT_PREFIX}25" http / 401 +} + +app_ready() { + service_is_up "${PORT_PREFIX}02" http "/health?db=1" && + service_is_up "${PORT_PREFIX}01" http /handler/sign-in +} + +wait_for_condition() { + local label="$1" + local timeout="$2" + local check_fn="$3" + local started=$SECONDS + local elapsed=0 + + log "Waiting for ${label}..." + while [ "$elapsed" -lt "$timeout" ]; do + if "$check_fn"; then + echo "" + log "${label} ready in ${elapsed}s" + return 0 + fi + sleep 1 + elapsed=$((SECONDS - started)) + printf "\r [%3ds] %s..." "$elapsed" "$label" + done + echo "" + return 1 +} + +build_qemu_cmd() { + local role="$1" + local role_dir + role_dir="$(instance_dir "$role")" + local base_img + base_img="$(image_path_for_role "$role")" + + if [ ! -f "$base_img" ]; then + err "Missing QEMU image: $base_img" + err "Run docker/local-emulator/qemu/build-image.sh $ARCH $role first." + exit 1 + fi + + mkdir -p "$role_dir" + local has_snapshot=false + if [ -f "$role_dir/disk.qcow2" ] && role_has_snapshot "$role"; then + has_snapshot=true + else + rm -f "$role_dir/disk.qcow2" + qemu-img create -f qcow2 -b "$base_img" -F qcow2 "$role_dir/disk.qcow2" >/dev/null + fi + + local qemu_bin machine cpu firmware_args=() + qemu_bin="$(qemu_binary_for_arch "$ARCH")" + case "$ARCH" in + arm64) + machine="virt" + cpu="max" + local firmware + firmware="$(find_aarch64_firmware)" + firmware_args=(-bios "$firmware") + ;; + amd64) + machine="q35" + if [ "$ACCEL" = "tcg" ] && [ "$HOST_ARCH" != "amd64" ]; then + cpu="qemu64" + else + cpu="max" + fi + ;; + esac + + local netdev="user,id=net0" + if [ "$role" = "deps" ]; then + netdev+=",hostfwd=tcp::${PORT_PREFIX}22-:22" + netdev+=",hostfwd=tcp::${PORT_PREFIX}28-:5432" + netdev+=",hostfwd=tcp::${PORT_PREFIX}29-:2500" + netdev+=",hostfwd=tcp::${PORT_PREFIX}05-:9001" + netdev+=",hostfwd=tcp::${PORT_PREFIX}30-:1100" + netdev+=",hostfwd=tcp::${PORT_PREFIX}13-:8071" + netdev+=",hostfwd=tcp::${PORT_PREFIX}21-:9090" + netdev+=",hostfwd=tcp::${PORT_PREFIX}25-:8080" + netdev+=",hostfwd=tcp::${PORT_PREFIX}36-:8123" + netdev+=",hostfwd=tcp::${PORT_PREFIX}37-:9009" + else + netdev+=",hostfwd=tcp::${PORT_PREFIX}23-:22" + netdev+=",hostfwd=tcp::${PORT_PREFIX}01-:${PORT_PREFIX}01" + netdev+=",hostfwd=tcp::${PORT_PREFIX}02-:${PORT_PREFIX}02" + fi + + local ram cpus + if [ "$role" = "deps" ]; then + ram="$DEPS_RAM" + cpus="$DEPS_CPUS" + else + ram="$APP_RAM" + cpus="$APP_CPUS" + fi + + QEMU_CMD=( + "$qemu_bin" + -machine "$machine" + -accel "$ACCEL" + -cpu "$cpu" + "${firmware_args[@]}" + -boot order=c + -m "$ram" + -smp "$cpus" + -drive "file=$role_dir/disk.qcow2,format=qcow2,if=virtio" + -drive "file=$(runtime_iso_path "$role"),format=raw,if=virtio,readonly=on" + -netdev "$netdev" + -device virtio-net-pci,netdev=net0 + -chardev "socket,id=monitor,path=$role_dir/monitor.sock,server=on,wait=off" + -mon "chardev=monitor,mode=control" + -serial "file:$role_dir/serial.log" + -display none + -daemonize + -pidfile "$role_dir/qemu.pid" + ) + + if [ "$has_snapshot" = "true" ]; then + QEMU_CMD+=(-loadvm "$SNAPSHOT_NAME") + IS_SNAPSHOT_RESTORE=true + fi +} + +is_running() { + local role_dir + role_dir="$(instance_dir "$1")" + if [ ! -f "$role_dir/qemu.pid" ]; then + return 1 + fi + local pid + pid="$(cat "$role_dir/qemu.pid")" + kill -0 "$pid" 2>/dev/null +} + +tail_role_logs() { + local role="$1" + local role_dir + role_dir="$(instance_dir "$role")" + if [ -f "$role_dir/serial.log" ]; then + echo "" + warn "Last serial log lines for ${role}:" + tail -40 "$role_dir/serial.log" || true + fi +} + +ensure_ports_free() { + local ports=("${PORT_PREFIX}01" "${PORT_PREFIX}02" "${PORT_PREFIX}05" "${PORT_PREFIX}13" "${PORT_PREFIX}16" "${PORT_PREFIX}21" "${PORT_PREFIX}25" "${PORT_PREFIX}28" "${PORT_PREFIX}29" "${PORT_PREFIX}30" "${PORT_PREFIX}36" "${PORT_PREFIX}37") + local port + for port in "${ports[@]}"; do + if lsof -iTCP:"$port" -sTCP:LISTEN >/dev/null 2>&1; then + err "Port $port is already in use. Stop the Docker emulator or other services first." + exit 1 + fi + done +} + +start_role() { + local role="$1" + local role_dir + role_dir="$(instance_dir "$role")" + mkdir -p "$role_dir" + : > "$role_dir/serial.log" + prepare_runtime_config_iso "$role" + build_qemu_cmd "$role" + "${QEMU_CMD[@]}" +} + +stop_role() { + local role="$1" + local role_dir + role_dir="$(instance_dir "$role")" + if [ ! -f "$role_dir/qemu.pid" ]; then + return 0 + fi + local pid + pid="$(cat "$role_dir/qemu.pid")" + if kill -0 "$pid" 2>/dev/null; then + if [ -S "$role_dir/monitor.sock" ]; then + echo '{"execute":"qmp_capabilities"}' | socat - UNIX-CONNECT:"$role_dir/monitor.sock" >/dev/null 2>&1 || true + echo '{"execute":"system_powerdown"}' | socat - UNIX-CONNECT:"$role_dir/monitor.sock" >/dev/null 2>&1 || true + sleep 3 + fi + if kill -0 "$pid" 2>/dev/null; then + kill "$pid" 2>/dev/null || true + sleep 1 + kill -9 "$pid" 2>/dev/null || true + fi + fi + rm -f "$role_dir/qemu.pid" "$role_dir/monitor.sock" "$role_dir/serial.log" + rm -rf "$role_dir/runtime-config" + rm -f "$role_dir/runtime-config.iso" +} + +all_ready() { + deps_ready && app_ready +} + +cmd_start() { + ensure_ports_free + mkdir -p "$RUN_DIR" + start_file_bridge + + IS_SNAPSHOT_RESTORE=false + + info "Starting QEMU local emulator" + info "Arch: $ARCH | Accel: $ACCEL | Prefix: $PORT_PREFIX" + + start_role deps + start_role dev-server + + if [ "$IS_SNAPSHOT_RESTORE" = "true" ]; then + info "Restoring from snapshot..." + if wait_for_condition "services (snapshot)" 30 all_ready; then + log "All services are green (restored from snapshot)." + return 0 + fi + warn "Snapshot restore failed. Resetting and doing fresh boot..." + stop_role dev-server + stop_role deps + rm -rf "$(instance_dir deps)" "$(instance_dir dev-server)" + IS_SNAPSHOT_RESTORE=false + start_role deps + start_role dev-server + fi + + info "Deps VM: ${DEPS_RAM}MB/${DEPS_CPUS} CPUs | App VM: ${APP_RAM}MB/${APP_CPUS} CPUs" + + if ! wait_for_condition "deps services" "$READY_TIMEOUT" deps_ready; then + tail_role_logs deps + exit 1 + fi + + if ! wait_for_condition "dashboard/backend" "$READY_TIMEOUT" app_ready; then + tail_role_logs dev-server + exit 1 + fi + + save_snapshot deps + save_snapshot dev-server + log "All services are green. Snapshot saved for fast restart." +} + +cmd_stop() { + stop_role dev-server + stop_role deps + stop_file_bridge + log "QEMU emulator stopped." +} + +cmd_reset() { + cmd_stop 2>/dev/null || true + rm -rf "$RUN_DIR" + log "Emulator state reset. Next start will be a fresh boot." +} + +print_service_status() { + local name="$1" + local port="$2" + local proto="$3" + local path="${4:-/}" + local expected_codes="${5:-200}" + if service_is_up "$port" "$proto" "$path" "$expected_codes"; then + echo -e " ${GREEN}●${NC} $name (:$port)" + else + echo -e " ${RED}●${NC} $name (:$port)" + fi +} + +cmd_status() { + echo "Guests:" + if is_running deps; then + echo -e " ${GREEN}●${NC} deps" + else + echo -e " ${RED}●${NC} deps" + fi + if is_running dev-server; then + echo -e " ${GREEN}●${NC} dev-server" + else + echo -e " ${RED}●${NC} dev-server" + fi + echo "" + echo "Services:" + print_service_status "Dashboard" "${PORT_PREFIX}01" http /handler/sign-in + print_service_status "Backend" "${PORT_PREFIX}02" http "/health?db=1" + print_service_status "PostgreSQL" "${PORT_PREFIX}28" tcp + print_service_status "Inbucket HTTP" "${PORT_PREFIX}05" http / + print_service_status "Host File Bridge" "${FILE_BRIDGE_PORT}" http /health + print_service_status "Svix" "${PORT_PREFIX}13" http /api/v1/health/ + print_service_status "MinIO" "${PORT_PREFIX}21" http /minio/health/live + print_service_status "QStash" "${PORT_PREFIX}25" http / 401 + print_service_status "ClickHouse" "${PORT_PREFIX}36" http /ping +} + +cmd_bench() { + local start_time end_time + cmd_stop >/dev/null 2>&1 || true + start_time="$(python3 - <<'PY' +import time +print(time.time()) +PY +)" + cmd_start + end_time="$(python3 - <<'PY' +import time +print(time.time()) +PY +)" + python3 - < { + const start = performance.now(); + try { + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), timeoutMs); + const res = await fetch(`http://127.0.0.1:${port}${path}`, { signal: controller.signal }); + clearTimeout(timeout); + return { up: res.ok || res.status < 500, latencyMs: Math.round(performance.now() - start) }; + } catch { + return { up: false, latencyMs: Math.round(performance.now() - start) }; + } +} + +async function checkTcp(port: number, timeoutMs = 3000): Promise<{ up: boolean; latencyMs: number }> { + const start = performance.now(); + return await new Promise((resolve) => { + const socket = net.createConnection({ host: '127.0.0.1', port }, () => { + socket.destroy(); + resolve({ up: true, latencyMs: Math.round(performance.now() - start) }); + }); + socket.on('error', () => resolve({ up: false, latencyMs: Math.round(performance.now() - start) })); + socket.setTimeout(timeoutMs, () => { + socket.destroy(); + resolve({ up: false, latencyMs: Math.round(performance.now() - start) }); + }); + }); +} + +export async function GET() { + const results = await Promise.all( + SERVICES.map(async (svc) => { + const check = svc.protocol === 'http' + ? await checkHttp(svc.port, svc.httpPath ?? '/') + : await checkTcp(svc.port); + return { + name: svc.name, + description: svc.description, + port: svc.port, + status: check.up ? 'up' as const : 'down' as const, + latencyMs: check.latencyMs, + }; + }) + ); + + return NextResponse.json({ + timestamp: new Date().toISOString(), + services: results, + summary: { + total: results.length, + up: results.filter((r) => r.status === 'up').length, + down: results.filter((r) => r.status === 'down').length, + }, + }); +} diff --git a/examples/demo/src/app/emulator-status/page.tsx b/examples/demo/src/app/emulator-status/page.tsx new file mode 100644 index 0000000000..542bff846b --- /dev/null +++ b/examples/demo/src/app/emulator-status/page.tsx @@ -0,0 +1,182 @@ +'use client'; + +import { runAsynchronously } from '@stackframe/stack-shared/dist/utils/promises'; +import { Card, CardContent, CardHeader, Typography } from '@stackframe/stack-ui'; +import { useCallback, useEffect, useState } from 'react'; + +type ServiceResult = { + name: string; + description: string; + port: number; + status: 'up' | 'down'; + latencyMs: number; +}; + +type StatusResponse = { + timestamp: string; + services: ServiceResult[]; + summary: { total: number; up: number; down: number }; +}; + +function StatusDot({ status }: { status: 'up' | 'down' | 'checking' }) { + const color = status === 'up' + ? 'bg-emerald-500' + : status === 'down' + ? 'bg-red-500' + : 'bg-yellow-400 animate-pulse'; + return ( + + ); +} + +function ServiceRow({ service }: { service: ServiceResult }) { + return ( +
+
+ +
+ {service.name} + {service.description} +
+
+
+ :{service.port} + {service.status === 'up' && ( + {service.latencyMs}ms + )} + + {service.status === 'up' ? 'Online' : 'Offline'} + +
+
+ ); +} + +export default function EmulatorStatusPage() { + const [data, setData] = useState(null); + const [loading, setLoading] = useState(true); + const [autoRefresh, setAutoRefresh] = useState(true); + + const fetchStatus = useCallback(async () => { + try { + const res = await fetch('/api/emulator-status', { cache: 'no-store' }); + const json = await res.json(); + setData(json as StatusResponse); + } catch { + // keep last known state + } finally { + setLoading(false); + } + }, []); + + useEffect(() => { + runAsynchronously(fetchStatus()); + if (!autoRefresh) return; + const interval = setInterval(() => { + runAsynchronously(fetchStatus()); + }, 5000); + return () => clearInterval(interval); + }, [fetchStatus, autoRefresh]); + + const summary = data?.summary; + const allUp = summary != null && summary.down === 0; + + return ( +
+
+
+
+ Local Emulator Status + + Monitoring services in the all-in-one dependencies container + +
+
+ + +
+
+ + + + {loading && !data ? ( +
+ + Checking services... +
+ ) : summary ? ( +
+
+ + + {allUp + ? 'All services operational' + : `${summary.down} of ${summary.total} services are offline`} + +
+
+ {summary.up} up + {summary.down > 0 && ( + {summary.down} down + )} + {data != null && updated {new Date(data.timestamp).toLocaleTimeString()}} +
+
+ ) : null} + + + + + + Services + + + {data?.services.map((svc) => ( + + ))} + {!data && loading && ( +
Loading...
+ )} + + + + + + Quick Start + + + Start the all-in-one local emulator dependencies: + 
+              {`# Start (single container with all services)
+pnpm run emulator-compose up --detach --build
+
+# Stop and remove volumes
+pnpm run emulator-compose down -v`}
+
+ + This single container replaces the 17+ containers from the full docker-compose setup. + + + +
+
+ ); +} diff --git a/examples/demo/src/components/header.tsx b/examples/demo/src/components/header.tsx index 28742b7de9..989cce998e 100644 --- a/examples/demo/src/components/header.tsx +++ b/examples/demo/src/components/header.tsx @@ -27,13 +27,16 @@ export default function Header() { Anonymous Test + + Emulator Status +
setTheme(theme === 'dark' ? 'light' : 'dark')} />
-
{/* Placeholder for fixed header */} +
); } diff --git a/package.json b/package.json index 762f7c594c..02575c0f7f 100644 --- a/package.json +++ b/package.json @@ -20,9 +20,22 @@ "restart-dev-environment": "pnpm pre && pnpm run build:packages && pnpm run codegen && pnpm run restart-deps && pnpm run restart-dev-in-background", "stop-dev-environment": "pnpm pre && pnpm run kill-dev:named && pnpm run stop-deps", "clean": "pnpm pre-no-codegen && turbo run clean && rimraf --glob **/.next && rimraf --glob **/.turbo && rimraf .turbo && rimraf --glob **/node_modules && rimraf node_modules", + "fml": "pnpm clean && pnpm i && (pnpm build || true) && pnpm codegen && pnpm restart-deps", "codegen": "pnpm pre && turbo run codegen && pnpm run generate-sdks", "codegen:backend": "pnpm pre && turbo run codegen --filter=@stackframe/backend...", "deps-compose": "docker compose -p stack-dependencies-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81} -f docker/dependencies/docker.compose.yaml", + "emulator-compose": "docker compose -p stack-local-emulator-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81} -f docker/local-emulator/docker-compose.yaml", + "start-emulator": "pnpm pre && pnpm run emulator-compose up --detach --build && pnpm run wait-until-emulator-is-ready && echo \"\\nLocal emulator started. Dashboard: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01 Backend: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02. 'pnpm run stop-emulator' to stop.\"", + "stop-emulator": "pnpm run emulator-compose kill && pnpm run emulator-compose down -v", + "restart-emulator": "pnpm pre && pnpm run stop-emulator && pnpm run start-emulator", + "wait-until-emulator-is-ready": "pnpx wait-on http-get://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02/health?db=1 http-get://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01/handler/sign-in", + "wait-until-postgres-is-ready:emulator": "until pg_isready -h localhost -p ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28 2>/dev/null; do sleep 1; done", + "emulator-qemu:build": "docker/local-emulator/qemu/build-image.sh", + "emulator-qemu:start": "docker/local-emulator/qemu/run-emulator.sh start", + "emulator-qemu:stop": "docker/local-emulator/qemu/run-emulator.sh stop", + "emulator-qemu:reset": "docker/local-emulator/qemu/run-emulator.sh reset", + "emulator-qemu:status": "docker/local-emulator/qemu/run-emulator.sh status", + "emulator-qemu:bench": "docker/local-emulator/qemu/run-emulator.sh bench", "stop-deps": "POSTGRES_DELAY_MS=0 pnpm run deps-compose kill && POSTGRES_DELAY_MS=0 pnpm run deps-compose down -v", "wait-until-postgres-is-ready:pg_isready": "until pg_isready -h localhost -p ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28 && pg_isready -h localhost -p ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}34; do sleep 1; done", "wait-until-postgres-is-ready": "command -v pg_isready >/dev/null 2>&1 && pnpm run wait-until-postgres-is-ready:pg_isready || sleep 10 # not everyone has pg_isready installed, so we fallback to sleeping", @@ -43,12 +56,14 @@ "fern": "pnpm pre && pnpm run --filter=@stackframe/docs fern", "dev:full": "pnpm pre && concurrently -k \"pnpm run generate-sdks:watch\" \"turbo run dev --concurrency 99999\"", "dev": "pnpm pre && concurrently -k \"pnpm run generate-sdks:watch\" \"pnpm run generate-openapi-docs:watch\" \"turbo run dev --concurrency 99999 --filter=./apps/* --filter=@stackframe/stack-docs --filter=./packages/* --filter=./examples/demo \"", + "dev:tui": "pnpm pre && (trap 'kill 0' EXIT; pnpm run generate-sdks:watch & pnpm run generate-openapi-docs:watch & turbo run dev --ui tui --concurrency 99999 --filter=./apps/* --filter=@stackframe/stack-docs --filter=./packages/* --filter=./examples/demo)", "dev:inspect": "pnpm pre && STACK_BACKEND_DEV_EXTRA_ARGS=\"--inspect\" pnpm run dev", "dev:profile": "pnpm pre && STACK_BACKEND_DEV_EXTRA_ARGS=\"--experimental-cpu-prof\" pnpm run dev", "dev:basic": "pnpm pre && concurrently -k \"pnpm run generate-sdks:watch\" \"turbo run dev --concurrency 99999 --filter=@stackframe/backend --filter=@stackframe/dashboard --filter=@stackframe/mock-oauth-server\"", "dev:docs": "pnpm pre && concurrently -k \"pnpm run generate-openapi-docs:watch\" \"turbo run dev --concurrency 99999 --filter=@stackframe/stack-docs\"", "dev:named": "pnpm pre && concurrently -k \"pnpm run dev\" \"node -e \\\"process.title='node (stack-named-dev-server)'; process.stdin.resume();\\\"\"", "kill-dev:named": "(pgrep -f 'stack-named-dev-server' | xargs -r -n1 pkill -P); echo 'Killed named dev server (if found). Sleeping to give some time for it to shut down...' && sleep 10", + "kms": "PREFIX=${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}; for p in 00 01 02 03 04 06 14; do pids=$(lsof -i :$PREFIX$p 2>/dev/null | grep LISTEN | awk '$1 != \"OrbStack\" {print $2}' | sort -u); [ -n \"$pids\" ] && echo $pids | xargs kill -9 2>/dev/null; done; echo Done.", "start": "pnpm pre && turbo run start --concurrency 99999", "start:backend": "pnpm pre && turbo run start --concurrency 99999 --filter=@stackframe/backend", "start:dashboard": "pnpm pre && turbo run start --concurrency 99999 --filter=@stackframe/dashboard", @@ -78,6 +93,7 @@ "chokidar-cli": "^3.0.0", "codebuff": "^1.0.261", "concurrently": "^8.2.2", + "dotenv-cli": "^7.3.0", "esbuild": "^0.24.0", "eslint": "^8.57.0", "eslint-config-next": "^14.2.17", @@ -89,12 +105,11 @@ "only-allow": "^1.2.1", "rimraf": "^5.0.10", "tsdown": "^0.20.3", - "turbo": "^2.2.3", + "turbo": "^2.8.15", "typescript": "5.9.3", "vite-tsconfig-paths": "^4.3.2", "vitest": "^1.6.0", - "wait-on": "^8.0.1", - "dotenv-cli": "^7.3.0" + "wait-on": "^8.0.1" }, "pnpm": { "overrides": { diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index c4e1b0e48f..ed8650826c 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -94,8 +94,8 @@ importers: specifier: ^0.20.3 version: 0.20.3(typescript@5.9.3) turbo: - specifier: ^2.2.3 - version: 2.2.3 + specifier: ^2.8.15 + version: 2.8.17 typescript: specifier: 5.9.3 version: 5.9.3 @@ -1483,10 +1483,10 @@ importers: version: link:../../packages/stack '@supabase/ssr': specifier: latest - version: 0.8.0(@supabase/supabase-js@2.95.3) + version: 0.9.0(@supabase/supabase-js@2.99.1) '@supabase/supabase-js': specifier: latest - version: 2.95.3 + version: 2.99.1 jose: specifier: ^5.2.2 version: 5.6.3 @@ -8704,33 +8704,33 @@ packages: resolution: {integrity: sha512-SXuhqhuR5FXaYgKTXzZJeqtVA6JKb9IZWaGeEUxHHiOcFy2p51wccO72bYpXwoK4D5pzQOIYLTuAc7etxyMmwg==} engines: {node: '>=12.16'} - '@supabase/auth-js@2.95.3': - resolution: {integrity: sha512-vD2YoS8E2iKIX0F7EwXTmqhUpaNsmbU6X2R0/NdFcs02oEfnHyNP/3M716f3wVJ2E5XHGiTFXki6lRckhJ0Thg==} + '@supabase/auth-js@2.99.1': + resolution: {integrity: sha512-x7lKKTvKjABJt/FYcRSPiTT01Xhm2FF8RhfL8+RHMkmlwmRQ88/lREupIHKwFPW0W6pTCJqkZb7Yhpw/EZ+fNw==} engines: {node: '>=20.0.0'} - '@supabase/functions-js@2.95.3': - resolution: {integrity: sha512-uTuOAKzs9R/IovW1krO0ZbUHSJnsnyJElTXIRhjJTqymIVGcHzkAYnBCJqd7468Fs/Foz1BQ7Dv6DCl05lr7ig==} + '@supabase/functions-js@2.99.1': + resolution: {integrity: sha512-WQE62W5geYImCO4jzFxCk/avnK7JmOdtqu2eiPz3zOaNiIJajNRSAwMMDgEGd2EMs+sUVYj1LfBjfmW3EzHgIA==} engines: {node: '>=20.0.0'} - '@supabase/postgrest-js@2.95.3': - resolution: {integrity: sha512-LTrRBqU1gOovxRm1vRXPItSMPBmEFqrfTqdPTRtzOILV4jPSueFz6pES5hpb4LRlkFwCPRmv3nQJ5N625V2Xrg==} + '@supabase/postgrest-js@2.99.1': + resolution: {integrity: sha512-gtw2ibJrADvfqrpUWXGNlrYUvxttF4WVWfPpTFKOb2IRj7B6YRWMDgcrYqIuD4ZEabK4m6YKQCCGy6clgf1lPA==} engines: {node: '>=20.0.0'} - '@supabase/realtime-js@2.95.3': - resolution: {integrity: sha512-D7EAtfU3w6BEUxDACjowWNJo/ZRo7sDIuhuOGKHIm9FHieGeoJV5R6GKTLtga/5l/6fDr2u+WcW/m8I9SYmaIw==} + '@supabase/realtime-js@2.99.1': + resolution: {integrity: sha512-9EDdy/5wOseGFqxW88ShV9JMRhm7f+9JGY5x+LqT8c7R0X1CTLwg5qie8FiBWcXTZ+68yYxVWunI+7W4FhkWOg==} engines: {node: '>=20.0.0'} - '@supabase/ssr@0.8.0': - resolution: {integrity: sha512-/PKk8kNFSs8QvvJ2vOww1mF5/c5W8y42duYtXvkOSe+yZKRgTTZywYG2l41pjhNomqESZCpZtXuWmYjFRMV+dw==} + '@supabase/ssr@0.9.0': + resolution: {integrity: sha512-UFY6otYV3yqCgV+AyHj80vNkTvbf1Gas2LW4dpbQ4ap6p6v3eB2oaDfcI99jsuJzwVBCFU4BJI+oDYyhNk1z0Q==} peerDependencies: - '@supabase/supabase-js': ^2.76.1 + '@supabase/supabase-js': ^2.97.0 - '@supabase/storage-js@2.95.3': - resolution: {integrity: sha512-4GxkJiXI3HHWjxpC3sDx1BVrV87O0hfX+wvJdqGv67KeCu+g44SPnII8y0LL/Wr677jB7tpjAxKdtVWf+xhc9A==} + '@supabase/storage-js@2.99.1': + resolution: {integrity: sha512-mf7zPfqofI62SOoyQJeNUVxe72E4rQsbWim6lTDPeLu3lHija/cP5utlQADGrjeTgOUN6znx/rWn7SjrETP1dw==} engines: {node: '>=20.0.0'} - '@supabase/supabase-js@2.95.3': - resolution: {integrity: sha512-Fukw1cUTQ6xdLiHDJhKKPu6svEPaCEDvThqCne3OaQyZvuq2qjhJAd91kJu3PXLG18aooCgYBaB6qQz35hhABg==} + '@supabase/supabase-js@2.99.1': + resolution: {integrity: sha512-5MRoYD9ffXq8F6a036dm65YoSHisC3by/d22mauKE99Vrwf792KxYIIr/iqCX7E4hkuugbPZ5EGYHTB7MKy6Vg==} engines: {node: '>=20.0.0'} '@swc/counter@0.1.3': @@ -12472,6 +12472,7 @@ packages: i18next-parser@9.0.2: resolution: {integrity: sha512-Q1yTZljBp1DcVAQD7LxduEqFRpjIeZc+5VnQ+gU8qG9WvY3U5rqK0IVONRWNtngh3orb197bfy1Sz4wlwcplxg==} engines: {node: '>=18.0.0 || >=20.0.0 || >=22.0.0', npm: '>=6', yarn: '>=1'} + deprecated: Project is deprecated, use i18next-cli instead hasBin: true i18next@21.10.0: @@ -15797,12 +15798,12 @@ packages: tar@6.2.1: resolution: {integrity: sha512-DZ4yORTwrbTj/7MZYq2w+/ZFdI6OZ/f9SFHR+71gIVUZhOQPHzVCLpvRnPgyaMpfWxxk/4ONva3GQSyNIKRv6A==} engines: {node: '>=10'} - deprecated: Old versions of tar are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exhorbitant rates) by contacting i@izs.me + deprecated: Old versions of tar are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me tar@7.4.3: resolution: {integrity: sha512-5S7Va8hKfV7W5U6g3aYxXmlPoZVAwUMy9AOKyF2fVuZa2UD3qZjg578OrLRt8PcNN1PleVaL/5/yYATNL0ICUw==} engines: {node: '>=18'} - deprecated: Old versions of tar are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exhorbitant rates) by contacting i@izs.me + deprecated: Old versions of tar are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me teex@1.0.1: resolution: {integrity: sha512-eYE6iEI62Ni1H8oIa7KlDU6uQBtqr4Eajni3wX7rpfXD8ysFx8z0+dri+KWEPWpBsxXfxu58x/0jvTVT1ekOSg==} @@ -16046,41 +16047,41 @@ packages: engines: {node: '>=18.0.0'} hasBin: true - turbo-darwin-64@2.2.3: - resolution: {integrity: sha512-Rcm10CuMKQGcdIBS3R/9PMeuYnv6beYIHqfZFeKWVYEWH69sauj4INs83zKMTUiZJ3/hWGZ4jet9AOwhsssLyg==} + turbo-darwin-64@2.8.17: + resolution: {integrity: sha512-ZFkv2hv7zHpAPEXBF6ouRRXshllOavYc+jjcrYyVHvxVTTwJWsBZwJ/gpPzmOKGvkSjsEyDO5V6aqqtZzwVF+Q==} cpu: [x64] os: [darwin] - turbo-darwin-arm64@2.2.3: - resolution: {integrity: sha512-+EIMHkuLFqUdJYsA3roj66t9+9IciCajgj+DVek+QezEdOJKcRxlvDOS2BUaeN8kEzVSsNiAGnoysFWYw4K0HA==} + turbo-darwin-arm64@2.8.17: + resolution: {integrity: sha512-5DXqhQUt24ycEryXDfMNKEkW5TBHs+QmU23a2qxXwwFDaJsWcPo2obEhBxxdEPOv7qmotjad+09RGeWCcJ9JDw==} cpu: [arm64] os: [darwin] - turbo-linux-64@2.2.3: - resolution: {integrity: sha512-UBhJCYnqtaeOBQLmLo8BAisWbc9v9daL9G8upLR+XGj6vuN/Nz6qUAhverN4Pyej1g4Nt1BhROnj6GLOPYyqxQ==} + turbo-linux-64@2.8.17: + resolution: {integrity: sha512-KLUbz6w7F73D/Ihh51hVagrKR0/CTsPEbRkvXLXvoND014XJ4BCrQUqSxlQ4/hu+nqp1v5WlM85/h3ldeyujuA==} cpu: [x64] os: [linux] - turbo-linux-arm64@2.2.3: - resolution: {integrity: sha512-hJYT9dN06XCQ3jBka/EWvvAETnHRs3xuO/rb5bESmDfG+d9yQjeTMlhRXKrr4eyIMt6cLDt1LBfyi+6CQ+VAwQ==} + turbo-linux-arm64@2.8.17: + resolution: {integrity: sha512-pJK67XcNJH40lTAjFu7s/rUlobgVXyB3A3lDoq+/JccB3hf+SysmkpR4Itlc93s8LEaFAI4mamhFuTV17Z6wOg==} cpu: [arm64] os: [linux] turbo-stream@2.4.0: resolution: {integrity: sha512-FHncC10WpBd2eOmGwpmQsWLDoK4cqsA/UT/GqNoaKOQnT8uzhtCbg3EoUDMvqpOSAI0S26mr0rkjzbOO6S3v1g==} - turbo-windows-64@2.2.3: - resolution: {integrity: sha512-NPrjacrZypMBF31b4HE4ROg4P3nhMBPHKS5WTpMwf7wydZ8uvdEHpESVNMOtqhlp857zbnKYgP+yJF30H3N2dQ==} + turbo-windows-64@2.8.17: + resolution: {integrity: sha512-EijeQ6zszDMmGZLP2vT2RXTs/GVi9rM0zv2/G4rNu2SSRSGFapgZdxgW4b5zUYLVaSkzmkpWlGfPfj76SW9yUg==} cpu: [x64] os: [win32] - turbo-windows-arm64@2.2.3: - resolution: {integrity: sha512-fnNrYBCqn6zgKPKLHu4sOkihBI/+0oYFr075duRxqUZ+1aLWTAGfHZLgjVeLh3zR37CVzuerGIPWAEkNhkWEIw==} + turbo-windows-arm64@2.8.17: + resolution: {integrity: sha512-crpfeMPkfECd4V1PQ/hMoiyVcOy04+bWedu/if89S15WhOalHZ2BYUi6DOJhZrszY+mTT99OwpOsj4wNfb/GHQ==} cpu: [arm64] os: [win32] - turbo@2.2.3: - resolution: {integrity: sha512-5lDvSqIxCYJ/BAd6rQGK/AzFRhBkbu4JHVMLmGh/hCb7U3CqSnr5Tjwfy9vc+/5wG2DJ6wttgAaA7MoCgvBKZQ==} + turbo@2.8.17: + resolution: {integrity: sha512-YwPsNSqU2f/RXU/+Kcb7cPkPZARxom4+me7LKEdN5jsvy2tpfze3zDZ4EiGrJnvOm9Avu9rK0aaYsP7qZ3iz7A==} hasBin: true type-check@0.4.0: @@ -25142,19 +25143,19 @@ snapshots: '@stripe/stripe-js@7.7.0': {} - '@supabase/auth-js@2.95.3': + '@supabase/auth-js@2.99.1': dependencies: tslib: 2.8.1 - '@supabase/functions-js@2.95.3': + '@supabase/functions-js@2.99.1': dependencies: tslib: 2.8.1 - '@supabase/postgrest-js@2.95.3': + '@supabase/postgrest-js@2.99.1': dependencies: tslib: 2.8.1 - '@supabase/realtime-js@2.95.3': + '@supabase/realtime-js@2.99.1': dependencies: '@types/phoenix': 1.6.6 '@types/ws': 8.18.1 @@ -25164,23 +25165,23 @@ snapshots: - bufferutil - utf-8-validate - '@supabase/ssr@0.8.0(@supabase/supabase-js@2.95.3)': + '@supabase/ssr@0.9.0(@supabase/supabase-js@2.99.1)': dependencies: - '@supabase/supabase-js': 2.95.3 + '@supabase/supabase-js': 2.99.1 cookie: 1.0.2 - '@supabase/storage-js@2.95.3': + '@supabase/storage-js@2.99.1': dependencies: iceberg-js: 0.8.1 tslib: 2.8.1 - '@supabase/supabase-js@2.95.3': + '@supabase/supabase-js@2.99.1': dependencies: - '@supabase/auth-js': 2.95.3 - '@supabase/functions-js': 2.95.3 - '@supabase/postgrest-js': 2.95.3 - '@supabase/realtime-js': 2.95.3 - '@supabase/storage-js': 2.95.3 + '@supabase/auth-js': 2.99.1 + '@supabase/functions-js': 2.99.1 + '@supabase/postgrest-js': 2.99.1 + '@supabase/realtime-js': 2.99.1 + '@supabase/storage-js': 2.99.1 transitivePeerDependencies: - bufferutil - utf-8-validate @@ -28541,7 +28542,7 @@ snapshots: eslint: 8.57.1 eslint-import-resolver-node: 0.3.9 eslint-import-resolver-typescript: 3.6.3(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-plugin-import@2.31.0)(eslint@8.57.1) - eslint-plugin-import: 2.31.0(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-typescript@3.6.3)(eslint@8.57.1) + eslint-plugin-import: 2.31.0(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint@8.57.1) eslint-plugin-jsx-a11y: 6.10.2(eslint@8.57.1) eslint-plugin-react: 7.37.2(eslint@8.57.1) eslint-plugin-react-hooks: 5.1.0(eslint@8.57.1) @@ -28565,7 +28566,7 @@ snapshots: debug: 4.4.3 enhanced-resolve: 5.17.0 eslint: 8.57.1 - eslint-module-utils: 2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint@8.57.1))(eslint@8.57.1))(eslint@8.57.1) + eslint-module-utils: 2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1)(eslint@8.57.1) eslint-plugin-import: 2.31.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-typescript@3.6.1)(eslint@8.57.1) fast-glob: 3.3.2 get-tsconfig: 4.8.1 @@ -28615,7 +28616,7 @@ snapshots: - eslint-import-resolver-webpack - supports-color - eslint-module-utils@2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint@8.57.1))(eslint@8.57.1))(eslint@8.57.1): + eslint-module-utils@2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1)(eslint@8.57.1): dependencies: debug: 3.2.7 optionalDependencies: @@ -28675,7 +28676,7 @@ snapshots: doctrine: 2.1.0 eslint: 8.57.1 eslint-import-resolver-node: 0.3.9 - eslint-module-utils: 2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint@8.57.1))(eslint@8.57.1))(eslint@8.57.1) + eslint-module-utils: 2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1)(eslint@8.57.1) hasown: 2.0.2 is-core-module: 2.15.1 is-glob: 4.0.3 @@ -34745,34 +34746,34 @@ snapshots: optionalDependencies: fsevents: 2.3.3 - turbo-darwin-64@2.2.3: + turbo-darwin-64@2.8.17: optional: true - turbo-darwin-arm64@2.2.3: + turbo-darwin-arm64@2.8.17: optional: true - turbo-linux-64@2.2.3: + turbo-linux-64@2.8.17: optional: true - turbo-linux-arm64@2.2.3: + turbo-linux-arm64@2.8.17: optional: true turbo-stream@2.4.0: {} - turbo-windows-64@2.2.3: + turbo-windows-64@2.8.17: optional: true - turbo-windows-arm64@2.2.3: + turbo-windows-arm64@2.8.17: optional: true - turbo@2.2.3: + turbo@2.8.17: optionalDependencies: - turbo-darwin-64: 2.2.3 - turbo-darwin-arm64: 2.2.3 - turbo-linux-64: 2.2.3 - turbo-linux-arm64: 2.2.3 - turbo-windows-64: 2.2.3 - turbo-windows-arm64: 2.2.3 + turbo-darwin-64: 2.8.17 + turbo-darwin-arm64: 2.8.17 + turbo-linux-64: 2.8.17 + turbo-linux-arm64: 2.8.17 + turbo-windows-64: 2.8.17 + turbo-windows-arm64: 2.8.17 type-check@0.4.0: dependencies: diff --git a/stack.config.ts b/stack.config.ts new file mode 100644 index 0000000000..c4470d209f --- /dev/null +++ b/stack.config.ts @@ -0,0 +1 @@ +export const config = {}; From 00106429addba5a5bf83f1bc7fbc1d43afc1620c Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Wed, 18 Mar 2026 13:46:27 -0700 Subject: [PATCH 02/30] Refactor QEMU emulator setup and consolidate cloud-init configurations - Unified the QEMU emulator setup by consolidating the previously separate VM roles (deps and dev-server) into a single VM, simplifying the architecture and reducing complexity. - Updated the `build-image.sh` script to adjust resource allocations and streamline image building processes. - Enhanced the `prepare-image-bundle.sh` script to support multiple Docker images, improving the packaging of emulator components. - Refactored cloud-init configurations to use a single definition for the emulator, improving maintainability and clarity. - Updated documentation in CLAUDE-KNOWLEDGE.md to reflect changes in the emulator's architecture and cloud-init usage. These changes significantly enhance the QEMU emulator's usability and performance for developers. --- claude/CLAUDE-KNOWLEDGE.md | 5 +- docker/local-emulator/qemu/build-image.sh | 143 ++++-------- .../qemu/cloud-init/deps/meta-data | 2 - .../qemu/cloud-init/deps/user-data | 115 ---------- .../qemu/cloud-init/dev-server/meta-data | 2 - .../qemu/cloud-init/emulator/meta-data | 2 + .../{dev-server => emulator}/user-data | 93 +++++--- .../local-emulator/qemu/cloud-init/user-data | 4 +- .../qemu/prepare-image-bundle.sh | 18 +- docker/local-emulator/qemu/run-emulator.sh | 217 +++++++----------- 10 files changed, 200 insertions(+), 401 deletions(-) delete mode 100644 docker/local-emulator/qemu/cloud-init/deps/meta-data delete mode 100644 docker/local-emulator/qemu/cloud-init/deps/user-data delete mode 100644 docker/local-emulator/qemu/cloud-init/dev-server/meta-data create mode 100644 docker/local-emulator/qemu/cloud-init/emulator/meta-data rename docker/local-emulator/qemu/cloud-init/{dev-server => emulator}/user-data (83%) diff --git a/claude/CLAUDE-KNOWLEDGE.md b/claude/CLAUDE-KNOWLEDGE.md index 03c767b0e9..5a5e03b213 100644 --- a/claude/CLAUDE-KNOWLEDGE.md +++ b/claude/CLAUDE-KNOWLEDGE.md @@ -143,4 +143,7 @@ Q: Why does the QEMU emulator use a deterministic file bridge token instead of a A: The file bridge token is baked into the VM's environment when the container starts. When restoring from a snapshot, the VM resumes with the old token. If the host bridge generates a new random token each time, the VM's token won't match and file bridge requests will fail. Using a deterministic token derived from the port prefix (`shasum -a 256` of `stack-local-emulator-$PORT_PREFIX`) ensures the same token on every start, making snapshot restore work seamlessly. Q: Why can't `qemu-img snapshot -l` be used to check snapshots while QEMU is running? -A: QEMU holds an exclusive write lock on the qcow2 file. `qemu-img` commands (including `snapshot -l`) fail with "Failed to get shared write lock". To check snapshots while QEMU runs, use QMP: `{"execute":"human-monitor-command","arguments":{"command-line":"info snapshots"}}`. To check snapshots when QEMU is stopped (e.g., in `role_has_snapshot`), `qemu-img snapshot -l` works fine. +A: QEMU holds an exclusive write lock on the qcow2 file. `qemu-img` commands (including `snapshot -l`) fail with "Failed to get shared write lock". To check snapshots while QEMU runs, use QMP: `{"execute":"human-monitor-command","arguments":{"command-line":"info snapshots"}}`. To check snapshots when QEMU is stopped (e.g., in `vm_has_snapshot`), `qemu-img snapshot -l` works fine. + +Q: Why was the QEMU emulator consolidated from two VMs to one? +A: Originally the emulator used two VMs: one for deps (postgres, redis, clickhouse, etc.) and one for the app (backend + dashboard). This was unnecessarily complex — both containers use `--network host` and can coexist in a single VM. The unified approach uses one VM with both Docker containers, all port forwards in a single QEMU netdev, one qcow2 overlay, and one snapshot. The cloud-init is in `docker/local-emulator/qemu/cloud-init/emulator/`. During the build phase, only the deps container starts (the app container fails because no STACKCFG ISO exists). At runtime, both containers start — deps first, then the app renders its env from STACKCFG and runs migrations. The deps host in the app's env is `127.0.0.1` (localhost within the same VM), while the file bridge URL still points to `10.0.2.2` (QEMU user-mode host gateway). Default resources: 8GB RAM / 6 CPUs (configurable via `EMULATOR_RAM` / `EMULATOR_CPUS`). diff --git a/docker/local-emulator/qemu/build-image.sh b/docker/local-emulator/qemu/build-image.sh index 0f8049e23b..d5c78adfaf 100755 --- a/docker/local-emulator/qemu/build-image.sh +++ b/docker/local-emulator/qemu/build-image.sh @@ -10,11 +10,10 @@ CLOUD_INIT_ROOT="$SCRIPT_DIR/cloud-init" PREPARE_IMAGE_BUNDLE_SCRIPT="$SCRIPT_DIR/prepare-image-bundle.sh" DEBIAN_VERSION="${DEBIAN_VERSION:-13}" -DISK_SIZE="${EMULATOR_DISK_SIZE:-16G}" +DISK_SIZE="${EMULATOR_DISK_SIZE:-12G}" RAM="${EMULATOR_BUILD_RAM:-4096}" -CPUS="${EMULATOR_BUILD_CPUS:-4}" +CPUS="${EMULATOR_BUILD_CPUS:-$(sysctl -n hw.ncpu 2>/dev/null || nproc 2>/dev/null || echo 4)}" PROVISION_TIMEOUT="${EMULATOR_PROVISION_TIMEOUT:-1800}" -PARALLEL_BUILDS="${EMULATOR_BUILD_PARALLEL:-2}" RED='\033[0;31m' GREEN='\033[0;32m' @@ -27,23 +26,16 @@ err() { echo -e "${RED}[build]${NC} $*" >&2; } detect_host TARGET_ARCH="${1:-$HOST_ARCH}" -TARGET_ROLE="${2:-all}" TARGET_ARCHS=() case "$TARGET_ARCH" in arm64) TARGET_ARCHS=(arm64) ;; amd64) TARGET_ARCHS=(amd64) ;; both) TARGET_ARCHS=(arm64 amd64) ;; - *) err "Usage: $0 [arm64|amd64|both] [all|deps|dev-server]"; exit 1 ;; + *) err "Usage: $0 [arm64|amd64|both]"; exit 1 ;; esac -TARGET_ROLES=() -case "$TARGET_ROLE" in - all) TARGET_ROLES=(deps dev-server) ;; - deps) TARGET_ROLES=(deps) ;; - dev-server) TARGET_ROLES=(dev-server) ;; - *) err "Usage: $0 [arm64|amd64|both] [all|deps|dev-server]"; exit 1 ;; -esac +DOCKER_IMAGES=(stack-local-emulator-deps stack-local-emulator-app) check_deps() { local missing=() @@ -123,79 +115,46 @@ qemu_cmd_prefix_for_arch() { esac } -image_name_for_role() { - case "$1" in - deps) echo "stack-local-emulator-deps" ;; - dev-server) echo "stack-local-emulator-app" ;; - *) return 1 ;; - esac -} - -prepare_cloud_init_dir() { - local role="$1" - local out_dir="$2" - mkdir -p "$out_dir" - cp "$CLOUD_INIT_ROOT/$role/meta-data" "$out_dir/meta-data" - cp "$CLOUD_INIT_ROOT/$role/user-data" "$out_dir/user-data" -} - -make_seed_iso() { - local iso_path="$1" - local role="$2" - local seed_dir - seed_dir="$(mktemp -d)" - prepare_cloud_init_dir "$role" "$seed_dir" - make_iso_from_dir "$iso_path" "cidata" "$seed_dir" - rm -rf "$seed_dir" -} - -prepare_role_bundle() { - local role="$1" - local out_path="$2" - local image_name - image_name="$(image_name_for_role "$role")" - "$PREPARE_IMAGE_BUNDLE_SCRIPT" "$out_path" "$image_name" -} - final_image_name() { - echo "$IMAGE_DIR/stack-emulator-$1-$2.qcow2" + echo "$IMAGE_DIR/stack-emulator-$1.qcow2" } prepare_bundle_artifacts() { local arch="$1" - local role="$2" - local bundle_tgz="$IMAGE_DIR/${role}-${arch}-docker-image.tar.gz" - local bundle_meta="$bundle_tgz.image-id" - local image_name image_id cached_image_id - image_name="$(image_name_for_role "$role")" - image_id="$(docker image inspect --format '{{.ID}}' "$image_name")" - cached_image_id="" + local bundle_tgz="$IMAGE_DIR/emulator-${arch}-docker-images.tar.gz" + local bundle_meta="$bundle_tgz.image-ids" + + local current_ids="" + for img in "${DOCKER_IMAGES[@]}"; do + current_ids+="$(docker image inspect --format '{{.ID}}' "$img")"$'\n' + done + + local cached_ids="" if [ -f "$bundle_meta" ]; then - cached_image_id="$(cat "$bundle_meta")" + cached_ids="$(cat "$bundle_meta")" fi - if [ -f "$bundle_tgz" ] && [ "$cached_image_id" = "$image_id" ]; then + if [ -f "$bundle_tgz" ] && [ "$cached_ids" = "$current_ids" ]; then log "Reusing bundle: $bundle_tgz" return 0 fi - log "Creating Docker image bundle for ${role} (${arch})..." - prepare_role_bundle "$role" "$bundle_tgz" - printf "%s" "$image_id" > "$bundle_meta" + log "Creating Docker image bundle (${arch})..." + "$PREPARE_IMAGE_BUNDLE_SCRIPT" "$bundle_tgz" "${DOCKER_IMAGES[@]}" + printf "%s" "$current_ids" > "$bundle_meta" } build_one() { - local role="$1" - local arch="$2" + local arch="$1" local base_img="$IMAGE_DIR/debian-${DEBIAN_VERSION}-base-${arch}.qcow2" + local bundle_tgz="$IMAGE_DIR/emulator-${arch}-docker-images.tar.gz" local final_img - final_img="$(final_image_name "$role" "$arch")" - local bundle_tgz="$IMAGE_DIR/${role}-${arch}-docker-image.tar.gz" + final_img="$(final_image_name "$arch")" - log "━━━ Building ${role} image (${arch}) ━━━" + log "━━━ Building emulator image (${arch}) ━━━" local tmp_dir - tmp_dir="$(mktemp -d /tmp/stack-qemu-build-${role}-${arch}-XXXXXX)" + tmp_dir="$(mktemp -d /tmp/stack-qemu-build-${arch}-XXXXXX)" local tmp_img="$tmp_dir/disk.qcow2" local seed_iso="$tmp_dir/seed.iso" local bundle_iso="$tmp_dir/bundle.iso" @@ -208,7 +167,13 @@ build_one() { cp "$base_img" "$tmp_img" qemu-img resize "$tmp_img" "$DISK_SIZE" >/dev/null 2>&1 || true - make_seed_iso "$seed_iso" "$role" + local seed_dir + seed_dir="$(mktemp -d)" + mkdir -p "$seed_dir" + cp "$CLOUD_INIT_ROOT/emulator/meta-data" "$seed_dir/meta-data" + cp "$CLOUD_INIT_ROOT/emulator/user-data" "$seed_dir/user-data" + make_iso_from_dir "$seed_iso" "cidata" "$seed_dir" + rm -rf "$seed_dir" mkdir -p "$bundle_dir" cp "$bundle_tgz" "$bundle_dir/img.tgz" @@ -240,12 +205,12 @@ build_one() { fi sleep 5 elapsed=$((SECONDS - start_time)) - printf "\r [%3ds / %ds] provisioning %s..." "$elapsed" "$PROVISION_TIMEOUT" "$role" + printf "\r [%3ds / %ds] provisioning emulator..." "$elapsed" "$PROVISION_TIMEOUT" done echo "" if ! grep -q "STACK_CLOUD_INIT_DONE" "$serial_log" 2>/dev/null; then - err "Provisioning timed out for ${role} (${arch})" + err "Provisioning timed out for emulator (${arch})" tail -50 "$serial_log" >&2 || true if kill -0 "$pid" 2>/dev/null; then kill "$pid" 2>/dev/null || true @@ -263,14 +228,14 @@ build_one() { done if kill -0 "$pid" 2>/dev/null; then - warn "Guest did not power off cleanly for ${role}; forcing shutdown." + warn "Guest did not power off cleanly; forcing shutdown." kill "$pid" 2>/dev/null || true sleep 2 kill -9 "$pid" 2>/dev/null || true fi cp "$tmp_img" "$final_img" - cp "$serial_log" "$IMAGE_DIR/provision-${role}-${arch}.log" + cp "$serial_log" "$IMAGE_DIR/provision-emulator-${arch}.log" rm -rf "$tmp_dir" qemu-img convert -O qcow2 -c "$final_img" "$final_img.tmp" @@ -278,42 +243,14 @@ build_one() { local size size="$(du -h "$final_img" | cut -f1)" - log "━━━ ${role} image ready: $final_img (${size}) ━━━" -} - -build_all_for_arch() { - local arch="$1" - local base_img="$IMAGE_DIR/debian-${DEBIAN_VERSION}-base-${arch}.qcow2" - download_cloud_image "$arch" "$base_img" - - local pids=() - local role - for role in "${TARGET_ROLES[@]}"; do - prepare_bundle_artifacts "$arch" "$role" & - pids+=("$!") - done - for pid in "${pids[@]}"; do - wait "$pid" - done - - pids=() - for role in "${TARGET_ROLES[@]}"; do - if [ "${#TARGET_ROLES[@]}" -gt 1 ] && [ "$PARALLEL_BUILDS" -gt 1 ]; then - build_one "$role" "$arch" & - pids+=("$!") - else - build_one "$role" "$arch" - fi - done - if [ "${#pids[@]}" -gt 0 ]; then - for pid in "${pids[@]}"; do - wait "$pid" - done - fi + log "━━━ Emulator image ready: $final_img (${size}) ━━━" } for arch in "${TARGET_ARCHS[@]}"; do - build_all_for_arch "$arch" + local_base="$IMAGE_DIR/debian-${DEBIAN_VERSION}-base-${arch}.qcow2" + download_cloud_image "$arch" "$local_base" + prepare_bundle_artifacts "$arch" + build_one "$arch" done log "Done. Start with: docker/local-emulator/qemu/run-emulator.sh start" diff --git a/docker/local-emulator/qemu/cloud-init/deps/meta-data b/docker/local-emulator/qemu/cloud-init/deps/meta-data deleted file mode 100644 index 56b7d7e91e..0000000000 --- a/docker/local-emulator/qemu/cloud-init/deps/meta-data +++ /dev/null @@ -1,2 +0,0 @@ -instance-id: stack-emulator-deps-001 -local-hostname: stack-emulator-deps diff --git a/docker/local-emulator/qemu/cloud-init/deps/user-data b/docker/local-emulator/qemu/cloud-init/deps/user-data deleted file mode 100644 index d449758250..0000000000 --- a/docker/local-emulator/qemu/cloud-init/deps/user-data +++ /dev/null @@ -1,115 +0,0 @@ -#cloud-config - -hostname: stack-emulator-deps -manage_etc_hosts: true - -users: - - name: stack - shell: /bin/bash - sudo: ALL=(ALL) NOPASSWD:ALL - lock_passwd: false - -chpasswd: - list: | - root:stack-emulator - stack:stack-emulator - expire: false - -package_update: true -package_upgrade: false - -packages: - - docker.io - - ca-certificates - - curl - - netcat-openbsd - - qemu-guest-agent - -write_files: - - path: /usr/local/bin/install-deps-container - permissions: '0755' - content: | - #!/bin/bash - set -euo pipefail - - mkdir -p /mnt/stack-bundle - bundle_device="$(readlink -f /dev/disk/by-label/STACKBUNDLE)" - mount -o ro "$bundle_device" /mnt/stack-bundle - - systemctl enable --now docker - until docker info >/dev/null 2>&1; do sleep 1; done - - gzip -dc /mnt/stack-bundle/img.tgz | docker load - - - path: /usr/local/bin/run-stack-deps-container - permissions: '0755' - content: | - #!/bin/bash - set -euo pipefail - - docker rm -f stack-deps >/dev/null 2>&1 || true - exec docker run \ - --rm \ - --name stack-deps \ - --network host \ - -v stack-postgres-data:/data/postgres \ - -v stack-redis-data:/data/redis \ - -v stack-clickhouse-data:/data/clickhouse \ - -v stack-minio-data:/data/minio \ - -v stack-inbucket-data:/data/inbucket \ - stack-local-emulator-deps - - - path: /usr/local/bin/stack-ready-signal - permissions: '0755' - content: | - #!/bin/bash - set -euo pipefail - - until nc -z 127.0.0.1 5432 >/dev/null 2>&1; do sleep 1; done - until curl -sf http://127.0.0.1:8123/ping >/dev/null 2>&1; do sleep 1; done - until curl -sf http://127.0.0.1:8071/api/v1/health/ >/dev/null 2>&1; do sleep 1; done - until curl -sf http://127.0.0.1:9090/minio/health/live >/dev/null 2>&1; do sleep 1; done - until [ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/ 2>/dev/null || true)" = "401" ]; do sleep 1; done - echo "STACK_EMULATOR_READY" > /dev/console 2>/dev/null || true - echo "STACK_EMULATOR_READY" > /dev/ttyAMA0 2>/dev/null || true - echo "STACK_EMULATOR_READY" > /dev/ttyS0 2>/dev/null || true - - - path: /etc/systemd/system/stack-deps.service - content: | - [Unit] - Description=Stack Auth local emulator deps container - Wants=network-online.target docker.service - After=network-online.target docker.service - - [Service] - Restart=always - RestartSec=5 - TimeoutStartSec=0 - ExecStart=/usr/local/bin/run-stack-deps-container - ExecStop=/usr/bin/docker stop stack-deps - - [Install] - WantedBy=multi-user.target - - - path: /etc/systemd/system/stack-ready.service - content: | - [Unit] - Description=Stack Auth deps readiness signal - After=stack-deps.service - - [Service] - Type=oneshot - ExecStart=/usr/local/bin/stack-ready-signal - - [Install] - WantedBy=multi-user.target - -runcmd: - - bash /usr/local/bin/install-deps-container - - systemctl daemon-reload - - systemctl enable --now stack-deps.service - - systemctl start stack-ready.service - - echo "STACK_CLOUD_INIT_DONE" > /dev/console 2>/dev/null || true - - echo "STACK_CLOUD_INIT_DONE" > /dev/ttyAMA0 2>/dev/null || true - - echo "STACK_CLOUD_INIT_DONE" > /dev/ttyS0 2>/dev/null || true - - shutdown -P now diff --git a/docker/local-emulator/qemu/cloud-init/dev-server/meta-data b/docker/local-emulator/qemu/cloud-init/dev-server/meta-data deleted file mode 100644 index 10f433d4bf..0000000000 --- a/docker/local-emulator/qemu/cloud-init/dev-server/meta-data +++ /dev/null @@ -1,2 +0,0 @@ -instance-id: stack-emulator-dev-server-001 -local-hostname: stack-emulator-dev-server diff --git a/docker/local-emulator/qemu/cloud-init/emulator/meta-data b/docker/local-emulator/qemu/cloud-init/emulator/meta-data new file mode 100644 index 0000000000..4a17d62aed --- /dev/null +++ b/docker/local-emulator/qemu/cloud-init/emulator/meta-data @@ -0,0 +1,2 @@ +instance-id: stack-emulator-001 +local-hostname: stack-emulator diff --git a/docker/local-emulator/qemu/cloud-init/dev-server/user-data b/docker/local-emulator/qemu/cloud-init/emulator/user-data similarity index 83% rename from docker/local-emulator/qemu/cloud-init/dev-server/user-data rename to docker/local-emulator/qemu/cloud-init/emulator/user-data index 17d1a62196..ef163c6592 100644 --- a/docker/local-emulator/qemu/cloud-init/dev-server/user-data +++ b/docker/local-emulator/qemu/cloud-init/emulator/user-data @@ -1,6 +1,6 @@ #cloud-config -hostname: stack-emulator-dev-server +hostname: stack-emulator manage_etc_hosts: true users: @@ -15,6 +15,8 @@ chpasswd: stack:stack-emulator expire: false +ssh_pwauth: true + package_update: true package_upgrade: false @@ -22,6 +24,7 @@ packages: - docker.io - ca-certificates - curl + - netcat-openbsd - qemu-guest-agent write_files: @@ -87,7 +90,7 @@ write_files: STACK_SPOTIFY_CLIENT_SECRET=MOCK STACK_ALLOW_SHARED_OAUTH_ACCESS_TOKENS=true - - path: /usr/local/bin/install-stack-app-container + - path: /usr/local/bin/install-emulator-containers permissions: '0755' content: | #!/bin/bash @@ -102,6 +105,24 @@ write_files: gzip -dc /mnt/stack-bundle/img.tgz | docker load + - path: /usr/local/bin/run-stack-deps-container + permissions: '0755' + content: | + #!/bin/bash + set -euo pipefail + + docker rm -f stack-deps >/dev/null 2>&1 || true + exec docker run \ + --rm \ + --name stack-deps \ + --network host \ + -v stack-postgres-data:/data/postgres \ + -v stack-redis-data:/data/redis \ + -v stack-clickhouse-data:/data/clickhouse \ + -v stack-minio-data:/data/minio \ + -v stack-inbucket-data:/data/inbucket \ + stack-local-emulator-deps + - path: /usr/local/bin/render-stack-env permissions: '0755' content: | @@ -117,6 +138,9 @@ write_files: source /etc/stack-auth/local-emulator.base.env set +a + # Deps runs in the same VM, so always localhost + DEPS_HOST=127.0.0.1 + cat > /run/stack-auth/local-emulator.env </dev/null 2>&1; do sleep 1; done + until curl -sf http://127.0.0.1:8123/ping >/dev/null 2>&1; do sleep 1; done + until curl -sf http://127.0.0.1:8071/api/v1/health/ >/dev/null 2>&1; do sleep 1; done + until curl -sf http://127.0.0.1:9090/minio/health/live >/dev/null 2>&1; do sleep 1; done + until [ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/ 2>/dev/null || true)" = "401" ]; do sleep 1; done - until curl -sf "http://127.0.0.1:${STACK_EMULATOR_PORT_PREFIX}02/health?db=1" >/dev/null 2>&1; do sleep 1; done - until curl -sf "http://127.0.0.1:${STACK_EMULATOR_PORT_PREFIX}01/handler/sign-in" >/dev/null 2>&1; do sleep 1; done - echo "STACK_EMULATOR_READY" > /dev/console 2>/dev/null || true - echo "STACK_EMULATOR_READY" > /dev/ttyAMA0 2>/dev/null || true - echo "STACK_EMULATOR_READY" > /dev/ttyS0 2>/dev/null || true - - - path: /etc/systemd/system/stack-app.service + - path: /etc/systemd/system/stack-deps.service content: | [Unit] - Description=Stack Auth local emulator app container + Description=Stack Auth local emulator deps container Wants=network-online.target docker.service After=network-online.target docker.service @@ -253,31 +272,37 @@ write_files: Restart=always RestartSec=5 TimeoutStartSec=0 - ExecStart=/usr/local/bin/run-stack-app-container - ExecStop=/usr/bin/docker stop stack-app + ExecStart=/usr/local/bin/run-stack-deps-container + ExecStop=/usr/bin/docker stop stack-deps [Install] WantedBy=multi-user.target - - path: /etc/systemd/system/stack-ready.service + - path: /etc/systemd/system/stack-app.service content: | [Unit] - Description=Stack Auth app readiness signal - After=stack-app.service + Description=Stack Auth local emulator app container + Wants=network-online.target docker.service + After=network-online.target docker.service stack-deps.service [Service] - Type=oneshot - ExecStart=/usr/local/bin/stack-ready-signal + Restart=always + RestartSec=5 + TimeoutStartSec=0 + ExecStart=/usr/local/bin/run-stack-app-container + ExecStop=/usr/bin/docker stop stack-app [Install] WantedBy=multi-user.target runcmd: - mkdir -p /etc/stack-auth - - bash /usr/local/bin/install-stack-app-container + - bash /usr/local/bin/install-emulator-containers - systemctl daemon-reload - - systemctl enable --now stack-app.service - - systemctl start stack-ready.service + - systemctl enable stack-deps.service + - systemctl enable stack-app.service + - systemctl start stack-deps.service + - bash /usr/local/bin/wait-for-deps - echo "STACK_CLOUD_INIT_DONE" > /dev/console 2>/dev/null || true - echo "STACK_CLOUD_INIT_DONE" > /dev/ttyAMA0 2>/dev/null || true - echo "STACK_CLOUD_INIT_DONE" > /dev/ttyS0 2>/dev/null || true diff --git a/docker/local-emulator/qemu/cloud-init/user-data b/docker/local-emulator/qemu/cloud-init/user-data index 8fe08d594f..6a39bc4d16 100644 --- a/docker/local-emulator/qemu/cloud-init/user-data +++ b/docker/local-emulator/qemu/cloud-init/user-data @@ -2,5 +2,5 @@ write_files: - path: /etc/motd content: | - Stack Auth local emulator now uses the split role-specific cloud-init - definitions in docker/local-emulator/qemu/cloud-init/{deps,dev-server}. + Stack Auth local emulator uses the unified cloud-init definition in + docker/local-emulator/qemu/cloud-init/emulator/. diff --git a/docker/local-emulator/qemu/prepare-image-bundle.sh b/docker/local-emulator/qemu/prepare-image-bundle.sh index 40dcd01b96..daec7a106e 100755 --- a/docker/local-emulator/qemu/prepare-image-bundle.sh +++ b/docker/local-emulator/qemu/prepare-image-bundle.sh @@ -2,10 +2,10 @@ set -euo pipefail OUTPUT_PATH="${1:-}" -IMAGE_NAME="${2:-}" +shift || true -if [ -z "$OUTPUT_PATH" ] || [ -z "$IMAGE_NAME" ]; then - echo "Usage: $0 " >&2 +if [ -z "$OUTPUT_PATH" ] || [ "$#" -eq 0 ]; then + echo "Usage: $0 [docker-image...]" >&2 exit 1 fi @@ -14,8 +14,9 @@ if ! command -v docker >/dev/null 2>&1; then exit 1 fi -if ! docker image inspect "$IMAGE_NAME" >/dev/null 2>&1; then - cat >&2 </dev/null 2>&1; then + cat >&2 < "$tmp_output" +docker save "$@" | gzip -c > "$tmp_output" mv "$tmp_output" "$OUTPUT_PATH" diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index 5688aad010..45a2d50bfe 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -9,14 +9,11 @@ IMAGE_DIR="$SCRIPT_DIR/images" RUN_DIR="/tmp/stack-emulator-run" FILE_BRIDGE_SCRIPT="$SCRIPT_DIR/host-file-bridge.mjs" -DEPS_RAM="${EMULATOR_DEPS_RAM:-4096}" -DEPS_CPUS="${EMULATOR_DEPS_CPUS:-4}" -APP_RAM="${EMULATOR_APP_RAM:-6144}" -APP_CPUS="${EMULATOR_APP_CPUS:-4}" +VM_RAM="${EMULATOR_RAM:-4096}" +VM_CPUS="${EMULATOR_CPUS:-4}" PORT_PREFIX="${PORT_PREFIX:-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}}" FILE_BRIDGE_PORT="${EMULATOR_FILE_BRIDGE_PORT:-${PORT_PREFIX}16}" READY_TIMEOUT="${EMULATOR_READY_TIMEOUT:-240}" -DEPS_HOST_ALIAS="10.0.2.2" RED='\033[0;31m' GREEN='\033[0;32m' @@ -161,43 +158,32 @@ select_accelerator() { select_accelerator -instance_dir() { - echo "$RUN_DIR/$1" -} +VM_DIR="$RUN_DIR/vm" -image_path_for_role() { - echo "$IMAGE_DIR/stack-emulator-$1-$ARCH.qcow2" +image_path() { + echo "$IMAGE_DIR/stack-emulator-$ARCH.qcow2" } runtime_iso_path() { - echo "$(instance_dir "$1")/runtime-config.iso" + echo "$VM_DIR/runtime-config.iso" } SNAPSHOT_NAME="ready" IS_SNAPSHOT_RESTORE=false -role_has_snapshot() { - local role_dir - role_dir="$(instance_dir "$1")" - [ -f "$role_dir/disk.qcow2" ] && - qemu-img snapshot -l "$role_dir/disk.qcow2" 2>/dev/null | grep -q "$SNAPSHOT_NAME" -} - -can_restore_snapshots() { - role_has_snapshot deps && role_has_snapshot dev-server +vm_has_snapshot() { + [ -f "$VM_DIR/disk.qcow2" ] && + qemu-img snapshot -l "$VM_DIR/disk.qcow2" 2>/dev/null | grep -q "$SNAPSHOT_NAME" } save_snapshot() { - local role="$1" - local role_dir - role_dir="$(instance_dir "$role")" - if [ ! -S "$role_dir/monitor.sock" ]; then - warn "No monitor socket for ${role}, skipping snapshot" + if [ ! -S "$VM_DIR/monitor.sock" ]; then + warn "No monitor socket, skipping snapshot" return 1 fi - log "Saving ${role} snapshot..." - local out_file="$role_dir/savevm.out" + log "Saving VM snapshot..." + local out_file="$VM_DIR/savevm.out" rm -f "$out_file" ( @@ -205,7 +191,7 @@ save_snapshot() { sleep 0.3 printf '{"execute":"human-monitor-command","arguments":{"command-line":"savevm %s"}}\n' "$SNAPSHOT_NAME" sleep 180 - ) | socat -t 180 - "UNIX-CONNECT:$role_dir/monitor.sock" > "$out_file" 2>/dev/null & + ) | socat -t 180 - "UNIX-CONNECT:$VM_DIR/monitor.sock" > "$out_file" 2>/dev/null & local pid=$! local elapsed=0 @@ -215,6 +201,7 @@ save_snapshot() { if [ -f "$out_file" ] && [ "$(grep -c '"return"' "$out_file" 2>/dev/null)" -ge 2 ]; then kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null || true rm -f "$out_file" + log "Snapshot saved." return 0 fi if ! kill -0 "$pid" 2>/dev/null; then @@ -224,22 +211,18 @@ save_snapshot() { kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null || true rm -f "$out_file" - warn "Snapshot save timed out for ${role}" + warn "Snapshot save timed out" return 1 } prepare_runtime_config_iso() { - local role="$1" - local role_dir - role_dir="$(instance_dir "$role")" - local cfg_dir="$role_dir/runtime-config" + local cfg_dir="$VM_DIR/runtime-config" local cfg_iso - cfg_iso="$(runtime_iso_path "$role")" + cfg_iso="$(runtime_iso_path)" rm -rf "$cfg_dir" mkdir -p "$cfg_dir" cat > "$cfg_dir/runtime.env" </dev/null + rm -f "$VM_DIR/disk.qcow2" + qemu-img create -f qcow2 -b "$base_img" -F qcow2 "$VM_DIR/disk.qcow2" >/dev/null fi local qemu_bin machine cpu firmware_args=() @@ -348,31 +332,20 @@ build_qemu_cmd() { esac local netdev="user,id=net0" - if [ "$role" = "deps" ]; then - netdev+=",hostfwd=tcp::${PORT_PREFIX}22-:22" - netdev+=",hostfwd=tcp::${PORT_PREFIX}28-:5432" - netdev+=",hostfwd=tcp::${PORT_PREFIX}29-:2500" - netdev+=",hostfwd=tcp::${PORT_PREFIX}05-:9001" - netdev+=",hostfwd=tcp::${PORT_PREFIX}30-:1100" - netdev+=",hostfwd=tcp::${PORT_PREFIX}13-:8071" - netdev+=",hostfwd=tcp::${PORT_PREFIX}21-:9090" - netdev+=",hostfwd=tcp::${PORT_PREFIX}25-:8080" - netdev+=",hostfwd=tcp::${PORT_PREFIX}36-:8123" - netdev+=",hostfwd=tcp::${PORT_PREFIX}37-:9009" - else - netdev+=",hostfwd=tcp::${PORT_PREFIX}23-:22" - netdev+=",hostfwd=tcp::${PORT_PREFIX}01-:${PORT_PREFIX}01" - netdev+=",hostfwd=tcp::${PORT_PREFIX}02-:${PORT_PREFIX}02" - fi - - local ram cpus - if [ "$role" = "deps" ]; then - ram="$DEPS_RAM" - cpus="$DEPS_CPUS" - else - ram="$APP_RAM" - cpus="$APP_CPUS" - fi + netdev+=",hostfwd=tcp::${PORT_PREFIX}22-:22" + # Deps services + netdev+=",hostfwd=tcp::${PORT_PREFIX}28-:5432" + netdev+=",hostfwd=tcp::${PORT_PREFIX}29-:2500" + netdev+=",hostfwd=tcp::${PORT_PREFIX}05-:9001" + netdev+=",hostfwd=tcp::${PORT_PREFIX}30-:1100" + netdev+=",hostfwd=tcp::${PORT_PREFIX}13-:8071" + netdev+=",hostfwd=tcp::${PORT_PREFIX}21-:9090" + netdev+=",hostfwd=tcp::${PORT_PREFIX}25-:8080" + netdev+=",hostfwd=tcp::${PORT_PREFIX}36-:8123" + netdev+=",hostfwd=tcp::${PORT_PREFIX}37-:9009" + # App services + netdev+=",hostfwd=tcp::${PORT_PREFIX}01-:${PORT_PREFIX}01" + netdev+=",hostfwd=tcp::${PORT_PREFIX}02-:${PORT_PREFIX}02" QEMU_CMD=( "$qemu_bin" @@ -381,18 +354,19 @@ build_qemu_cmd() { -cpu "$cpu" "${firmware_args[@]}" -boot order=c - -m "$ram" - -smp "$cpus" - -drive "file=$role_dir/disk.qcow2,format=qcow2,if=virtio" - -drive "file=$(runtime_iso_path "$role"),format=raw,if=virtio,readonly=on" + -m "$VM_RAM" + -smp "$VM_CPUS" + -drive "file=$VM_DIR/disk.qcow2,format=qcow2,if=virtio" + -drive "file=$(runtime_iso_path),format=raw,if=virtio,readonly=on" -netdev "$netdev" -device virtio-net-pci,netdev=net0 - -chardev "socket,id=monitor,path=$role_dir/monitor.sock,server=on,wait=off" + -device virtio-balloon-pci + -chardev "socket,id=monitor,path=$VM_DIR/monitor.sock,server=on,wait=off" -mon "chardev=monitor,mode=control" - -serial "file:$role_dir/serial.log" + -serial "file:$VM_DIR/serial.log" -display none -daemonize - -pidfile "$role_dir/qemu.pid" + -pidfile "$VM_DIR/qemu.pid" ) if [ "$has_snapshot" = "true" ]; then @@ -402,24 +376,19 @@ build_qemu_cmd() { } is_running() { - local role_dir - role_dir="$(instance_dir "$1")" - if [ ! -f "$role_dir/qemu.pid" ]; then + if [ ! -f "$VM_DIR/qemu.pid" ]; then return 1 fi local pid - pid="$(cat "$role_dir/qemu.pid")" + pid="$(cat "$VM_DIR/qemu.pid")" kill -0 "$pid" 2>/dev/null } -tail_role_logs() { - local role="$1" - local role_dir - role_dir="$(instance_dir "$role")" - if [ -f "$role_dir/serial.log" ]; then +tail_vm_logs() { + if [ -f "$VM_DIR/serial.log" ]; then echo "" - warn "Last serial log lines for ${role}:" - tail -40 "$role_dir/serial.log" || true + warn "Last serial log lines:" + tail -40 "$VM_DIR/serial.log" || true fi } @@ -434,30 +403,24 @@ ensure_ports_free() { done } -start_role() { - local role="$1" - local role_dir - role_dir="$(instance_dir "$role")" - mkdir -p "$role_dir" - : > "$role_dir/serial.log" - prepare_runtime_config_iso "$role" - build_qemu_cmd "$role" +start_vm() { + mkdir -p "$VM_DIR" + : > "$VM_DIR/serial.log" + prepare_runtime_config_iso + build_qemu_cmd "${QEMU_CMD[@]}" } -stop_role() { - local role="$1" - local role_dir - role_dir="$(instance_dir "$role")" - if [ ! -f "$role_dir/qemu.pid" ]; then +stop_vm() { + if [ ! -f "$VM_DIR/qemu.pid" ]; then return 0 fi local pid - pid="$(cat "$role_dir/qemu.pid")" + pid="$(cat "$VM_DIR/qemu.pid")" if kill -0 "$pid" 2>/dev/null; then - if [ -S "$role_dir/monitor.sock" ]; then - echo '{"execute":"qmp_capabilities"}' | socat - UNIX-CONNECT:"$role_dir/monitor.sock" >/dev/null 2>&1 || true - echo '{"execute":"system_powerdown"}' | socat - UNIX-CONNECT:"$role_dir/monitor.sock" >/dev/null 2>&1 || true + if [ -S "$VM_DIR/monitor.sock" ]; then + echo '{"execute":"qmp_capabilities"}' | socat - UNIX-CONNECT:"$VM_DIR/monitor.sock" >/dev/null 2>&1 || true + echo '{"execute":"system_powerdown"}' | socat - UNIX-CONNECT:"$VM_DIR/monitor.sock" >/dev/null 2>&1 || true sleep 3 fi if kill -0 "$pid" 2>/dev/null; then @@ -466,13 +429,9 @@ stop_role() { kill -9 "$pid" 2>/dev/null || true fi fi - rm -f "$role_dir/qemu.pid" "$role_dir/monitor.sock" "$role_dir/serial.log" - rm -rf "$role_dir/runtime-config" - rm -f "$role_dir/runtime-config.iso" -} - -all_ready() { - deps_ready && app_ready + rm -f "$VM_DIR/qemu.pid" "$VM_DIR/monitor.sock" "$VM_DIR/serial.log" + rm -rf "$VM_DIR/runtime-config" + rm -f "$VM_DIR/runtime-config.iso" } cmd_start() { @@ -485,8 +444,7 @@ cmd_start() { info "Starting QEMU local emulator" info "Arch: $ARCH | Accel: $ACCEL | Prefix: $PORT_PREFIX" - start_role deps - start_role dev-server + start_vm if [ "$IS_SNAPSHOT_RESTORE" = "true" ]; then info "Restoring from snapshot..." @@ -495,34 +453,30 @@ cmd_start() { return 0 fi warn "Snapshot restore failed. Resetting and doing fresh boot..." - stop_role dev-server - stop_role deps - rm -rf "$(instance_dir deps)" "$(instance_dir dev-server)" + stop_vm + rm -rf "$VM_DIR" IS_SNAPSHOT_RESTORE=false - start_role deps - start_role dev-server + start_vm fi - info "Deps VM: ${DEPS_RAM}MB/${DEPS_CPUS} CPUs | App VM: ${APP_RAM}MB/${APP_CPUS} CPUs" + info "VM: ${VM_RAM}MB / ${VM_CPUS} CPUs" if ! wait_for_condition "deps services" "$READY_TIMEOUT" deps_ready; then - tail_role_logs deps + tail_vm_logs exit 1 fi if ! wait_for_condition "dashboard/backend" "$READY_TIMEOUT" app_ready; then - tail_role_logs dev-server + tail_vm_logs exit 1 fi - save_snapshot deps - save_snapshot dev-server + save_snapshot log "All services are green. Snapshot saved for fast restart." } cmd_stop() { - stop_role dev-server - stop_role deps + stop_vm stop_file_bridge log "QEMU emulator stopped." } @@ -547,16 +501,11 @@ print_service_status() { } cmd_status() { - echo "Guests:" - if is_running deps; then - echo -e " ${GREEN}●${NC} deps" - else - echo -e " ${RED}●${NC} deps" - fi - if is_running dev-server; then - echo -e " ${GREEN}●${NC} dev-server" + echo "VM:" + if is_running; then + echo -e " ${GREEN}●${NC} emulator" else - echo -e " ${RED}●${NC} dev-server" + echo -e " ${RED}●${NC} emulator" fi echo "" echo "Services:" From c2e4698dfcc1a6692e3c3890a0c31e6372cf5892 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Wed, 18 Mar 2026 15:33:44 -0700 Subject: [PATCH 03/30] Refactor local emulator configuration management and remove obsolete file bridge - Removed the `readConfigFileContentIfExists` and `writeConfigToFile` functions from the local emulator module, simplifying file handling. - Updated the `readConfigFromFile` function to utilize environment variables for configuration content, enhancing flexibility. - Refactored tests to align with the new configuration management approach, ensuring accurate validation of file reading and handling. - Consolidated Docker environment variables into a new `base.env` file for improved organization and maintainability. - Removed the host file bridge script and related logic, streamlining the emulator's architecture and reducing complexity. These changes significantly enhance the local emulator's usability and maintainability for developers. --- .../internal/local-emulator/project/route.tsx | 7 - apps/backend/src/lib/config.tsx | 10 +- apps/backend/src/lib/local-emulator.test.ts | 82 ++------ apps/backend/src/lib/local-emulator.ts | 112 ++--------- docker/local-emulator/base.env | 66 +++++++ docker/local-emulator/docker-compose.yaml | 75 +------- docker/local-emulator/qemu/build-image.sh | 12 +- .../qemu/cloud-init/emulator/user-data | 175 +++--------------- .../local-emulator/qemu/cloud-init/meta-data | 2 - .../local-emulator/qemu/cloud-init/user-data | 6 - .../local-emulator/qemu/host-file-bridge.mjs | 139 -------------- .../local-emulator/qemu/prepare-app-bundle.sh | 5 - .../qemu/prepare-image-bundle.sh | 36 ---- docker/local-emulator/qemu/run-emulator.sh | 156 ++++------------ .../demo/src/app/emulator-status/page.tsx | 2 +- 15 files changed, 167 insertions(+), 718 deletions(-) create mode 100644 docker/local-emulator/base.env delete mode 100644 docker/local-emulator/qemu/cloud-init/meta-data delete mode 100644 docker/local-emulator/qemu/cloud-init/user-data delete mode 100644 docker/local-emulator/qemu/host-file-bridge.mjs delete mode 100755 docker/local-emulator/qemu/prepare-app-bundle.sh delete mode 100755 docker/local-emulator/qemu/prepare-image-bundle.sh diff --git a/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx b/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx index 996039a84c..0635cd388d 100644 --- a/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx +++ b/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx @@ -4,9 +4,7 @@ import { LOCAL_EMULATOR_ONLY_ENDPOINT_MESSAGE, LOCAL_EMULATOR_OWNER_TEAM_ID, isLocalEmulatorEnabled, - readConfigFileContentIfExists, readConfigFromFile, - writeConfigToFile, } from "@/lib/local-emulator"; import { DEFAULT_BRANCH_ID, getSoleTenancyFromProjectBranch } from "@/lib/tenancies"; import { getPrismaClientForTenancy, globalPrismaClient } from "@/prisma-client"; @@ -193,11 +191,6 @@ export const POST = createSmartRouteHandler({ const absoluteFilePath = path.resolve(req.body.absolute_file_path); - const fileContent = await readConfigFileContentIfExists(absoluteFilePath) ?? ""; - if (fileContent.trim() === "") { - await writeConfigToFile(absoluteFilePath, {}); - } - await assertLocalEmulatorOwnerTeamReadiness(); const projectId = await getOrCreateLocalEmulatorProjectId(absoluteFilePath); diff --git a/apps/backend/src/lib/config.tsx b/apps/backend/src/lib/config.tsx index 078dd9be83..c1c90f410d 100644 --- a/apps/backend/src/lib/config.tsx +++ b/apps/backend/src/lib/config.tsx @@ -11,7 +11,7 @@ import { Result } from "@stackframe/stack-shared/dist/utils/results"; import { deindent, stringCompare } from "@stackframe/stack-shared/dist/utils/strings"; import * as yup from "yup"; import { RawQuery, globalPrismaClient, rawQuery } from "../prisma-client"; -import { LOCAL_EMULATOR_ENV_CONFIG_BLOCKED_MESSAGE, getLocalEmulatorFilePath, isLocalEmulatorEnabled, isLocalEmulatorProject, readConfigFromFile, writeConfigToFile } from "./local-emulator"; +import { LOCAL_EMULATOR_ENV_CONFIG_BLOCKED_MESSAGE, isLocalEmulatorEnabled, isLocalEmulatorProject, readConfigFromFile } from "./local-emulator"; import { listPermissionDefinitionsFromConfig } from "./permissions"; type BranchConfigSourceApi = yup.InferType; @@ -266,14 +266,6 @@ export async function setBranchConfigOverride(options: { }): Promise { const newConfig = migrateConfigOverride("branch", options.branchConfigOverride); - if (isLocalEmulatorEnabled() && await isLocalEmulatorProject(options.projectId)) { - const filePath = await getLocalEmulatorFilePath(options.projectId); - if (filePath) { - await writeConfigToFile(filePath, newConfig); - return; - } - } - // large configs make our DB slow; let's prevent them early const newConfigString = JSON.stringify(newConfig); if (newConfigString.length > 1_000_000) { diff --git a/apps/backend/src/lib/local-emulator.test.ts b/apps/backend/src/lib/local-emulator.test.ts index 4fbdf4d7da..028c5000ba 100644 --- a/apps/backend/src/lib/local-emulator.test.ts +++ b/apps/backend/src/lib/local-emulator.test.ts @@ -1,91 +1,33 @@ import { afterEach, describe, expect, it, vi } from "vitest"; -import { readConfigFileContentIfExists, readConfigFromFile, writeConfigToFile } from "./local-emulator"; +import { readConfigFromFile } from "./local-emulator"; -describe("local emulator file bridge", () => { +describe("local emulator config", () => { afterEach(() => { vi.unstubAllEnvs(); vi.restoreAllMocks(); - vi.unstubAllGlobals(); }); - it("reads config files through the host file bridge when configured", async () => { - vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL", "http://127.0.0.1:8116"); - vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN", "bridge-token"); + it("reads config from STACK_LOCAL_EMULATOR_CONFIG_CONTENT env var when set", async () => { + const content = `export const config = { auth: { allowLocalhost: true } };\n`; + vi.stubEnv("STACK_LOCAL_EMULATOR_CONFIG_CONTENT", Buffer.from(content).toString("base64")); - const fetchMock = vi.fn(async () => { - return new Response(JSON.stringify({ - exists: true, - content: "export const config = { auth: { allowLocalhost: true } };\n", - }), { - status: 200, - headers: { - "Content-Type": "application/json", - }, - }); - }); - vi.stubGlobal("fetch", fetchMock); - - await expect(readConfigFromFile("/Users/tester/example/stack.config.ts")).resolves.toMatchInlineSnapshot(` + await expect(readConfigFromFile("/irrelevant/path/stack.config.ts")).resolves.toMatchInlineSnapshot(` { "auth": { "allowLocalhost": true, }, } `); - expect(fetchMock).toHaveBeenCalledTimes(1); - expect(fetchMock).toHaveBeenCalledWith( - new URL("/read", "http://127.0.0.1:8116"), - expect.objectContaining({ - method: "POST", - headers: expect.objectContaining({ - "Content-Type": "application/json", - "X-Stack-Emulator-Token": "bridge-token", - }), - }), - ); }); - it("returns null for missing files when the host file bridge reports they do not exist", async () => { - vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL", "http://127.0.0.1:8116"); - vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN", "bridge-token"); - - vi.stubGlobal("fetch", vi.fn(async () => { - return new Response(JSON.stringify({ exists: false }), { - status: 200, - headers: { - "Content-Type": "application/json", - }, - }); - })); - - await expect(readConfigFileContentIfExists("/Users/tester/example/missing-stack.config.ts")).resolves.toBeNull(); + it("returns empty object when env var is not set and file does not exist", async () => { + await expect(readConfigFromFile("/nonexistent/stack.config.ts")).resolves.toEqual({}); }); - it("writes config files through the host file bridge when configured", async () => { - vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL", "http://127.0.0.1:8116"); - vi.stubEnv("STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN", "bridge-token"); - - const fetchMock = vi.fn(async () => { - return new Response(JSON.stringify({ ok: true }), { - status: 200, - headers: { - "Content-Type": "application/json", - }, - }); - }); - vi.stubGlobal("fetch", fetchMock); - - await writeConfigToFile("/Users/tester/example/stack.config.ts", { teams: { enabled: true } }); + it("returns empty object when env var content is empty", async () => { + const content = ``; + vi.stubEnv("STACK_LOCAL_EMULATOR_CONFIG_CONTENT", Buffer.from(content).toString("base64")); - expect(fetchMock).toHaveBeenCalledWith( - new URL("/write", "http://127.0.0.1:8116"), - expect.objectContaining({ - method: "POST", - body: JSON.stringify({ - path: "/Users/tester/example/stack.config.ts", - content: `export const config = ${JSON.stringify({ teams: { enabled: true } }, null, 2)};\n`, - }), - }), - ); + await expect(readConfigFromFile("/irrelevant/path/stack.config.ts")).resolves.toEqual({}); }); }); diff --git a/apps/backend/src/lib/local-emulator.ts b/apps/backend/src/lib/local-emulator.ts index a13815f059..690a5a493d 100644 --- a/apps/backend/src/lib/local-emulator.ts +++ b/apps/backend/src/lib/local-emulator.ts @@ -3,7 +3,7 @@ import path from "path"; import { createJiti } from "jiti"; import { isValidConfig } from "@stackframe/stack-shared/dist/config/format"; import { getEnvVariable } from "@stackframe/stack-shared/dist/utils/env"; -import { StackAssertionError, StatusError } from "@stackframe/stack-shared/dist/utils/errors"; +import { StatusError } from "@stackframe/stack-shared/dist/utils/errors"; import { globalPrismaClient } from "@/prisma-client"; export const LOCAL_EMULATOR_ADMIN_USER_ID = "63abbc96-5329-454a-ba56-e0460173c6c1"; @@ -36,102 +36,17 @@ export async function isLocalEmulatorProject(projectId: string) { return project !== null; } -export async function getLocalEmulatorFilePath(projectId: string): Promise { - const result = await globalPrismaClient.localEmulatorProject.findUnique({ - where: { projectId }, - select: { absoluteFilePath: true }, - }); - return result?.absoluteFilePath ?? null; -} - -function getLocalEmulatorFileBridgeConfig() { - const url = getEnvVariable("STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL", ""); - const token = getEnvVariable("STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN", ""); - if (url === "") { - return null; - } - if (token === "") { - throw new StackAssertionError("STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN must be set when STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL is configured."); - } - return { url, token }; -} - -function isObject(value: unknown): value is Record { - return typeof value === "object" && value !== null; -} - -async function requestLocalEmulatorFileBridge(pathname: string, body: Record): Promise { - const bridgeConfig = getLocalEmulatorFileBridgeConfig(); - if (bridgeConfig === null) { - throw new StackAssertionError("Local emulator file bridge is not configured."); - } - - const response = await fetch(new URL(pathname, bridgeConfig.url), { - method: "POST", - headers: { - "Content-Type": "application/json", - "X-Stack-Emulator-Token": bridgeConfig.token, - }, - body: JSON.stringify(body), - }); - - const responseText = await response.text(); - if (!response.ok) { - throw new StackAssertionError(`Local emulator file bridge request failed: ${response.status} ${responseText || response.statusText}`); - } - - try { - return JSON.parse(responseText) as unknown; - } catch { - throw new StackAssertionError(`Local emulator file bridge returned invalid JSON for ${pathname}.`); - } -} - -export async function readConfigFileContentIfExists(filePath: string): Promise { - const bridgeConfig = getLocalEmulatorFileBridgeConfig(); - if (bridgeConfig !== null) { - const responseJson = await requestLocalEmulatorFileBridge("/read", { path: filePath }); - if (!isObject(responseJson) || typeof responseJson.exists !== "boolean") { - throw new StackAssertionError("Local emulator file bridge returned an invalid read response.", { responseJson }); - } - if (!responseJson.exists) { - return null; - } - if (typeof responseJson.content !== "string") { - throw new StackAssertionError("Local emulator file bridge read response is missing file content.", { responseJson }); - } - return responseJson.content; - } - - try { - return await fs.readFile(filePath, "utf-8"); - } catch (error: unknown) { - if ((error as NodeJS.ErrnoException).code === "ENOENT") { - return null; - } - throw error; - } -} - -async function writeConfigFileContent(filePath: string, content: string): Promise { - const bridgeConfig = getLocalEmulatorFileBridgeConfig(); - if (bridgeConfig !== null) { - const responseJson = await requestLocalEmulatorFileBridge("/write", { path: filePath, content }); - if (!isObject(responseJson) || responseJson.ok !== true) { - throw new StackAssertionError("Local emulator file bridge returned an invalid write response.", { responseJson }); - } - return; - } - - const dir = path.dirname(filePath); - await fs.mkdir(dir, { recursive: true }); - await fs.writeFile(filePath, content, "utf-8"); -} - export async function readConfigFromFile(filePath: string): Promise> { - const content = await readConfigFileContentIfExists(filePath); - if (content === null) { - throw new StatusError(StatusError.BadRequest, `Config file not found: ${filePath}`); + const configContentBase64 = getEnvVariable("STACK_LOCAL_EMULATOR_CONFIG_CONTENT", ""); + const content = configContentBase64 !== "" + ? Buffer.from(configContentBase64, "base64").toString("utf-8") + : await fs.readFile(filePath, "utf-8").catch((error: unknown) => { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; + throw error; + }); + + if (content === null || content.trim() === "") { + return {}; } const jiti = createJiti(import.meta.url, { cache: false }); @@ -142,8 +57,3 @@ export async function readConfigFromFile(filePath: string): Promise): Promise { - const content = `export const config = ${JSON.stringify(config, null, 2)};\n`; - await writeConfigFileContent(filePath, content); -} diff --git a/docker/local-emulator/base.env b/docker/local-emulator/base.env new file mode 100644 index 0000000000..b5bca9b838 --- /dev/null +++ b/docker/local-emulator/base.env @@ -0,0 +1,66 @@ +NEXT_PUBLIC_STACK_DOCS_BASE_URL=https://docs.stack-auth.com +NEXT_PUBLIC_STACK_IS_LOCAL_EMULATOR=true +NEXT_PUBLIC_STACK_PROJECT_ID=internal +NEXT_PUBLIC_STACK_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only +STACK_SECRET_SERVER_KEY=this-secret-server-key-is-for-local-development-only +STACK_SERVER_SECRET=23-wuNpik0gIW4mruTz25rbIvhuuvZFrLOLtL7J4tyo +STACK_CHANGELOG_URL=https://raw.githubusercontent.com/stack-auth/stack-auth/refs/heads/dev/CHANGELOG.md +STACK_SEED_ENABLE_DUMMY_PROJECT=true +STACK_SEED_INTERNAL_PROJECT_SIGN_UP_ENABLED=true +STACK_SEED_INTERNAL_PROJECT_OTP_ENABLED=true +STACK_SEED_INTERNAL_PROJECT_ALLOW_LOCALHOST=true +STACK_SEED_INTERNAL_PROJECT_OAUTH_PROVIDERS=github,spotify,google,microsoft +STACK_SEED_INTERNAL_PROJECT_USER_INTERNAL_ACCESS=true +STACK_SEED_INTERNAL_PROJECT_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only +STACK_SEED_INTERNAL_PROJECT_SECRET_SERVER_KEY=this-secret-server-key-is-for-local-development-only +STACK_SEED_INTERNAL_PROJECT_SUPER_SECRET_ADMIN_KEY=this-super-secret-admin-key-is-for-local-development-only +STACK_SVIX_API_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NTUxNDA2MzksImV4cCI6MTk3MDUwMDYzOSwibmJmIjoxNjU1MTQwNjM5LCJpc3MiOiJzdml4LXNlcnZlciIsInN1YiI6Im9yZ18yM3JiOFlkR3FNVDBxSXpwZ0d3ZFhmSGlyTXUifQ.En8w77ZJWbd0qrMlHHupHUB-4cx17RfzFykseg95SUk +STACK_OPENAI_API_KEY=mock_openai_api_key +STACK_OPENROUTER_API_KEY=FORWARD_TO_PRODUCTION +STACK_STRIPE_SECRET_KEY=sk_test_mockstripekey +STACK_STRIPE_WEBHOOK_SECRET=mock_stripe_webhook_secret +STACK_RESEND_API_KEY=mock_resend_api_key +STACK_RESEND_WEBHOOK_SECRET=mock_resend_webhook_secret +STACK_DNSIMPLE_API_TOKEN=mock_dnsimple_api_token +STACK_DNSIMPLE_ACCOUNT_ID=mock_dnsimple_account_id +STACK_DNSIMPLE_API_BASE_URL=https://api.dnsimple.com/v2 +STACK_FREESTYLE_API_KEY=mock_stack_freestyle_key +STACK_VERCEL_SANDBOX_TOKEN=vercel_sandbox_disabled_for_local_development +CRON_SECRET=mock_cron_secret +STACK_S3_REGION=us-east-1 +STACK_S3_ACCESS_KEY_ID=s3mockroot +STACK_S3_SECRET_ACCESS_KEY=s3mockroot +STACK_S3_BUCKET=stack-storage +STACK_S3_PRIVATE_BUCKET=stack-storage-private +STACK_AWS_REGION=us-east-1 +STACK_AWS_ACCESS_KEY_ID=test +STACK_AWS_SECRET_ACCESS_KEY=test +STACK_QSTASH_TOKEN=eyJVc2VySUQiOiJkZWZhdWx0VXNlciIsIlBhc3N3b3JkIjoiZGVmYXVsdFBhc3N3b3JkIn0= +STACK_QSTASH_CURRENT_SIGNING_KEY=sig_7kYjw48mhY7kAjqNGcy6cr29RJ6r +STACK_QSTASH_NEXT_SIGNING_KEY=sig_5ZB6DVzB1wjE8S6rZ7eenA8Pdnhs +STACK_CLICKHOUSE_ADMIN_USER=stackframe +STACK_CLICKHOUSE_ADMIN_PASSWORD=PASSWORD-PLACEHOLDER--9gKyMxJeMx +STACK_CLICKHOUSE_EXTERNAL_PASSWORD=PASSWORD-PLACEHOLDER--EZeHscBMzE +STACK_EMAIL_PORT=2500 +STACK_EMAIL_SECURE=false +STACK_EMAIL_USERNAME=does-not-matter +STACK_EMAIL_PASSWORD=does-not-matter +STACK_EMAIL_SENDER=noreply@example.com +STACK_DEFAULT_EMAIL_CAPACITY_PER_HOUR=10000 +STACK_EMAIL_MONITOR_PROJECT_ID=internal +STACK_EMAIL_MONITOR_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only +STACK_EMAIL_MONITOR_RESEND_EMAIL_DOMAIN=stack-generated.example.com +STACK_EMAIL_MONITOR_RESEND_EMAIL_API_KEY=this-is-a-fake-key +STACK_EMAIL_MONITOR_SECRET_TOKEN=this-secret-token-is-for-local-development-only +STACK_EMAIL_MONITOR_USE_INBUCKET=true +STACK_FEATUREBASE_JWT_SECRET=secret-value +STACK_FORWARD_MOCK_OAUTH_SERVER=false +STACK_GITHUB_CLIENT_ID=MOCK +STACK_GITHUB_CLIENT_SECRET=MOCK +STACK_GOOGLE_CLIENT_ID=MOCK +STACK_GOOGLE_CLIENT_SECRET=MOCK +STACK_MICROSOFT_CLIENT_ID=MOCK +STACK_MICROSOFT_CLIENT_SECRET=MOCK +STACK_SPOTIFY_CLIENT_ID=MOCK +STACK_SPOTIFY_CLIENT_SECRET=MOCK +STACK_ALLOW_SHARED_OAUTH_ACCESS_TOKENS=true diff --git a/docker/local-emulator/docker-compose.yaml b/docker/local-emulator/docker-compose.yaml index 7d5158b59b..116cc7df0c 100644 --- a/docker/local-emulator/docker-compose.yaml +++ b/docker/local-emulator/docker-compose.yaml @@ -46,87 +46,26 @@ services: volumes: - "${HOME}:${HOME}" - "/tmp:/tmp" + env_file: ./base.env environment: + # Port-prefixed URLs (host-specific) NEXT_PUBLIC_STACK_PORT_PREFIX: "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}" NEXT_PUBLIC_STACK_API_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02" NEXT_PUBLIC_STACK_DASHBOARD_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01" - NEXT_PUBLIC_STACK_DOCS_BASE_URL: "https://docs.stack-auth.com" - NEXT_PUBLIC_STACK_IS_LOCAL_EMULATOR: "true" - NEXT_PUBLIC_STACK_PROJECT_ID: "internal" - NEXT_PUBLIC_STACK_PUBLISHABLE_CLIENT_KEY: "this-publishable-client-key-is-for-local-development-only" - STACK_SECRET_SERVER_KEY: "this-secret-server-key-is-for-local-development-only" - STACK_SERVER_SECRET: "23-wuNpik0gIW4mruTz25rbIvhuuvZFrLOLtL7J4tyo" NEXT_PUBLIC_STACK_SVIX_SERVER_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}13" - STACK_CHANGELOG_URL: "https://raw.githubusercontent.com/stack-auth/stack-auth/refs/heads/dev/CHANGELOG.md" - STACK_SEED_ENABLE_DUMMY_PROJECT: "true" - STACK_SEED_INTERNAL_PROJECT_SIGN_UP_ENABLED: "true" - STACK_SEED_INTERNAL_PROJECT_OTP_ENABLED: "true" - STACK_SEED_INTERNAL_PROJECT_ALLOW_LOCALHOST: "true" - STACK_SEED_INTERNAL_PROJECT_OAUTH_PROVIDERS: "github,spotify,google,microsoft" - STACK_SEED_INTERNAL_PROJECT_USER_INTERNAL_ACCESS: "true" - STACK_SEED_INTERNAL_PROJECT_PUBLISHABLE_CLIENT_KEY: "this-publishable-client-key-is-for-local-development-only" - STACK_SEED_INTERNAL_PROJECT_SECRET_SERVER_KEY: "this-secret-server-key-is-for-local-development-only" - STACK_SEED_INTERNAL_PROJECT_SUPER_SECRET_ADMIN_KEY: "this-super-secret-admin-key-is-for-local-development-only" + STACK_S3_PUBLIC_ENDPOINT: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}21/stack-storage" + STACK_EMAIL_MONITOR_VERIFICATION_CALLBACK_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01/handler/email-verification" STACK_OAUTH_MOCK_URL: "http://host.docker.internal:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}14" - STACK_GITHUB_CLIENT_ID: "MOCK" - STACK_GITHUB_CLIENT_SECRET: "MOCK" - STACK_GOOGLE_CLIENT_ID: "MOCK" - STACK_GOOGLE_CLIENT_SECRET: "MOCK" - STACK_MICROSOFT_CLIENT_ID: "MOCK" - STACK_MICROSOFT_CLIENT_SECRET: "MOCK" - STACK_SPOTIFY_CLIENT_ID: "MOCK" - STACK_SPOTIFY_CLIENT_SECRET: "MOCK" - STACK_ALLOW_SHARED_OAUTH_ACCESS_TOKENS: "true" + BACKEND_PORT: "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02" + DASHBOARD_PORT: "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01" + # Deps service hostnames (stack-deps container, not localhost) STACK_DATABASE_CONNECTION_STRING: "postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@stack-deps:5432/stackframe" STACK_EMAIL_HOST: "stack-deps" - STACK_EMAIL_PORT: "2500" - STACK_EMAIL_SECURE: "false" - STACK_EMAIL_USERNAME: "does-not-matter" - STACK_EMAIL_PASSWORD: "does-not-matter" - STACK_EMAIL_SENDER: "noreply@example.com" - STACK_DEFAULT_EMAIL_CAPACITY_PER_HOUR: "10000" STACK_SVIX_SERVER_URL: "http://stack-deps:8071" - STACK_SVIX_API_KEY: "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NTUxNDA2MzksImV4cCI6MTk3MDUwMDYzOSwibmJmIjoxNjU1MTQwNjM5LCJpc3MiOiJzdml4LXNlcnZlciIsInN1YiI6Im9yZ18yM3JiOFlkR3FNVDBxSXpwZ0d3ZFhmSGlyTXUifQ.En8w77ZJWbd0qrMlHHupHUB-4cx17RfzFykseg95SUk" - STACK_OPENAI_API_KEY: "mock_openai_api_key" - STACK_OPENROUTER_API_KEY: "FORWARD_TO_PRODUCTION" - STACK_STRIPE_SECRET_KEY: "sk_test_mockstripekey" - STACK_STRIPE_WEBHOOK_SECRET: "mock_stripe_webhook_secret" - STACK_RESEND_API_KEY: "mock_resend_api_key" - STACK_RESEND_WEBHOOK_SECRET: "mock_resend_webhook_secret" - STACK_DNSIMPLE_API_TOKEN: "mock_dnsimple_api_token" - STACK_DNSIMPLE_ACCOUNT_ID: "mock_dnsimple_account_id" - STACK_DNSIMPLE_API_BASE_URL: "https://api.dnsimple.com/v2" - STACK_FREESTYLE_API_KEY: "mock_stack_freestyle_key" - STACK_VERCEL_SANDBOX_TOKEN: "vercel_sandbox_disabled_for_local_development" - CRON_SECRET: "mock_cron_secret" STACK_S3_ENDPOINT: "http://stack-deps:9090" - STACK_S3_PUBLIC_ENDPOINT: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}21/stack-storage" - STACK_S3_REGION: "us-east-1" - STACK_S3_ACCESS_KEY_ID: "s3mockroot" - STACK_S3_SECRET_ACCESS_KEY: "s3mockroot" - STACK_S3_BUCKET: "stack-storage" - STACK_S3_PRIVATE_BUCKET: "stack-storage-private" - STACK_AWS_REGION: "us-east-1" - STACK_AWS_ACCESS_KEY_ID: "test" - STACK_AWS_SECRET_ACCESS_KEY: "test" STACK_QSTASH_URL: "http://stack-deps:8080" - STACK_QSTASH_TOKEN: "eyJVc2VySUQiOiJkZWZhdWx0VXNlciIsIlBhc3N3b3JkIjoiZGVmYXVsdFBhc3N3b3JkIn0=" - STACK_QSTASH_CURRENT_SIGNING_KEY: "sig_7kYjw48mhY7kAjqNGcy6cr29RJ6r" - STACK_QSTASH_NEXT_SIGNING_KEY: "sig_5ZB6DVzB1wjE8S6rZ7eenA8Pdnhs" STACK_CLICKHOUSE_URL: "http://stack-deps:8123" - STACK_CLICKHOUSE_ADMIN_USER: "stackframe" - STACK_CLICKHOUSE_ADMIN_PASSWORD: "PASSWORD-PLACEHOLDER--9gKyMxJeMx" - STACK_CLICKHOUSE_EXTERNAL_PASSWORD: "PASSWORD-PLACEHOLDER--EZeHscBMzE" - STACK_EMAIL_MONITOR_VERIFICATION_CALLBACK_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01/handler/email-verification" - STACK_EMAIL_MONITOR_PROJECT_ID: "internal" - STACK_EMAIL_MONITOR_PUBLISHABLE_CLIENT_KEY: "this-publishable-client-key-is-for-local-development-only" - STACK_EMAIL_MONITOR_RESEND_EMAIL_DOMAIN: "stack-generated.example.com" - STACK_EMAIL_MONITOR_RESEND_EMAIL_API_KEY: "this-is-a-fake-key" STACK_EMAIL_MONITOR_INBUCKET_API_URL: "http://stack-deps:9001" - STACK_EMAIL_MONITOR_USE_INBUCKET: "true" - STACK_EMAIL_MONITOR_SECRET_TOKEN: "this-secret-token-is-for-local-development-only" - STACK_FEATUREBASE_JWT_SECRET: "secret-value" - STACK_FORWARD_MOCK_OAUTH_SERVER: "false" extra_hosts: - "host.docker.internal:host-gateway" healthcheck: diff --git a/docker/local-emulator/qemu/build-image.sh b/docker/local-emulator/qemu/build-image.sh index d5c78adfaf..77c714e22f 100755 --- a/docker/local-emulator/qemu/build-image.sh +++ b/docker/local-emulator/qemu/build-image.sh @@ -7,7 +7,6 @@ source "$SCRIPT_DIR/common.sh" IMAGE_DIR="$SCRIPT_DIR/images" CLOUD_INIT_ROOT="$SCRIPT_DIR/cloud-init" -PREPARE_IMAGE_BUNDLE_SCRIPT="$SCRIPT_DIR/prepare-image-bundle.sh" DEBIAN_VERSION="${DEBIAN_VERSION:-13}" DISK_SIZE="${EMULATOR_DISK_SIZE:-12G}" @@ -140,7 +139,16 @@ prepare_bundle_artifacts() { fi log "Creating Docker image bundle (${arch})..." - "$PREPARE_IMAGE_BUNDLE_SCRIPT" "$bundle_tgz" "${DOCKER_IMAGES[@]}" + for img in "${DOCKER_IMAGES[@]}"; do + if ! docker image inspect "$img" >/dev/null 2>&1; then + err "Missing Docker image: $img. Build the local emulator images first, then rerun the QEMU image build." + exit 1 + fi + done + local tmp_bundle="${bundle_tgz}.tmp" + rm -f "$tmp_bundle" + docker save "${DOCKER_IMAGES[@]}" | gzip -c > "$tmp_bundle" + mv "$tmp_bundle" "$bundle_tgz" printf "%s" "$current_ids" > "$bundle_meta" } diff --git a/docker/local-emulator/qemu/cloud-init/emulator/user-data b/docker/local-emulator/qemu/cloud-init/emulator/user-data index ef163c6592..a126eb2ee4 100644 --- a/docker/local-emulator/qemu/cloud-init/emulator/user-data +++ b/docker/local-emulator/qemu/cloud-init/emulator/user-data @@ -15,7 +15,7 @@ chpasswd: stack:stack-emulator expire: false -ssh_pwauth: true +ssh_pwauth: false package_update: true package_upgrade: false @@ -28,68 +28,6 @@ packages: - qemu-guest-agent write_files: - - path: /etc/stack-auth/local-emulator.base.env - content: | - NEXT_PUBLIC_STACK_DOCS_BASE_URL=https://docs.stack-auth.com - NEXT_PUBLIC_STACK_IS_LOCAL_EMULATOR=true - NEXT_PUBLIC_STACK_PROJECT_ID=internal - NEXT_PUBLIC_STACK_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only - STACK_SECRET_SERVER_KEY=this-secret-server-key-is-for-local-development-only - STACK_SERVER_SECRET=23-wuNpik0gIW4mruTz25rbIvhuuvZFrLOLtL7J4tyo - STACK_CHANGELOG_URL=https://raw.githubusercontent.com/stack-auth/stack-auth/refs/heads/dev/CHANGELOG.md - STACK_SEED_ENABLE_DUMMY_PROJECT=true - STACK_SEED_INTERNAL_PROJECT_SIGN_UP_ENABLED=true - STACK_SEED_INTERNAL_PROJECT_OTP_ENABLED=true - STACK_SEED_INTERNAL_PROJECT_ALLOW_LOCALHOST=true - STACK_SEED_INTERNAL_PROJECT_OAUTH_PROVIDERS=github,spotify,google,microsoft - STACK_SEED_INTERNAL_PROJECT_USER_INTERNAL_ACCESS=true - STACK_SEED_INTERNAL_PROJECT_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only - STACK_SEED_INTERNAL_PROJECT_SECRET_SERVER_KEY=this-secret-server-key-is-for-local-development-only - STACK_SEED_INTERNAL_PROJECT_SUPER_SECRET_ADMIN_KEY=this-super-secret-admin-key-is-for-local-development-only - STACK_SVIX_API_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NTUxNDA2MzksImV4cCI6MTk3MDUwMDYzOSwibmJmIjoxNjU1MTQwNjM5LCJpc3MiOiJzdml4LXNlcnZlciIsInN1YiI6Im9yZ18yM3JiOFlkR3FNVDBxSXpwZ0d3ZFhmSGlyTXUifQ.En8w77ZJWbd0qrMlHHupHUB-4cx17RfzFykseg95SUk - STACK_OPENAI_API_KEY=mock_openai_api_key - STACK_OPENROUTER_API_KEY=FORWARD_TO_PRODUCTION - STACK_STRIPE_SECRET_KEY=sk_test_mockstripekey - STACK_STRIPE_WEBHOOK_SECRET=mock_stripe_webhook_secret - STACK_RESEND_API_KEY=mock_resend_api_key - STACK_RESEND_WEBHOOK_SECRET=mock_resend_webhook_secret - STACK_DNSIMPLE_API_TOKEN=mock_dnsimple_api_token - STACK_DNSIMPLE_ACCOUNT_ID=mock_dnsimple_account_id - STACK_DNSIMPLE_API_BASE_URL=https://api.dnsimple.com/v2 - STACK_FREESTYLE_API_KEY=mock_stack_freestyle_key - STACK_VERCEL_SANDBOX_TOKEN=vercel_sandbox_disabled_for_local_development - CRON_SECRET=mock_cron_secret - STACK_S3_REGION=us-east-1 - STACK_S3_ACCESS_KEY_ID=s3mockroot - STACK_S3_SECRET_ACCESS_KEY=s3mockroot - STACK_S3_BUCKET=stack-storage - STACK_S3_PRIVATE_BUCKET=stack-storage-private - STACK_AWS_REGION=us-east-1 - STACK_AWS_ACCESS_KEY_ID=test - STACK_AWS_SECRET_ACCESS_KEY=test - STACK_QSTASH_TOKEN=eyJVc2VySUQiOiJkZWZhdWx0VXNlciIsIlBhc3N3b3JkIjoiZGVmYXVsdFBhc3N3b3JkIn0= - STACK_QSTASH_CURRENT_SIGNING_KEY=sig_7kYjw48mhY7kAjqNGcy6cr29RJ6r - STACK_QSTASH_NEXT_SIGNING_KEY=sig_5ZB6DVzB1wjE8S6rZ7eenA8Pdnhs - STACK_CLICKHOUSE_ADMIN_USER=stackframe - STACK_CLICKHOUSE_ADMIN_PASSWORD=PASSWORD-PLACEHOLDER--9gKyMxJeMx - STACK_CLICKHOUSE_EXTERNAL_PASSWORD=PASSWORD-PLACEHOLDER--EZeHscBMzE - STACK_EMAIL_MONITOR_PROJECT_ID=internal - STACK_EMAIL_MONITOR_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only - STACK_EMAIL_MONITOR_RESEND_EMAIL_DOMAIN=stack-generated.example.com - STACK_EMAIL_MONITOR_RESEND_EMAIL_API_KEY=this-is-a-fake-key - STACK_EMAIL_MONITOR_SECRET_TOKEN=this-secret-token-is-for-local-development-only - STACK_FEATUREBASE_JWT_SECRET=secret-value - STACK_FORWARD_MOCK_OAUTH_SERVER=false - STACK_GITHUB_CLIENT_ID=MOCK - STACK_GITHUB_CLIENT_SECRET=MOCK - STACK_GOOGLE_CLIENT_ID=MOCK - STACK_GOOGLE_CLIENT_SECRET=MOCK - STACK_MICROSOFT_CLIENT_ID=MOCK - STACK_MICROSOFT_CLIENT_SECRET=MOCK - STACK_SPOTIFY_CLIENT_ID=MOCK - STACK_SPOTIFY_CLIENT_SECRET=MOCK - STACK_ALLOW_SHARED_OAUTH_ACCESS_TOKENS=true - - path: /usr/local/bin/install-emulator-containers permissions: '0755' content: | @@ -135,103 +73,43 @@ write_files: set -a source /mnt/stack-runtime/runtime.env - source /etc/stack-auth/local-emulator.base.env + source /mnt/stack-runtime/base.env set +a # Deps runs in the same VM, so always localhost DEPS_HOST=127.0.0.1 + P="$STACK_EMULATOR_PORT_PREFIX" + + { + # Static vars from base config and runtime (e.g. API keys, feature flags) + cat /mnt/stack-runtime/base.env + cat /mnt/stack-runtime/runtime.env - cat > /run/stack-auth/local-emulator.env < /run/stack-auth/local-emulator.env - path: /usr/local/bin/run-stack-app-container permissions: '0755' @@ -296,7 +174,8 @@ write_files: WantedBy=multi-user.target runcmd: - - mkdir -p /etc/stack-auth + - systemctl disable --now ssh || true + - systemctl mask ssh || true - bash /usr/local/bin/install-emulator-containers - systemctl daemon-reload - systemctl enable stack-deps.service diff --git a/docker/local-emulator/qemu/cloud-init/meta-data b/docker/local-emulator/qemu/cloud-init/meta-data deleted file mode 100644 index 15c4b5b8e0..0000000000 --- a/docker/local-emulator/qemu/cloud-init/meta-data +++ /dev/null @@ -1,2 +0,0 @@ -instance-id: stack-emulator-split -local-hostname: stack-emulator diff --git a/docker/local-emulator/qemu/cloud-init/user-data b/docker/local-emulator/qemu/cloud-init/user-data deleted file mode 100644 index 6a39bc4d16..0000000000 --- a/docker/local-emulator/qemu/cloud-init/user-data +++ /dev/null @@ -1,6 +0,0 @@ -#cloud-config -write_files: - - path: /etc/motd - content: | - Stack Auth local emulator uses the unified cloud-init definition in - docker/local-emulator/qemu/cloud-init/emulator/. diff --git a/docker/local-emulator/qemu/host-file-bridge.mjs b/docker/local-emulator/qemu/host-file-bridge.mjs deleted file mode 100644 index 22d14c5c37..0000000000 --- a/docker/local-emulator/qemu/host-file-bridge.mjs +++ /dev/null @@ -1,139 +0,0 @@ -#!/usr/bin/env node - -import http from "node:http"; -import fs from "node:fs/promises"; -import os from "node:os"; -import path from "node:path"; - -const port = Number.parseInt(process.env.STACK_QEMU_FILE_BRIDGE_PORT ?? "", 10) || 8116; -const host = process.env.STACK_QEMU_FILE_BRIDGE_HOST ?? "0.0.0.0"; -const token = process.env.STACK_QEMU_FILE_BRIDGE_TOKEN ?? ""; - -if (token === "") { - console.error("STACK_QEMU_FILE_BRIDGE_TOKEN is required"); - process.exit(1); -} - -const allowedRoots = [os.homedir(), "/tmp"].map((root) => path.resolve(root)); - -function isWithinRoot(filePath, rootPath) { - return filePath === rootPath || filePath.startsWith(`${rootPath}${path.sep}`); -} - -function isAllowedPath(filePath) { - return allowedRoots.some((rootPath) => isWithinRoot(filePath, rootPath)); -} - -function sendJson(res, statusCode, body) { - res.writeHead(statusCode, { "Content-Type": "application/json" }); - res.end(JSON.stringify(body)); -} - -function sendText(res, statusCode, message) { - res.writeHead(statusCode, { "Content-Type": "text/plain; charset=utf-8" }); - res.end(message); -} - -async function readBody(req) { - let body = ""; - for await (const chunk of req) { - body += chunk.toString(); - } - return body; -} - -function parseRequestBody(bodyText) { - try { - const body = JSON.parse(bodyText); - return typeof body === "object" && body !== null ? body : null; - } catch { - return null; - } -} - -async function handleRead(res, body) { - const requestedPath = body.path; - if (typeof requestedPath !== "string") { - sendText(res, 400, "Body must include a string 'path'."); - return; - } - - const filePath = path.resolve(requestedPath); - if (!isAllowedPath(filePath)) { - sendText(res, 403, `Path is not allowed: ${filePath}`); - return; - } - - try { - const content = await fs.readFile(filePath, "utf-8"); - sendJson(res, 200, { exists: true, content }); - } catch (error) { - if (error?.code === "ENOENT") { - sendJson(res, 200, { exists: false }); - return; - } - throw error; - } -} - -async function handleWrite(res, body) { - const requestedPath = body.path; - const content = body.content; - if (typeof requestedPath !== "string" || typeof content !== "string") { - sendText(res, 400, "Body must include string 'path' and 'content' fields."); - return; - } - - const filePath = path.resolve(requestedPath); - if (!isAllowedPath(filePath)) { - sendText(res, 403, `Path is not allowed: ${filePath}`); - return; - } - - await fs.mkdir(path.dirname(filePath), { recursive: true }); - await fs.writeFile(filePath, content, "utf-8"); - sendJson(res, 200, { ok: true }); -} - -const server = http.createServer(async (req, res) => { - try { - if (req.url === "/health") { - sendText(res, 200, "ok"); - return; - } - - if (req.method !== "POST") { - sendText(res, 405, "Method not allowed"); - return; - } - - if (req.headers["x-stack-emulator-token"] !== token) { - sendText(res, 401, "Unauthorized"); - return; - } - - const requestBody = parseRequestBody(await readBody(req)); - if (requestBody === null) { - sendText(res, 400, "Invalid JSON body."); - return; - } - - if (req.url === "/read") { - await handleRead(res, requestBody); - return; - } - - if (req.url === "/write") { - await handleWrite(res, requestBody); - return; - } - - sendText(res, 404, "Not found"); - } catch (error) { - sendText(res, 500, error instanceof Error ? error.message : String(error)); - } -}); - -server.listen(port, host, () => { - console.log(`stack-qemu-file-bridge listening on ${host}:${port}`); -}); diff --git a/docker/local-emulator/qemu/prepare-app-bundle.sh b/docker/local-emulator/qemu/prepare-app-bundle.sh deleted file mode 100755 index 6f89115e2c..0000000000 --- a/docker/local-emulator/qemu/prepare-app-bundle.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" -exec "$SCRIPT_DIR/prepare-image-bundle.sh" "${1:-}" stack-local-emulator-app diff --git a/docker/local-emulator/qemu/prepare-image-bundle.sh b/docker/local-emulator/qemu/prepare-image-bundle.sh deleted file mode 100755 index daec7a106e..0000000000 --- a/docker/local-emulator/qemu/prepare-image-bundle.sh +++ /dev/null @@ -1,36 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -OUTPUT_PATH="${1:-}" -shift || true - -if [ -z "$OUTPUT_PATH" ] || [ "$#" -eq 0 ]; then - echo "Usage: $0 [docker-image...]" >&2 - exit 1 -fi - -if ! command -v docker >/dev/null 2>&1; then - echo "docker is required to package emulator images" >&2 - exit 1 -fi - -for IMAGE_NAME in "$@"; do - if ! docker image inspect "$IMAGE_NAME" >/dev/null 2>&1; then - cat >&2 < "$tmp_output" -mv "$tmp_output" "$OUTPUT_PATH" diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index 45a2d50bfe..6a14932f7b 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -7,13 +7,12 @@ source "$SCRIPT_DIR/common.sh" IMAGE_DIR="$SCRIPT_DIR/images" RUN_DIR="/tmp/stack-emulator-run" -FILE_BRIDGE_SCRIPT="$SCRIPT_DIR/host-file-bridge.mjs" VM_RAM="${EMULATOR_RAM:-4096}" VM_CPUS="${EMULATOR_CPUS:-4}" PORT_PREFIX="${PORT_PREFIX:-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}}" -FILE_BRIDGE_PORT="${EMULATOR_FILE_BRIDGE_PORT:-${PORT_PREFIX}16}" READY_TIMEOUT="${EMULATOR_READY_TIMEOUT:-240}" +CONFIG_FILE="" RED='\033[0;31m' GREEN='\033[0;32m' @@ -26,113 +25,6 @@ warn() { echo -e "${YELLOW}[emulator]${NC} $*"; } err() { echo -e "${RED}[emulator]${NC} $*" >&2; } info() { echo -e "${CYAN}[emulator]${NC} $*"; } -file_bridge_pidfile() { - echo "$RUN_DIR/host-file-bridge.pid" -} - -file_bridge_logfile() { - echo "$RUN_DIR/host-file-bridge.log" -} - -file_bridge_tokenfile() { - echo "$RUN_DIR/host-file-bridge.token" -} - -ensure_file_bridge_token() { - local token_file - token_file="$(file_bridge_tokenfile)" - # Deterministic token so snapshots can reuse the same value across restarts - FILE_BRIDGE_TOKEN="$(printf 'stack-local-emulator-%s' "$PORT_PREFIX" | shasum -a 256 | head -c 48)" - mkdir -p "$RUN_DIR" - printf "%s" "$FILE_BRIDGE_TOKEN" > "$token_file" -} - -is_file_bridge_running() { - local pidfile - pidfile="$(file_bridge_pidfile)" - if [ ! -f "$pidfile" ]; then - return 1 - fi - local pid - pid="$(cat "$pidfile")" - kill -0 "$pid" 2>/dev/null -} - -start_file_bridge() { - ensure_file_bridge_token - if is_file_bridge_running; then - return 0 - fi - - if [ ! -f "$FILE_BRIDGE_SCRIPT" ]; then - err "Missing host file bridge script: $FILE_BRIDGE_SCRIPT" - exit 1 - fi - - local pid - pid="$( - STACK_QEMU_FILE_BRIDGE_PORT="$FILE_BRIDGE_PORT" \ - STACK_QEMU_FILE_BRIDGE_HOST="0.0.0.0" \ - STACK_QEMU_FILE_BRIDGE_TOKEN="$FILE_BRIDGE_TOKEN" \ - python3 - "$FILE_BRIDGE_SCRIPT" "$(file_bridge_logfile)" <<'PY' -import os -import subprocess -import sys - -script_path = sys.argv[1] -log_path = sys.argv[2] - -with open(log_path, "ab", buffering=0) as log_file: - process = subprocess.Popen( - ["node", script_path], - stdin=subprocess.DEVNULL, - stdout=log_file, - stderr=log_file, - start_new_session=True, - env=os.environ.copy(), - close_fds=True, - ) - -print(process.pid) -PY - )" - echo "$pid" > "$(file_bridge_pidfile)" - - local elapsed=0 - while [ "$elapsed" -lt 15 ]; do - if curl -sf "http://127.0.0.1:${FILE_BRIDGE_PORT}/health" >/dev/null 2>&1; then - return 0 - fi - if ! kill -0 "$pid" 2>/dev/null; then - err "Host file bridge exited unexpectedly." - tail -40 "$(file_bridge_logfile)" 2>/dev/null || true - exit 1 - fi - sleep 1 - elapsed=$((elapsed + 1)) - done - - err "Timed out waiting for host file bridge on port ${FILE_BRIDGE_PORT}." - tail -40 "$(file_bridge_logfile)" 2>/dev/null || true - exit 1 -} - -stop_file_bridge() { - local pidfile - pidfile="$(file_bridge_pidfile)" - if [ ! -f "$pidfile" ]; then - return 0 - fi - - local pid - pid="$(cat "$pidfile")" - if kill -0 "$pid" 2>/dev/null; then - kill "$pid" 2>/dev/null || true - sleep 1 - kill -9 "$pid" 2>/dev/null || true - fi - rm -f "$pidfile" "$(file_bridge_logfile)" "$(file_bridge_tokenfile)" -} detect_host ARCH="${EMULATOR_ARCH:-$HOST_ARCH}" @@ -221,11 +113,13 @@ prepare_runtime_config_iso() { cfg_iso="$(runtime_iso_path)" rm -rf "$cfg_dir" mkdir -p "$cfg_dir" - cat > "$cfg_dir/runtime.env" < "$cfg_dir/runtime.env" + cp "$SCRIPT_DIR/../base.env" "$cfg_dir/base.env" make_iso_from_dir "$cfg_iso" "STACKCFG" "$cfg_dir" } @@ -332,7 +226,6 @@ build_qemu_cmd() { esac local netdev="user,id=net0" - netdev+=",hostfwd=tcp::${PORT_PREFIX}22-:22" # Deps services netdev+=",hostfwd=tcp::${PORT_PREFIX}28-:5432" netdev+=",hostfwd=tcp::${PORT_PREFIX}29-:2500" @@ -393,7 +286,7 @@ tail_vm_logs() { } ensure_ports_free() { - local ports=("${PORT_PREFIX}01" "${PORT_PREFIX}02" "${PORT_PREFIX}05" "${PORT_PREFIX}13" "${PORT_PREFIX}16" "${PORT_PREFIX}21" "${PORT_PREFIX}25" "${PORT_PREFIX}28" "${PORT_PREFIX}29" "${PORT_PREFIX}30" "${PORT_PREFIX}36" "${PORT_PREFIX}37") + local ports=("${PORT_PREFIX}01" "${PORT_PREFIX}02" "${PORT_PREFIX}05" "${PORT_PREFIX}13" "${PORT_PREFIX}21" "${PORT_PREFIX}25" "${PORT_PREFIX}28" "${PORT_PREFIX}29" "${PORT_PREFIX}30" "${PORT_PREFIX}36" "${PORT_PREFIX}37") local port for port in "${ports[@]}"; do if lsof -iTCP:"$port" -sTCP:LISTEN >/dev/null 2>&1; then @@ -435,9 +328,13 @@ stop_vm() { } cmd_start() { + if [ -n "$CONFIG_FILE" ] && [ ! -f "$CONFIG_FILE" ]; then + err "Config file not found: $CONFIG_FILE" + exit 1 + fi + ensure_ports_free mkdir -p "$RUN_DIR" - start_file_bridge IS_SNAPSHOT_RESTORE=false @@ -477,7 +374,6 @@ cmd_start() { cmd_stop() { stop_vm - stop_file_bridge log "QEMU emulator stopped." } @@ -513,7 +409,6 @@ cmd_status() { print_service_status "Backend" "${PORT_PREFIX}02" http "/health?db=1" print_service_status "PostgreSQL" "${PORT_PREFIX}28" tcp print_service_status "Inbucket HTTP" "${PORT_PREFIX}05" http / - print_service_status "Host File Bridge" "${FILE_BRIDGE_PORT}" http /health print_service_status "Svix" "${PORT_PREFIX}13" http /api/v1/health/ print_service_status "MinIO" "${PORT_PREFIX}21" http /minio/health/live print_service_status "QStash" "${PORT_PREFIX}25" http / 401 @@ -541,7 +436,24 @@ print(f"Startup time: {end_time - start_time:.1f}s") PY } -ACTION="${1:-start}" +ACTION="start" + +while [[ $# -gt 0 ]]; do + case "$1" in + --config-file) + CONFIG_FILE="$2" + shift 2 + ;; + start|stop|reset|status|bench) + ACTION="$1" + shift + ;; + *) + echo "Usage: $0 [start|stop|reset|status|bench] [--config-file ]" + exit 1 + ;; + esac +done case "$ACTION" in start) cmd_start ;; @@ -549,8 +461,4 @@ case "$ACTION" in reset) cmd_reset ;; status) cmd_status ;; bench) cmd_bench ;; - *) - echo "Usage: $0 [start|stop|reset|status|bench]" - exit 1 - ;; esac diff --git a/examples/demo/src/app/emulator-status/page.tsx b/examples/demo/src/app/emulator-status/page.tsx index 542bff846b..32b1a9d9ac 100644 --- a/examples/demo/src/app/emulator-status/page.tsx +++ b/examples/demo/src/app/emulator-status/page.tsx @@ -137,7 +137,7 @@ export default function EmulatorStatusPage() { {summary.down > 0 && ( {summary.down} down )} - {data != null && updated {new Date(data.timestamp).toLocaleTimeString()}} + updated {new Date(data.timestamp).toLocaleTimeString()}
) : null} From 4508f24d3c34d3b2c35e6d4b865b2321d8cac049 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Wed, 18 Mar 2026 18:09:36 -0700 Subject: [PATCH 04/30] Enhance Dockerfile for local emulator and server setup - Updated the local emulator Dockerfile to include a multi-stage build process, improving dependency management and reducing image size. - Added Node.js build stages and integrated pnpm for package management, enhancing build efficiency. - Implemented a migration-pruner stage to remove unnecessary node_modules, optimizing runtime performance. - Refined the server Dockerfile to utilize pruned node_modules for runtime, ensuring a leaner image. - Enhanced entrypoint and initialization scripts for better service startup processes. - Updated exposed ports to include backend and dashboard services, improving accessibility. These changes significantly improve the Docker setup for both local emulator and server, enhancing performance and maintainability. --- docker/local-emulator/Dockerfile | 6 +++--- docker/server/Dockerfile | 33 ++++++++++++++++++++++++++++++-- 2 files changed, 34 insertions(+), 5 deletions(-) diff --git a/docker/local-emulator/Dockerfile b/docker/local-emulator/Dockerfile index 6b7eaa7ebd..e389d06f87 100644 --- a/docker/local-emulator/Dockerfile +++ b/docker/local-emulator/Dockerfile @@ -35,7 +35,8 @@ RUN apt-get update && \ gosu \ procps \ libssl3 \ - && rm -rf /var/lib/apt/lists/* + && apt-get purge -y --auto-remove gnupg2 lsb-release \ + && rm -rf /var/lib/apt/lists/* /usr/share/doc /usr/share/man /usr/share/i18n # Inbucket COPY --from=inbucket-bin /opt/inbucket /opt/inbucket @@ -53,8 +54,7 @@ COPY --from=minio-bin /usr/bin/minio /usr/local/bin/minio COPY --from=mc-bin /usr/bin/mc /usr/local/bin/mc # QStash -COPY --from=qstash-bin /qstash-binary /usr/local/bin/qstash -RUN chmod +x /usr/local/bin/qstash +COPY --from=qstash-bin --chmod=755 /qstash-binary /usr/local/bin/qstash RUN mkdir -p \ /data/postgres \ diff --git a/docker/server/Dockerfile b/docker/server/Dockerfile index 56e9764452..7baf68a5e2 100644 --- a/docker/server/Dockerfile +++ b/docker/server/Dockerfile @@ -59,6 +59,35 @@ RUN cd apps/backend && pnpm build-self-host-migration-script +# Prune node_modules for runtime: remove dev tools, heavy UI packages, +# duplicate framework copies, and native binaries not needed by the +# migration script or server at runtime. +FROM builder AS migration-pruner +RUN cp -a /app/node_modules /pruned-node_modules && \ + cd /pruned-node_modules/.pnpm && \ + rm -rf \ + # Dev tools (never needed at runtime) + typescript@* eslint@* eslint-*@* @typescript-eslint+*@* \ + prettier@* vitest@* jsdom@* turbo@* turbo-*@* \ + tsdown@* @changesets+*@* codebuff@* \ + @testing-library+*@* vite@* vite-*@* @vitejs+*@* \ + # Heavy UI packages (already traced into Next.js standalone bundles) + monaco-editor@* \ + three@* three-globe@* globe.gl@* react-globe*@* \ + react-icons@* lucide-react@* @phosphor-icons+*@* \ + # Large optional packages not needed by migration script + posthog-js@* \ + @prisma+studio-core@* @prisma+dev@* @prisma+query-plan-executor@* \ + convex@* @electric-sql+*@* \ + # Duplicate Next.js copies (keep only one for next/headers.js resolution) + 'next@16.1.5_@babel+core@7.29.0*' 'next@16.1.5_@babel+core@7.28.5*' \ + next@14* @next+swc-*@14* \ + # Native build binaries not needed at runtime + @esbuild+*@* esbuild@* @rolldown+*@* \ + # Duplicate date-fns versions (keep v4 only) + date-fns@2* date-fns@3* + + # Final image FROM node:${NODE_VERSION}-slim @@ -82,8 +111,8 @@ COPY --from=builder --chown=node:node /app/apps/dashboard/.next/standalone ./ COPY --from=builder --chown=node:node /app/apps/dashboard/.next/static ./apps/dashboard/.next/static COPY --from=builder --chown=node:node /app/apps/dashboard/public ./apps/dashboard/public -# Restore workspace node_modules needed by non-Next runtime scripts (e.g. migrations) -COPY --from=builder --chown=node:node /app/node_modules ./node_modules +# Restore pruned workspace node_modules needed by non-Next runtime scripts (e.g. migrations) +COPY --from=migration-pruner --chown=node:node /pruned-node_modules ./node_modules COPY --from=builder --chown=node:node /app/packages ./packages # Add the entrypoint script From dec62a0c268116ac0e00e121f61c948590e50556 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Wed, 18 Mar 2026 18:10:21 -0700 Subject: [PATCH 05/30] Refactor QEMU emulator build and snapshot handling - Updated the `build-image.sh` script to simplify the Docker image list, focusing on the main emulator image. - Enhanced the image compression process by adding progress logging during the conversion to qcow2 format. - Improved the `run-emulator.sh` script to provide real-time feedback while saving snapshots, enhancing user experience. These changes streamline the build process and improve the usability of the emulator's snapshot functionality. --- docker/local-emulator/qemu/build-image.sh | 3 ++- docker/local-emulator/qemu/run-emulator.sh | 6 +++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docker/local-emulator/qemu/build-image.sh b/docker/local-emulator/qemu/build-image.sh index 77c714e22f..4ca2d3faa0 100755 --- a/docker/local-emulator/qemu/build-image.sh +++ b/docker/local-emulator/qemu/build-image.sh @@ -246,7 +246,8 @@ build_one() { cp "$serial_log" "$IMAGE_DIR/provision-emulator-${arch}.log" rm -rf "$tmp_dir" - qemu-img convert -O qcow2 -c "$final_img" "$final_img.tmp" + log "Compressing final image (this may take several minutes)..." + qemu-img convert -p -O qcow2 -c "$final_img" "$final_img.tmp" mv "$final_img.tmp" "$final_img" local size diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index 6a14932f7b..d7e182db64 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -87,10 +87,13 @@ save_snapshot() { local pid=$! local elapsed=0 + local snap_start=$SECONDS while [ "$elapsed" -lt 120 ]; do sleep 2 - elapsed=$((elapsed + 2)) + elapsed=$((SECONDS - snap_start)) + printf "\r [%3ds] saving snapshot..." "$elapsed" if [ -f "$out_file" ] && [ "$(grep -c '"return"' "$out_file" 2>/dev/null)" -ge 2 ]; then + echo "" kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null || true rm -f "$out_file" log "Snapshot saved." @@ -100,6 +103,7 @@ save_snapshot() { break fi done + echo "" kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null || true rm -f "$out_file" From 580ff10b9fda29c54749eefb142856ff289b27ce Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Wed, 18 Mar 2026 18:24:29 -0700 Subject: [PATCH 06/30] Refactor local emulator setup and consolidate services - Renamed and restructured the Docker Compose configuration to unify the local emulator services into a single `stack-local` service, simplifying the architecture. - Updated the Dockerfile to include a multi-stage build process for the local emulator, integrating the backend and dashboard into a single image. - Introduced a new `start-app.sh` script to manage service initialization and health checks, ensuring all dependencies are ready before starting the application. - Enhanced the cloud-init user-data script to streamline the setup process and improve service management. - Removed obsolete service definitions and scripts, reducing complexity and improving maintainability. These changes significantly enhance the local emulator's usability and performance for developers. --- docker/local-emulator/Dockerfile | 130 ++++++++++++++++-- docker/local-emulator/docker-compose.yaml | 59 +++----- docker/local-emulator/qemu/build-image.sh | 2 +- .../qemu/cloud-init/emulator/user-data | 72 ++++------ docker/local-emulator/start-app.sh | 20 +++ docker/local-emulator/supervisord.conf | 13 ++ docker/server/Dockerfile | 35 +---- 7 files changed, 204 insertions(+), 127 deletions(-) create mode 100644 docker/local-emulator/start-app.sh diff --git a/docker/local-emulator/Dockerfile b/docker/local-emulator/Dockerfile index e389d06f87..52eab95b68 100644 --- a/docker/local-emulator/Dockerfile +++ b/docker/local-emulator/Dockerfile @@ -1,5 +1,95 @@ -# Stack Auth Local Emulator — All-in-One Dependencies Image +# Stack Auth Local Emulator — All-in-One Image # Packages: PostgreSQL 16, Redis 7, Inbucket, Svix, ClickHouse, MinIO, QStash +# + built Stack Auth backend and dashboard + +ARG NODE_VERSION=22.21.1 + +# ── Node.js build stages ────────────────────────────────────────────────────── + +FROM node:${NODE_VERSION} AS node-base + +WORKDIR /app + +RUN apt-get update && \ + apt-get upgrade -y && \ + rm -rf /var/lib/apt/lists + +ENV PNPM_HOME=/pnpm +ENV PATH=$PNPM_HOME:$PATH + +RUN corepack enable +RUN corepack prepare pnpm@10.23.0 --activate +RUN pnpm add -g turbo +RUN pnpm add -g tsx + + +FROM node-base AS pruner + +COPY . . + +RUN tsx ./scripts/generate-sdks.ts + +# https://turbo.build/repo/docs/guides/tools/docker +RUN turbo prune --scope=@stackframe/backend --scope=@stackframe/dashboard --docker + + +FROM node-base AS builder + +# copy over package.json files and install dependencies +COPY --from=pruner /app/out/json/ . +COPY --from=pruner /app/out/pnpm-lock.yaml . +COPY .gitignore . +COPY pnpm-workspace.yaml . +COPY turbo.json . +COPY configs ./configs +RUN --mount=type=cache,id=pnpm,target=/pnpm/store STACK_SKIP_TEMPLATE_GENERATION=true pnpm install --frozen-lockfile + +# copy over the rest of the code for the build +COPY --from=pruner /app/out/full/ . + +# docs are currently required for the NextJS backend build, but won't exist in the final image +COPY docs ./docs + +# https://nextjs.org/docs/pages/api-reference/next-config-js/output +ENV NEXT_CONFIG_OUTPUT=standalone + +# Build the backend NextJS app +RUN pnpm turbo run docker-build --filter=@stackframe/backend... --filter=@stackframe/dashboard... + +# Build the self-host seed script +RUN cd apps/backend && pnpm build-self-host-migration-script + + +# Prune node_modules for runtime: remove dev tools, heavy UI packages, +# duplicate framework copies, and native binaries not needed by the +# migration script or server at runtime. +FROM builder AS migration-pruner +RUN cp -a /app/node_modules /pruned-node_modules && \ + cd /pruned-node_modules/.pnpm && \ + rm -rf \ + # Dev tools (never needed at runtime) + typescript@* eslint@* eslint-*@* @typescript-eslint+*@* \ + prettier@* vitest@* jsdom@* turbo@* turbo-*@* \ + tsdown@* @changesets+*@* codebuff@* \ + @testing-library+*@* vite@* vite-*@* @vitejs+*@* \ + # Heavy UI packages (already traced into Next.js standalone bundles) + monaco-editor@* \ + three@* three-globe@* globe.gl@* react-globe*@* \ + react-icons@* lucide-react@* @phosphor-icons+*@* \ + # Large optional packages not needed by migration script + posthog-js@* \ + @prisma+studio-core@* @prisma+dev@* @prisma+query-plan-executor@* \ + convex@* @electric-sql+*@* \ + # Duplicate Next.js copies (keep only one for next/headers.js resolution) + 'next@16.1.5_@babel+core@7.29.0*' 'next@16.1.5_@babel+core@7.28.5*' \ + next@14* @next+swc-*@14* \ + # Native build binaries not needed at runtime + @esbuild+*@* esbuild@* @rolldown+*@* \ + # Duplicate date-fns versions (keep v4 only) + date-fns@2* date-fns@3* + + +# ── Service binary stages ───────────────────────────────────────────────────── FROM inbucket/inbucket:3.1.0 AS inbucket-bin FROM svix/svix-server:latest AS svix-bin @@ -12,6 +102,9 @@ RUN cp $(which qstash) /qstash-binary 2>/dev/null || \ cp $(find / -name 'qstash' -type f -executable 2>/dev/null | head -1) /qstash-binary || \ { echo "ERROR: qstash binary not found" >&2; exit 1; } + +# ── Final image ─────────────────────────────────────────────────────────────── + FROM debian:trixie-slim ENV DEBIAN_FRONTEND=noninteractive @@ -35,9 +128,14 @@ RUN apt-get update && \ gosu \ procps \ libssl3 \ + openssl \ + socat \ && apt-get purge -y --auto-remove gnupg2 lsb-release \ && rm -rf /var/lib/apt/lists/* /usr/share/doc /usr/share/man /usr/share/i18n +# Node.js runtime (binary only — app bundles include all JS dependencies) +COPY --from=node-base /usr/local/bin/node /usr/local/bin/node + # Inbucket COPY --from=inbucket-bin /opt/inbucket /opt/inbucket @@ -56,6 +154,19 @@ COPY --from=mc-bin /usr/bin/mc /usr/local/bin/mc # QStash COPY --from=qstash-bin --chmod=755 /qstash-binary /usr/local/bin/qstash +# App +WORKDIR /app +COPY --from=builder /app/apps/backend/.next/standalone ./ +COPY --from=builder /app/apps/backend/.next/static ./apps/backend/.next/static +COPY --from=builder /app/apps/backend/prisma ./apps/backend/prisma +COPY --from=builder /app/apps/backend/dist ./apps/backend/dist +COPY --from=builder /app/apps/backend/node_modules ./apps/backend/node_modules +COPY --from=builder /app/apps/dashboard/.next/standalone ./ +COPY --from=builder /app/apps/dashboard/.next/static ./apps/dashboard/.next/static +COPY --from=builder /app/apps/dashboard/public ./apps/dashboard/public +COPY --from=migration-pruner /pruned-node_modules ./node_modules +COPY --from=builder /app/packages ./packages + RUN mkdir -p \ /data/postgres \ /data/redis \ @@ -71,15 +182,18 @@ RUN mkdir -p \ /etc/clickhouse-server \ && chown -R postgres:postgres /data/postgres -COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf -COPY entrypoint.sh /entrypoint.sh -COPY init-services.sh /init-services.sh -COPY clickhouse-config.xml /etc/clickhouse-server/config.xml -COPY clickhouse-users.xml /etc/clickhouse-server/users.xml -RUN chmod +x /entrypoint.sh /init-services.sh +COPY docker/local-emulator/supervisord.conf /etc/supervisor/conf.d/supervisord.conf +COPY docker/local-emulator/entrypoint.sh /entrypoint.sh +COPY docker/local-emulator/init-services.sh /init-services.sh +COPY docker/local-emulator/start-app.sh /start-app.sh +COPY docker/local-emulator/clickhouse-config.xml /etc/clickhouse-server/config.xml +COPY docker/local-emulator/clickhouse-users.xml /etc/clickhouse-server/users.xml +COPY docker/server/entrypoint.sh /app-entrypoint.sh +RUN chmod +x /entrypoint.sh /init-services.sh /start-app.sh /app-entrypoint.sh # PostgreSQL: 5432, Redis: 6379, Inbucket: 2500/9001/1100, # Svix: 8071, ClickHouse: 8123/9009, MinIO: 9090, QStash: 8080 -EXPOSE 5432 6379 2500 9001 1100 8071 8123 9009 9090 8080 +# Backend: 8102, Dashboard: 8101 +EXPOSE 5432 6379 2500 9001 1100 8071 8123 9009 9090 8080 8101 8102 ENTRYPOINT ["/entrypoint.sh"] diff --git a/docker/local-emulator/docker-compose.yaml b/docker/local-emulator/docker-compose.yaml index 116cc7df0c..82696d5d95 100644 --- a/docker/local-emulator/docker-compose.yaml +++ b/docker/local-emulator/docker-compose.yaml @@ -1,12 +1,12 @@ -# Stack Auth Local Emulator — all-in-one deps + app via docker compose. +# Stack Auth Local Emulator — all-in-one image via docker compose. # Ports follow the ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}XX convention. services: - stack-deps: + stack-local: build: - context: . - dockerfile: Dockerfile - image: stack-local-emulator-deps + context: ../.. + dockerfile: docker/local-emulator/Dockerfile + image: stack-local-emulator ports: - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28:5432" # PostgreSQL - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}29:2500" # Inbucket SMTP @@ -17,33 +17,14 @@ services: - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}25:8080" # QStash - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}36:8123" # ClickHouse HTTP - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}37:9009" # ClickHouse Native + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01:8101" # Dashboard + - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02:8102" # Backend volumes: - postgres-data:/data/postgres - redis-data:/data/redis - clickhouse-data:/data/clickhouse - minio-data:/data/minio - inbucket-data:/data/inbucket - extra_hosts: - - "host.docker.internal:host-gateway" - healthcheck: - test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -p 5432 -U postgres && curl -sf http://127.0.0.1:8123/ping"] - interval: 10s - timeout: 5s - retries: 10 - start_period: 30s - - stack-app: - build: - context: ../.. - dockerfile: docker/server/Dockerfile - image: stack-local-emulator-app - depends_on: - stack-deps: - condition: service_healthy - ports: - - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01:8101" # Dashboard - - "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02:8102" # Backend - volumes: - "${HOME}:${HOME}" - "/tmp:/tmp" env_file: ./base.env @@ -56,24 +37,24 @@ services: STACK_S3_PUBLIC_ENDPOINT: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}21/stack-storage" STACK_EMAIL_MONITOR_VERIFICATION_CALLBACK_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01/handler/email-verification" STACK_OAUTH_MOCK_URL: "http://host.docker.internal:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}14" - BACKEND_PORT: "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02" - DASHBOARD_PORT: "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01" - # Deps service hostnames (stack-deps container, not localhost) - STACK_DATABASE_CONNECTION_STRING: "postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@stack-deps:5432/stackframe" - STACK_EMAIL_HOST: "stack-deps" - STACK_SVIX_SERVER_URL: "http://stack-deps:8071" - STACK_S3_ENDPOINT: "http://stack-deps:9090" - STACK_QSTASH_URL: "http://stack-deps:8080" - STACK_CLICKHOUSE_URL: "http://stack-deps:8123" - STACK_EMAIL_MONITOR_INBUCKET_API_URL: "http://stack-deps:9001" + BACKEND_PORT: "8102" + DASHBOARD_PORT: "8101" + # Everything is in one container, so deps are at localhost + STACK_DATABASE_CONNECTION_STRING: "postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@127.0.0.1:5432/stackframe" + STACK_EMAIL_HOST: "127.0.0.1" + STACK_SVIX_SERVER_URL: "http://127.0.0.1:8071" + STACK_S3_ENDPOINT: "http://127.0.0.1:9090" + STACK_QSTASH_URL: "http://127.0.0.1:8080" + STACK_CLICKHOUSE_URL: "http://127.0.0.1:8123" + STACK_EMAIL_MONITOR_INBUCKET_API_URL: "http://127.0.0.1:9001" extra_hosts: - "host.docker.internal:host-gateway" healthcheck: - test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:8102/health?db=1 >/dev/null && curl -fsS http://127.0.0.1:8101/handler/sign-in >/dev/null"] + test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -p 5432 -U postgres && curl -sf http://127.0.0.1:8123/ping && curl -fsS 'http://127.0.0.1:8102/health?db=1' >/dev/null && curl -fsS http://127.0.0.1:8101/handler/sign-in >/dev/null"] interval: 10s timeout: 5s - retries: 24 - start_period: 60s + retries: 30 + start_period: 120s volumes: postgres-data: diff --git a/docker/local-emulator/qemu/build-image.sh b/docker/local-emulator/qemu/build-image.sh index 4ca2d3faa0..feca447fad 100755 --- a/docker/local-emulator/qemu/build-image.sh +++ b/docker/local-emulator/qemu/build-image.sh @@ -34,7 +34,7 @@ case "$TARGET_ARCH" in *) err "Usage: $0 [arm64|amd64|both]"; exit 1 ;; esac -DOCKER_IMAGES=(stack-local-emulator-deps stack-local-emulator-app) +DOCKER_IMAGES=(stack-local-emulator) check_deps() { local missing=() diff --git a/docker/local-emulator/qemu/cloud-init/emulator/user-data b/docker/local-emulator/qemu/cloud-init/emulator/user-data index a126eb2ee4..eaf09755e3 100644 --- a/docker/local-emulator/qemu/cloud-init/emulator/user-data +++ b/docker/local-emulator/qemu/cloud-init/emulator/user-data @@ -43,24 +43,6 @@ write_files: gzip -dc /mnt/stack-bundle/img.tgz | docker load - - path: /usr/local/bin/run-stack-deps-container - permissions: '0755' - content: | - #!/bin/bash - set -euo pipefail - - docker rm -f stack-deps >/dev/null 2>&1 || true - exec docker run \ - --rm \ - --name stack-deps \ - --network host \ - -v stack-postgres-data:/data/postgres \ - -v stack-redis-data:/data/redis \ - -v stack-clickhouse-data:/data/clickhouse \ - -v stack-minio-data:/data/minio \ - -v stack-inbucket-data:/data/inbucket \ - stack-local-emulator-deps - - path: /usr/local/bin/render-stack-env permissions: '0755' content: | @@ -76,7 +58,7 @@ write_files: source /mnt/stack-runtime/base.env set +a - # Deps runs in the same VM, so always localhost + # Everything runs in the same container, so always localhost DEPS_HOST=127.0.0.1 P="$STACK_EMULATOR_PORT_PREFIX" @@ -111,21 +93,26 @@ write_files: COMPUTED } > /run/stack-auth/local-emulator.env - - path: /usr/local/bin/run-stack-app-container + - path: /usr/local/bin/run-stack-container permissions: '0755' content: | #!/bin/bash set -euo pipefail /usr/local/bin/render-stack-env - docker rm -f stack-app >/dev/null 2>&1 || true + docker rm -f stack >/dev/null 2>&1 || true exec docker run \ --rm \ - --name stack-app \ + --name stack \ --network host \ --add-host host.docker.internal:host-gateway \ --env-file /run/stack-auth/local-emulator.env \ - stack-local-emulator-app + -v stack-postgres-data:/data/postgres \ + -v stack-redis-data:/data/redis \ + -v stack-clickhouse-data:/data/clickhouse \ + -v stack-minio-data:/data/minio \ + -v stack-inbucket-data:/data/inbucket \ + stack-local-emulator - path: /usr/local/bin/wait-for-deps permissions: '0755' @@ -139,10 +126,10 @@ write_files: until curl -sf http://127.0.0.1:9090/minio/health/live >/dev/null 2>&1; do sleep 1; done until [ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/ 2>/dev/null || true)" = "401" ]; do sleep 1; done - - path: /etc/systemd/system/stack-deps.service + - path: /etc/systemd/system/stack.service content: | [Unit] - Description=Stack Auth local emulator deps container + Description=Stack Auth local emulator Wants=network-online.target docker.service After=network-online.target docker.service @@ -150,25 +137,8 @@ write_files: Restart=always RestartSec=5 TimeoutStartSec=0 - ExecStart=/usr/local/bin/run-stack-deps-container - ExecStop=/usr/bin/docker stop stack-deps - - [Install] - WantedBy=multi-user.target - - - path: /etc/systemd/system/stack-app.service - content: | - [Unit] - Description=Stack Auth local emulator app container - Wants=network-online.target docker.service - After=network-online.target docker.service stack-deps.service - - [Service] - Restart=always - RestartSec=5 - TimeoutStartSec=0 - ExecStart=/usr/local/bin/run-stack-app-container - ExecStop=/usr/bin/docker stop stack-app + ExecStart=/usr/local/bin/run-stack-container + ExecStop=/usr/bin/docker stop stack [Install] WantedBy=multi-user.target @@ -178,10 +148,18 @@ runcmd: - systemctl mask ssh || true - bash /usr/local/bin/install-emulator-containers - systemctl daemon-reload - - systemctl enable stack-deps.service - - systemctl enable stack-app.service - - systemctl start stack-deps.service + - systemctl enable stack.service + - docker run --rm --name stack-build-init + --network host + -e STACK_DEPS_ONLY=true + -v stack-postgres-data:/data/postgres + -v stack-redis-data:/data/redis + -v stack-clickhouse-data:/data/clickhouse + -v stack-minio-data:/data/minio + -v stack-inbucket-data:/data/inbucket + -d stack-local-emulator - bash /usr/local/bin/wait-for-deps + - docker stop stack-build-init || true - echo "STACK_CLOUD_INIT_DONE" > /dev/console 2>/dev/null || true - echo "STACK_CLOUD_INIT_DONE" > /dev/ttyAMA0 2>/dev/null || true - echo "STACK_CLOUD_INIT_DONE" > /dev/ttyS0 2>/dev/null || true diff --git a/docker/local-emulator/start-app.sh b/docker/local-emulator/start-app.sh new file mode 100644 index 0000000000..4b2e610c31 --- /dev/null +++ b/docker/local-emulator/start-app.sh @@ -0,0 +1,20 @@ +#!/bin/bash +set -e + +# In deps-only mode (used during QEMU image build), skip app startup entirely. +# The build only needs the infrastructure services to initialize; the app +# requires runtime env vars that are not available at build time. +if [ "${STACK_DEPS_ONLY:-false}" = "true" ]; then + echo "Deps-only mode: app startup skipped." + while true; do sleep 3600; done +fi + +# Wait for all infrastructure services to be ready before running migrations +# and starting the backend/dashboard. +until pg_isready -h 127.0.0.1 -p 5432 -U postgres >/dev/null 2>&1; do sleep 2; done +until curl -sf http://127.0.0.1:8123/ping >/dev/null 2>&1; do sleep 2; done +until curl -sf http://127.0.0.1:8071/api/v1/health/ >/dev/null 2>&1; do sleep 2; done +until curl -sf http://127.0.0.1:9090/minio/health/live >/dev/null 2>&1; do sleep 2; done +until [ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/ 2>/dev/null || true)" = "401" ]; do sleep 2; done + +exec /app-entrypoint.sh diff --git a/docker/local-emulator/supervisord.conf b/docker/local-emulator/supervisord.conf index f3de1dcc14..e8b1fc4782 100644 --- a/docker/local-emulator/supervisord.conf +++ b/docker/local-emulator/supervisord.conf @@ -133,3 +133,16 @@ stdout_logfile=/dev/stdout stdout_logfile_maxbytes=0 stderr_logfile=/dev/stderr stderr_logfile_maxbytes=0 + +; --- Stack Auth backend + dashboard --- + +[program:stack-app] +command=/start-app.sh +autostart=true +autorestart=unexpected +startsecs=0 +priority=60 +stdout_logfile=/dev/stdout +stdout_logfile_maxbytes=0 +stderr_logfile=/dev/stderr +stderr_logfile_maxbytes=0 diff --git a/docker/server/Dockerfile b/docker/server/Dockerfile index 7baf68a5e2..8b5f8de19d 100644 --- a/docker/server/Dockerfile +++ b/docker/server/Dockerfile @@ -59,35 +59,6 @@ RUN cd apps/backend && pnpm build-self-host-migration-script -# Prune node_modules for runtime: remove dev tools, heavy UI packages, -# duplicate framework copies, and native binaries not needed by the -# migration script or server at runtime. -FROM builder AS migration-pruner -RUN cp -a /app/node_modules /pruned-node_modules && \ - cd /pruned-node_modules/.pnpm && \ - rm -rf \ - # Dev tools (never needed at runtime) - typescript@* eslint@* eslint-*@* @typescript-eslint+*@* \ - prettier@* vitest@* jsdom@* turbo@* turbo-*@* \ - tsdown@* @changesets+*@* codebuff@* \ - @testing-library+*@* vite@* vite-*@* @vitejs+*@* \ - # Heavy UI packages (already traced into Next.js standalone bundles) - monaco-editor@* \ - three@* three-globe@* globe.gl@* react-globe*@* \ - react-icons@* lucide-react@* @phosphor-icons+*@* \ - # Large optional packages not needed by migration script - posthog-js@* \ - @prisma+studio-core@* @prisma+dev@* @prisma+query-plan-executor@* \ - convex@* @electric-sql+*@* \ - # Duplicate Next.js copies (keep only one for next/headers.js resolution) - 'next@16.1.5_@babel+core@7.29.0*' 'next@16.1.5_@babel+core@7.28.5*' \ - next@14* @next+swc-*@14* \ - # Native build binaries not needed at runtime - @esbuild+*@* esbuild@* @rolldown+*@* \ - # Duplicate date-fns versions (keep v4 only) - date-fns@2* date-fns@3* - - # Final image FROM node:${NODE_VERSION}-slim @@ -96,7 +67,7 @@ WORKDIR /app # Install packages needed for deployment RUN apt-get update && \ apt-get upgrade -y && \ - apt-get install -y curl openssl socat && \ + apt-get install -y openssl socat && \ rm -rf /var/lib/apt/lists # Copy built backend @@ -111,8 +82,8 @@ COPY --from=builder --chown=node:node /app/apps/dashboard/.next/standalone ./ COPY --from=builder --chown=node:node /app/apps/dashboard/.next/static ./apps/dashboard/.next/static COPY --from=builder --chown=node:node /app/apps/dashboard/public ./apps/dashboard/public -# Restore pruned workspace node_modules needed by non-Next runtime scripts (e.g. migrations) -COPY --from=migration-pruner --chown=node:node /pruned-node_modules ./node_modules +# Restore workspace node_modules needed by non-Next runtime scripts (e.g. migrations) +COPY --from=builder --chown=node:node /app/node_modules ./node_modules COPY --from=builder --chown=node:node /app/packages ./packages # Add the entrypoint script From e3592bb9f59d205f5c9b0585f47416e4e819e6b1 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Wed, 18 Mar 2026 18:37:16 -0700 Subject: [PATCH 07/30] Add base image fingerprinting for QEMU snapshots - Introduced a new function to generate a fingerprint of the base QEMU image, allowing for detection of changes since the last snapshot. - Updated snapshot handling logic to discard stale snapshots if the base image has changed, improving reliability and efficiency. - Enhanced the `run-emulator.sh` script to save the fingerprint after a snapshot is created, ensuring accurate tracking of image state. These changes enhance the snapshot management process in the QEMU emulator, providing better consistency and performance for developers. --- docker/local-emulator/qemu/run-emulator.sh | 25 ++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index d7e182db64..af1c9f6ea0 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -63,6 +63,17 @@ runtime_iso_path() { SNAPSHOT_NAME="ready" IS_SNAPSHOT_RESTORE=false +# Returns a fast fingerprint (size:mtime) of the base QEMU image. +# Used to detect whether the image has changed since the snapshot was saved. +base_image_fingerprint() { + local img="$1" + case "$HOST_OS" in + darwin) stat -f "%z:%m" "$img" 2>/dev/null ;; + linux) stat -c "%s:%Y" "$img" 2>/dev/null ;; + *) stat -f "%z:%m" "$img" 2>/dev/null || stat -c "%s:%Y" "$img" 2>/dev/null ;; + esac +} + vm_has_snapshot() { [ -f "$VM_DIR/disk.qcow2" ] && qemu-img snapshot -l "$VM_DIR/disk.qcow2" 2>/dev/null | grep -q "$SNAPSHOT_NAME" @@ -96,6 +107,7 @@ save_snapshot() { echo "" kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null || true rm -f "$out_file" + base_image_fingerprint "$(image_path)" > "$VM_DIR/base-image.fingerprint" log "Snapshot saved." return 0 fi @@ -202,9 +214,18 @@ build_qemu_cmd() { mkdir -p "$VM_DIR" local has_snapshot=false + local fingerprint_file="$VM_DIR/base-image.fingerprint" + local current_fp + current_fp="$(base_image_fingerprint "$base_img")" if [ -f "$VM_DIR/disk.qcow2" ] && vm_has_snapshot; then - has_snapshot=true - else + if [ -f "$fingerprint_file" ] && [ "$(cat "$fingerprint_file")" = "$current_fp" ]; then + has_snapshot=true + else + warn "QEMU base image has changed — discarding stale snapshot." + rm -f "$VM_DIR/disk.qcow2" "$fingerprint_file" + fi + fi + if [ "$has_snapshot" = "false" ]; then rm -f "$VM_DIR/disk.qcow2" qemu-img create -f qcow2 -b "$base_img" -F qcow2 "$VM_DIR/disk.qcow2" >/dev/null fi From 5a729ec89d4627d8f1e706755e1b22e3121380d0 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Wed, 18 Mar 2026 19:40:27 -0700 Subject: [PATCH 08/30] Enhance configuration and permissions in local emulator setup - Updated `stack.config.ts` to enable various application features including authentication, emails, teams, RBAC, API keys, webhooks, and launch checklist. - Modified `seed.ts` to grant team permissions for the local emulator admin user, improving access control. - Adjusted `run-emulator.sh` to allow auto-detection of the configuration file, enhancing usability. - Updated `.gitignore` to include the new `run/` directory, ensuring cleaner repository management. These changes improve the local emulator's configuration management and user experience for developers. --- apps/backend/prisma/seed.ts | 7 +++++++ docker/local-emulator/qemu/.gitignore | 1 + docker/local-emulator/qemu/run-emulator.sh | 2 +- stack.config.ts | 10 +++++++++- 4 files changed, 18 insertions(+), 2 deletions(-) diff --git a/apps/backend/prisma/seed.ts b/apps/backend/prisma/seed.ts index fd5573cc2f..5f1517ae8f 100644 --- a/apps/backend/prisma/seed.ts +++ b/apps/backend/prisma/seed.ts @@ -496,6 +496,13 @@ export async function seed() { } else { console.log('Ensured emulator user is a member of emulator team'); } + + await grantTeamPermission(internalPrisma, { + tenancy: internalTenancy, + teamId: LOCAL_EMULATOR_OWNER_TEAM_ID, + userId: LOCAL_EMULATOR_ADMIN_USER_ID, + permissionId: "team_admin", + }); } console.log('Seeding complete!'); diff --git a/docker/local-emulator/qemu/.gitignore b/docker/local-emulator/qemu/.gitignore index e43b0f9889..6f1a609192 100644 --- a/docker/local-emulator/qemu/.gitignore +++ b/docker/local-emulator/qemu/.gitignore @@ -1 +1,2 @@ .DS_Store +run/ diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index af1c9f6ea0..6c4d2ad3f3 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -6,7 +6,7 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" source "$SCRIPT_DIR/common.sh" IMAGE_DIR="$SCRIPT_DIR/images" -RUN_DIR="/tmp/stack-emulator-run" +RUN_DIR="${EMULATOR_RUN_DIR:-$SCRIPT_DIR/run}" VM_RAM="${EMULATOR_RAM:-4096}" VM_CPUS="${EMULATOR_CPUS:-4}" diff --git a/stack.config.ts b/stack.config.ts index c4470d209f..e2f9c498d4 100644 --- a/stack.config.ts +++ b/stack.config.ts @@ -1 +1,9 @@ -export const config = {}; +export const config = { + "apps.installed.authentication.enabled": true, + "apps.installed.emails.enabled": true, + "apps.installed.teams.enabled": true, + "apps.installed.rbac.enabled": true, + "apps.installed.api-keys.enabled": true, + "apps.installed.webhooks.enabled": true, + "apps.installed.launch-checklist.enabled": true, +}; From c1a15ff4706fed145896a6abf6df9b210495c64e Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Wed, 18 Mar 2026 21:11:04 -0700 Subject: [PATCH 09/30] Enhance local emulator configuration and file management - Updated `stack.config.ts` to enable password sign-in for authentication. - Refactored `config.tsx` to include functionality for writing configuration changes back to the local emulator's config file. - Introduced `writeConfigToFile` and `getLocalEmulatorFilePath` functions in `local-emulator.ts` to improve file handling and path resolution. - Enhanced `run-emulator.sh` to support mounting the host filesystem, facilitating better integration with the local environment. These changes improve the local emulator's configuration management and usability for developers. --- apps/backend/src/lib/config.tsx | 10 +++- apps/backend/src/lib/local-emulator.ts | 47 +++++++++++++++---- .../qemu/cloud-init/emulator/user-data | 12 +++++ docker/local-emulator/qemu/run-emulator.sh | 16 +------ stack.config.ts | 1 + 5 files changed, 63 insertions(+), 23 deletions(-) diff --git a/apps/backend/src/lib/config.tsx b/apps/backend/src/lib/config.tsx index c1c90f410d..c73c09a1af 100644 --- a/apps/backend/src/lib/config.tsx +++ b/apps/backend/src/lib/config.tsx @@ -11,7 +11,7 @@ import { Result } from "@stackframe/stack-shared/dist/utils/results"; import { deindent, stringCompare } from "@stackframe/stack-shared/dist/utils/strings"; import * as yup from "yup"; import { RawQuery, globalPrismaClient, rawQuery } from "../prisma-client"; -import { LOCAL_EMULATOR_ENV_CONFIG_BLOCKED_MESSAGE, isLocalEmulatorEnabled, isLocalEmulatorProject, readConfigFromFile } from "./local-emulator"; +import { LOCAL_EMULATOR_ENV_CONFIG_BLOCKED_MESSAGE, getLocalEmulatorFilePath, isLocalEmulatorEnabled, isLocalEmulatorProject, readConfigFromFile, writeConfigToFile } from "./local-emulator"; import { listPermissionDefinitionsFromConfig } from "./permissions"; type BranchConfigSourceApi = yup.InferType; @@ -295,6 +295,14 @@ export async function setBranchConfigOverride(options: { config: newConfig, }, }); + + // In the local emulator, write config changes back to the config file + if (isLocalEmulatorEnabled()) { + const filePath = await getLocalEmulatorFilePath(options.projectId); + if (filePath) { + await writeConfigToFile(filePath, newConfig); + } + } } /** diff --git a/apps/backend/src/lib/local-emulator.ts b/apps/backend/src/lib/local-emulator.ts index 690a5a493d..08e45baf22 100644 --- a/apps/backend/src/lib/local-emulator.ts +++ b/apps/backend/src/lib/local-emulator.ts @@ -36,24 +36,55 @@ export async function isLocalEmulatorProject(projectId: string) { return project !== null; } +/** + * Resolves the file path for config files in the local emulator. + * + * In the QEMU emulator, the host filesystem is mounted at /host via virtio-9p. + * The DB stores absolute host paths (e.g. /Users/foo/project/stack.config.ts), so we + * try /host/ first, then fall back to the original path for non-QEMU environments + * (e.g. Docker Compose where the path is directly accessible). + */ +async function resolveConfigFilePath(filePath: string): Promise { + const hostMountedPath = path.join("/host", filePath); + try { + await fs.access(hostMountedPath); + return hostMountedPath; + } catch { + return filePath; + } +} + export async function readConfigFromFile(filePath: string): Promise> { - const configContentBase64 = getEnvVariable("STACK_LOCAL_EMULATOR_CONFIG_CONTENT", ""); - const content = configContentBase64 !== "" - ? Buffer.from(configContentBase64, "base64").toString("utf-8") - : await fs.readFile(filePath, "utf-8").catch((error: unknown) => { - if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; - throw error; - }); + const resolvedPath = await resolveConfigFilePath(filePath); + const content = await fs.readFile(resolvedPath, "utf-8").catch((error: unknown) => { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; + throw error; + }); if (content === null || content.trim() === "") { return {}; } const jiti = createJiti(import.meta.url, { cache: false }); - const mod = jiti.evalModule(content, { filename: filePath }) as Record; + const mod = jiti.evalModule(content, { filename: resolvedPath }) as Record; const config = mod.config; if (!isValidConfig(config)) { throw new StatusError(StatusError.BadRequest, `Invalid config in ${filePath}. The file must export a 'config' object.`); } return config; } + +export async function writeConfigToFile(filePath: string, config: Record): Promise { + const resolvedPath = await resolveConfigFilePath(filePath); + const configString = JSON.stringify(config, null, 2); + const content = `export const config = ${configString};\n`; + await fs.writeFile(resolvedPath, content, "utf-8"); +} + +export async function getLocalEmulatorFilePath(projectId: string): Promise { + const project = await globalPrismaClient.localEmulatorProject.findUnique({ + where: { projectId }, + select: { absoluteFilePath: true }, + }); + return project?.absoluteFilePath ?? null; +} diff --git a/docker/local-emulator/qemu/cloud-init/emulator/user-data b/docker/local-emulator/qemu/cloud-init/emulator/user-data index eaf09755e3..953173101c 100644 --- a/docker/local-emulator/qemu/cloud-init/emulator/user-data +++ b/docker/local-emulator/qemu/cloud-init/emulator/user-data @@ -93,12 +93,23 @@ write_files: COMPUTED } > /run/stack-auth/local-emulator.env + - path: /usr/local/bin/mount-host-fs + permissions: '0755' + content: | + #!/bin/bash + set -euo pipefail + mkdir -p /host + if ! mountpoint -q /host; then + mount -t 9p -o trans=virtio,version=9p2000.L hostfs /host 2>/dev/null || true + fi + - path: /usr/local/bin/run-stack-container permissions: '0755' content: | #!/bin/bash set -euo pipefail + /usr/local/bin/mount-host-fs /usr/local/bin/render-stack-env docker rm -f stack >/dev/null 2>&1 || true exec docker run \ @@ -112,6 +123,7 @@ write_files: -v stack-clickhouse-data:/data/clickhouse \ -v stack-minio-data:/data/minio \ -v stack-inbucket-data:/data/inbucket \ + -v /host:/host \ stack-local-emulator - path: /usr/local/bin/wait-for-deps diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index 6c4d2ad3f3..ce5e8b6dd4 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -12,7 +12,6 @@ VM_RAM="${EMULATOR_RAM:-4096}" VM_CPUS="${EMULATOR_CPUS:-4}" PORT_PREFIX="${PORT_PREFIX:-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}}" READY_TIMEOUT="${EMULATOR_READY_TIMEOUT:-240}" -CONFIG_FILE="" RED='\033[0;31m' GREEN='\033[0;32m' @@ -131,9 +130,6 @@ prepare_runtime_config_iso() { mkdir -p "$cfg_dir" { printf "STACK_EMULATOR_PORT_PREFIX=%s\n" "$PORT_PREFIX" - if [ -n "$CONFIG_FILE" ]; then - printf "STACK_LOCAL_EMULATOR_CONFIG_CONTENT=%s\n" "$(base64 < "$CONFIG_FILE")" - fi } > "$cfg_dir/runtime.env" cp "$SCRIPT_DIR/../base.env" "$cfg_dir/base.env" make_iso_from_dir "$cfg_iso" "STACKCFG" "$cfg_dir" @@ -279,6 +275,7 @@ build_qemu_cmd() { -netdev "$netdev" -device virtio-net-pci,netdev=net0 -device virtio-balloon-pci + -virtfs "local,path=/,mount_tag=hostfs,security_model=none" -chardev "socket,id=monitor,path=$VM_DIR/monitor.sock,server=on,wait=off" -mon "chardev=monitor,mode=control" -serial "file:$VM_DIR/serial.log" @@ -353,11 +350,6 @@ stop_vm() { } cmd_start() { - if [ -n "$CONFIG_FILE" ] && [ ! -f "$CONFIG_FILE" ]; then - err "Config file not found: $CONFIG_FILE" - exit 1 - fi - ensure_ports_free mkdir -p "$RUN_DIR" @@ -465,16 +457,12 @@ ACTION="start" while [[ $# -gt 0 ]]; do case "$1" in - --config-file) - CONFIG_FILE="$2" - shift 2 - ;; start|stop|reset|status|bench) ACTION="$1" shift ;; *) - echo "Usage: $0 [start|stop|reset|status|bench] [--config-file ]" + echo "Usage: $0 [start|stop|reset|status|bench]" exit 1 ;; esac diff --git a/stack.config.ts b/stack.config.ts index e2f9c498d4..e5d702ec5a 100644 --- a/stack.config.ts +++ b/stack.config.ts @@ -6,4 +6,5 @@ export const config = { "apps.installed.api-keys.enabled": true, "apps.installed.webhooks.enabled": true, "apps.installed.launch-checklist.enabled": true, + "auth.password.allowSignIn": true }; From 0496cd2c4d753bd63388b80942fcd366f2f9a078 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 11:29:22 -0700 Subject: [PATCH 10/30] Add QEMU emulator build workflow and emulator command integration - Introduced a new GitHub Actions workflow for building and publishing QEMU emulator images, triggered on pushes to main and dev branches, as well as pull requests. - Added a new `emulator` command in the Stack CLI to manage QEMU emulator operations, including pulling images, starting, stopping, and checking status. - Implemented functionality for downloading emulator images from GitHub Releases and managing snapshots, enhancing the local development experience. These changes significantly improve the automation and usability of the QEMU emulator for developers. --- .github/workflows/qemu-emulator-build.yaml | 189 +++++++++++++++++ packages/stack-cli/src/commands/emulator.ts | 215 ++++++++++++++++++++ packages/stack-cli/src/index.ts | 2 + 3 files changed, 406 insertions(+) create mode 100644 .github/workflows/qemu-emulator-build.yaml create mode 100644 packages/stack-cli/src/commands/emulator.ts diff --git a/.github/workflows/qemu-emulator-build.yaml b/.github/workflows/qemu-emulator-build.yaml new file mode 100644 index 0000000000..a268dba824 --- /dev/null +++ b/.github/workflows/qemu-emulator-build.yaml @@ -0,0 +1,189 @@ +name: Build & Publish QEMU Emulator Images + +on: + push: + branches: + - main + - dev + pull_request: + paths: + - 'docker/local-emulator/**' + - 'packages/stack-cli/src/commands/emulator.ts' + - '.github/workflows/qemu-emulator-build.yaml' + workflow_dispatch: + inputs: + publish: + description: 'Publish images to GitHub Releases' + type: boolean + default: false + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ github.ref != 'refs/heads/main' && github.ref != 'refs/heads/dev' }} + +env: + EMULATOR_IMAGE_NAME: stack-local-emulator + +jobs: + build: + name: Build QEMU Image (${{ matrix.arch }}) + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - arch: amd64 + runner: ubicloud-standard-8 + docker_platform: linux/amd64 + - arch: arm64 + runner: ubicloud-standard-8-arm + docker_platform: linux/arm64 + + steps: + - uses: actions/checkout@v6 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Install QEMU dependencies + run: | + sudo apt-get update + sudo apt-get install -y qemu-system-x86 qemu-system-arm qemu-utils genisoimage socat qemu-efi-aarch64 + + - name: Build Docker emulator image + run: | + docker buildx build \ + --platform ${{ matrix.docker_platform }} \ + --tag ${{ env.EMULATOR_IMAGE_NAME }} \ + --load \ + -f docker/local-emulator/Dockerfile \ + . + + - name: Build QEMU image + run: | + chmod +x docker/local-emulator/qemu/build-image.sh + docker/local-emulator/qemu/build-image.sh ${{ matrix.arch }} + + - name: Start emulator (creates runtime state) + run: | + chmod +x docker/local-emulator/qemu/run-emulator.sh + EMULATOR_ARCH=${{ matrix.arch }} \ + EMULATOR_READY_TIMEOUT=300 \ + docker/local-emulator/qemu/run-emulator.sh start + + - name: Verify services are healthy + run: | + EMULATOR_ARCH=${{ matrix.arch }} \ + docker/local-emulator/qemu/run-emulator.sh status + + - name: Stop emulator (saves snapshot) + run: | + EMULATOR_ARCH=${{ matrix.arch }} \ + docker/local-emulator/qemu/run-emulator.sh stop + + - name: Package snapshot image + run: | + SNAPSHOT_IMG="docker/local-emulator/qemu/run/vm/disk.qcow2" + BASE_IMG="docker/local-emulator/qemu/images/stack-emulator-${{ matrix.arch }}.qcow2" + + # Compress snapshot image (contains VM state for instant restore) + qemu-img convert -p -O qcow2 -c "$SNAPSHOT_IMG" \ + "stack-emulator-${{ matrix.arch }}-snapshot.qcow2" + + # Copy base image (no snapshot, clean boot) + cp "$BASE_IMG" "stack-emulator-${{ matrix.arch }}-base.qcow2" + + - name: Upload snapshot artifact + uses: actions/upload-artifact@v4 + with: + name: qemu-emulator-${{ matrix.arch }}-snapshot + path: stack-emulator-${{ matrix.arch }}-snapshot.qcow2 + retention-days: 30 + compression-level: 0 + + - name: Upload base artifact + uses: actions/upload-artifact@v4 + with: + name: qemu-emulator-${{ matrix.arch }}-base + path: stack-emulator-${{ matrix.arch }}-base.qcow2 + retention-days: 30 + compression-level: 0 + + publish: + name: Publish to GitHub Releases + needs: build + if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev' || (github.event_name == 'workflow_dispatch' && inputs.publish) + runs-on: ubuntu-latest + permissions: + contents: write + + steps: + - uses: actions/checkout@v6 + + - name: Download all artifacts + uses: actions/download-artifact@v4 + with: + path: artifacts + + - name: Prepare release assets + run: | + mkdir -p release + SHORT_SHA="${GITHUB_SHA:0:8}" + BRANCH="${GITHUB_REF_NAME}" + DATE="$(date -u +%Y%m%d)" + TAG="emulator-${BRANCH}-${DATE}-${SHORT_SHA}" + echo "RELEASE_TAG=${TAG}" >> "$GITHUB_ENV" + echo "SHORT_SHA=${SHORT_SHA}" >> "$GITHUB_ENV" + + for f in artifacts/qemu-emulator-*/*.qcow2; do + cp "$f" release/ + done + + ls -lh release/ + + - name: Create GitHub Release + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + gh release create "${{ env.RELEASE_TAG }}" \ + --title "QEMU Emulator — ${{ github.ref_name }} (${{ env.SHORT_SHA }})" \ + --notes "$(cat <<'EOF' + ## QEMU Emulator Images + + Built from \`${{ github.ref_name }}\` @ \`${{ github.sha }}\` + + ### Images + | File | Description | + |------|-------------| + | `stack-emulator-arm64-snapshot.qcow2` | ARM64 with pre-warmed snapshot (instant boot) | + | `stack-emulator-arm64-base.qcow2` | ARM64 base image (clean boot) | + | `stack-emulator-amd64-snapshot.qcow2` | AMD64 with pre-warmed snapshot (instant boot) | + | `stack-emulator-amd64-base.qcow2` | AMD64 base image (clean boot) | + + ### Usage + ```bash + # Pull and run the latest emulator via Stack CLI + stack emulator pull --snapshot + stack emulator run + ``` + EOF + )" \ + --prerelease \ + release/* + + - name: Update latest tag for branch + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + LATEST_TAG="emulator-${{ github.ref_name }}-latest" + # Delete existing latest release if it exists + gh release delete "$LATEST_TAG" --yes 2>/dev/null || true + git tag -d "$LATEST_TAG" 2>/dev/null || true + git push origin ":refs/tags/$LATEST_TAG" 2>/dev/null || true + + gh release create "$LATEST_TAG" \ + --title "QEMU Emulator — ${{ github.ref_name }} (latest)" \ + --notes "Latest emulator images from \`${{ github.ref_name }}\`. Auto-updated on each build." \ + --prerelease \ + release/* diff --git a/packages/stack-cli/src/commands/emulator.ts b/packages/stack-cli/src/commands/emulator.ts new file mode 100644 index 0000000000..c195c7b360 --- /dev/null +++ b/packages/stack-cli/src/commands/emulator.ts @@ -0,0 +1,215 @@ +import { Command } from "commander"; +import { execSync, spawn } from "child_process"; +import { existsSync, mkdirSync, renameSync, unlinkSync, createWriteStream } from "fs"; +import { join, resolve } from "path"; +import { CliError } from "../lib/errors.js"; + +const DEFAULT_REPO = "stack-auth/stack-auth"; +const DEFAULT_BRANCH = "dev"; + +function detectArch(): string { + const arch = process.arch; + if (arch === "arm64") { + return "arm64"; + } else if (arch === "x64") { + return "amd64"; + } else { + throw new CliError(`Unsupported architecture: ${arch}`); + } +} + +function findQemuDir(): string { + // Walk up from this file to find the repo root, then locate qemu dir + // When running from the repo, it's relative to the monorepo root + // Try common locations + const candidates = [ + resolve(process.cwd(), "docker/local-emulator/qemu"), + resolve(process.cwd(), "../docker/local-emulator/qemu"), + ]; + + for (const candidate of candidates) { + if (existsSync(join(candidate, "run-emulator.sh"))) { + return candidate; + } + } + + throw new CliError( + "Could not find QEMU emulator directory. Run this from the stack-auth repo root." + ); +} + +function runScript(qemuDir: string, script: string, args: string[], env?: Record) { + const scriptPath = join(qemuDir, script); + const result = spawn(scriptPath, args, { + stdio: "inherit", + env: { ...process.env, ...env }, + cwd: qemuDir, + }); + + return new Promise((resolve, reject) => { + result.on("close", (code) => { + if (code === 0) { + resolve(); + } else { + reject(new CliError(`${script} exited with code ${code}`)); + } + }); + result.on("error", (err) => { + reject(new CliError(`Failed to run ${script}: ${err.message}`)); + }); + }); +} + +function ghRelease(args: string[]): string { + try { + return execSync(`gh ${args.join(" ")}`, { encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"] }).trim(); + } catch (err: unknown) { + if (err instanceof Error && "stderr" in err) { + throw new CliError(`GitHub CLI error: ${(err as { stderr: string }).stderr}`); + } + throw new CliError("GitHub CLI (gh) is required. Install: https://cli.github.com/"); + } +} + +export function registerEmulatorCommand(program: Command) { + const emulator = program + .command("emulator") + .description("Manage the QEMU local emulator"); + + emulator + .command("pull") + .description("Download the latest emulator image from GitHub Releases") + .option("--snapshot", "Download snapshot image (instant boot)", true) + .option("--base", "Download base image (clean boot)") + .option("--arch ", "Target architecture (arm64 or amd64)") + .option("--branch ", `Release branch (default: ${DEFAULT_BRANCH})`) + .option("--tag ", "Specific release tag") + .option("--repo ", `GitHub repository (default: ${DEFAULT_REPO})`) + .action(async (opts) => { + const arch = opts.arch || detectArch(); + const repo = opts.repo || DEFAULT_REPO; + const branch = opts.branch || DEFAULT_BRANCH; + const variant = opts.base ? "base" : "snapshot"; + const tag = opts.tag || `emulator-${branch}-latest`; + const asset = `stack-emulator-${arch}-${variant}.qcow2`; + + const qemuDir = findQemuDir(); + const imageDir = join(qemuDir, "images"); + mkdirSync(imageDir, { recursive: true }); + + const dest = join(imageDir, `stack-emulator-${arch}.qcow2`); + const tmpDest = `${dest}.download`; + + console.log(`Pulling ${variant} image for ${arch} from release ${tag}...`); + + try { + execSync( + `gh release download ${JSON.stringify(tag)} --repo ${JSON.stringify(repo)} --pattern ${JSON.stringify(asset)} --output ${JSON.stringify(tmpDest)} --clobber`, + { stdio: "inherit" } + ); + } catch { + if (existsSync(tmpDest)) unlinkSync(tmpDest); + throw new CliError( + `Failed to download ${asset} from release ${tag}.\nRun 'stack emulator list-releases' to see available releases.` + ); + } + + renameSync(tmpDest, dest); + console.log(`Downloaded: ${dest}`); + + if (variant === "snapshot") { + const runDir = join(qemuDir, "run", "vm"); + mkdirSync(runDir, { recursive: true }); + execSync(`cp ${JSON.stringify(dest)} ${JSON.stringify(join(runDir, "disk.qcow2"))}`); + console.log("Snapshot image installed — next start will restore instantly."); + } + }); + + emulator + .command("run") + .description("Start the emulator (auto-pulls if no image exists)") + .option("--arch ", "Target architecture") + .action(async (opts) => { + const arch = opts.arch || detectArch(); + const qemuDir = findQemuDir(); + const img = join(qemuDir, "images", `stack-emulator-${arch}.qcow2`); + + if (!existsSync(img)) { + console.log("No emulator image found. Pulling latest snapshot..."); + // Re-invoke pull via the same program + await program.parseAsync(["node", "stack", "emulator", "pull", "--arch", arch], { from: "user" }); + } + + await runScript(qemuDir, "run-emulator.sh", ["start"], { EMULATOR_ARCH: arch }); + }); + + emulator + .command("stop") + .description("Stop the emulator (saves snapshot)") + .option("--no-snapshot", "Skip saving snapshot before stopping") + .action(async (opts) => { + const qemuDir = findQemuDir(); + const env: Record = {}; + if (opts.snapshot === false) { + env.SKIP_SNAPSHOT = "true"; + } + await runScript(qemuDir, "run-emulator.sh", ["stop"], env); + }); + + emulator + .command("reset") + .description("Reset emulator state for a fresh boot") + .action(async () => { + const qemuDir = findQemuDir(); + await runScript(qemuDir, "run-emulator.sh", ["reset"]); + }); + + emulator + .command("status") + .description("Show emulator and service health") + .action(async () => { + const qemuDir = findQemuDir(); + await runScript(qemuDir, "run-emulator.sh", ["status"]); + }); + + emulator + .command("snapshot") + .description("Save a snapshot of the running emulator") + .action(async () => { + const qemuDir = findQemuDir(); + await runScript(qemuDir, "run-emulator.sh", ["snapshot"]); + }); + + emulator + .command("build") + .description("Build the QEMU emulator image locally") + .option("--arch ", "Target architecture (arm64, amd64, or both)") + .action(async (opts) => { + const arch = opts.arch || detectArch(); + const qemuDir = findQemuDir(); + await runScript(qemuDir, "build-image.sh", [arch]); + }); + + emulator + .command("list-releases") + .description("List available emulator releases") + .option("--repo ", `GitHub repository (default: ${DEFAULT_REPO})`) + .action(async (opts) => { + const repo = opts.repo || DEFAULT_REPO; + console.log(`Available emulator releases from ${repo}:\n`); + try { + const output = ghRelease(["release", "list", "--repo", repo, "--limit", "20"]); + const lines = output.split("\n").filter((l) => l.toLowerCase().includes("emulator")); + if (lines.length === 0) { + console.log("No emulator releases found."); + } else { + for (const line of lines) { + console.log(line); + } + } + } catch (err) { + if (err instanceof CliError) throw err; + throw new CliError("Failed to list releases. Ensure GitHub CLI (gh) is installed and authenticated."); + } + }); +} diff --git a/packages/stack-cli/src/index.ts b/packages/stack-cli/src/index.ts index 08af5837ed..69f4ddc372 100644 --- a/packages/stack-cli/src/index.ts +++ b/packages/stack-cli/src/index.ts @@ -9,6 +9,7 @@ import { registerExecCommand } from "./commands/exec.js"; import { registerConfigCommand } from "./commands/config-file.js"; import { registerInitCommand } from "./commands/init.js"; import { registerProjectCommand } from "./commands/project.js"; +import { registerEmulatorCommand } from "./commands/emulator.js"; const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename); @@ -29,6 +30,7 @@ registerExecCommand(program); registerConfigCommand(program); registerInitCommand(program); registerProjectCommand(program); +registerEmulatorCommand(program); async function main() { try { From 4a43a88bf41d3c91571fb28dfe70888e544830d8 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 12:26:54 -0700 Subject: [PATCH 11/30] Update emulator configuration and enhance CLI functionality - Modified `stack.config.ts` to adjust authentication settings, enabling OTP and passkey sign-ins while disabling password sign-in. - Updated the GitHub Actions workflow for the QEMU emulator to improve the start and stop commands, enhancing clarity in the process. - Refactored the CLI commands for the emulator, removing snapshot options and streamlining the pull command for image downloads. - Added new tests for the emulator CLI to validate help and argument parsing, ensuring a better developer experience. These changes improve the configuration management and usability of the QEMU emulator for developers. --- .github/workflows/qemu-emulator-build.yaml | 38 +++------ apps/e2e/tests/general/cli.test.ts | 53 +++++++++++++ claude/CLAUDE-KNOWLEDGE.md | 10 +-- docker/local-emulator/qemu/run-emulator.sh | 87 ++------------------- packages/stack-cli/src/commands/emulator.ts | 35 ++------- stack.config.ts | 5 +- 6 files changed, 82 insertions(+), 146 deletions(-) diff --git a/.github/workflows/qemu-emulator-build.yaml b/.github/workflows/qemu-emulator-build.yaml index a268dba824..c4042d60ad 100644 --- a/.github/workflows/qemu-emulator-build.yaml +++ b/.github/workflows/qemu-emulator-build.yaml @@ -65,7 +65,7 @@ jobs: chmod +x docker/local-emulator/qemu/build-image.sh docker/local-emulator/qemu/build-image.sh ${{ matrix.arch }} - - name: Start emulator (creates runtime state) + - name: Start emulator and verify run: | chmod +x docker/local-emulator/qemu/run-emulator.sh EMULATOR_ARCH=${{ matrix.arch }} \ @@ -77,36 +77,21 @@ jobs: EMULATOR_ARCH=${{ matrix.arch }} \ docker/local-emulator/qemu/run-emulator.sh status - - name: Stop emulator (saves snapshot) + - name: Stop emulator run: | EMULATOR_ARCH=${{ matrix.arch }} \ docker/local-emulator/qemu/run-emulator.sh stop - - name: Package snapshot image + - name: Package image run: | - SNAPSHOT_IMG="docker/local-emulator/qemu/run/vm/disk.qcow2" BASE_IMG="docker/local-emulator/qemu/images/stack-emulator-${{ matrix.arch }}.qcow2" + cp "$BASE_IMG" "stack-emulator-${{ matrix.arch }}.qcow2" - # Compress snapshot image (contains VM state for instant restore) - qemu-img convert -p -O qcow2 -c "$SNAPSHOT_IMG" \ - "stack-emulator-${{ matrix.arch }}-snapshot.qcow2" - - # Copy base image (no snapshot, clean boot) - cp "$BASE_IMG" "stack-emulator-${{ matrix.arch }}-base.qcow2" - - - name: Upload snapshot artifact - uses: actions/upload-artifact@v4 - with: - name: qemu-emulator-${{ matrix.arch }}-snapshot - path: stack-emulator-${{ matrix.arch }}-snapshot.qcow2 - retention-days: 30 - compression-level: 0 - - - name: Upload base artifact + - name: Upload image artifact uses: actions/upload-artifact@v4 with: - name: qemu-emulator-${{ matrix.arch }}-base - path: stack-emulator-${{ matrix.arch }}-base.qcow2 + name: qemu-emulator-${{ matrix.arch }} + path: stack-emulator-${{ matrix.arch }}.qcow2 retention-days: 30 compression-level: 0 @@ -156,15 +141,12 @@ jobs: ### Images | File | Description | |------|-------------| - | `stack-emulator-arm64-snapshot.qcow2` | ARM64 with pre-warmed snapshot (instant boot) | - | `stack-emulator-arm64-base.qcow2` | ARM64 base image (clean boot) | - | `stack-emulator-amd64-snapshot.qcow2` | AMD64 with pre-warmed snapshot (instant boot) | - | `stack-emulator-amd64-base.qcow2` | AMD64 base image (clean boot) | + | `stack-emulator-arm64.qcow2` | ARM64 emulator image | + | `stack-emulator-amd64.qcow2` | AMD64 emulator image | ### Usage ```bash - # Pull and run the latest emulator via Stack CLI - stack emulator pull --snapshot + stack emulator pull stack emulator run ``` EOF diff --git a/apps/e2e/tests/general/cli.test.ts b/apps/e2e/tests/general/cli.test.ts index 82d5a7ba10..18650659a6 100644 --- a/apps/e2e/tests/general/cli.test.ts +++ b/apps/e2e/tests/general/cli.test.ts @@ -463,3 +463,56 @@ describe("Stack CLI", () => { expect(stdout).toContain("STACK AUTH SETUP INSTRUCTIONS"); }); }); + +// Emulator CLI tests — no backend required, just validates help/arg parsing +describe("Stack CLI — Emulator", () => { + function runCliBare( + args: string[], + ): Promise<{ stdout: string, stderr: string, exitCode: number | null }> { + return new Promise((resolve) => { + execFile("node", [CLI_BIN, ...args], { + env: { PATH: process.env.PATH ?? "", HOME: process.env.HOME ?? "", CI: "1" }, + timeout: 15_000, + }, (error, stdout, stderr) => { + resolve({ + stdout: stdout.toString(), + stderr: stderr.toString(), + exitCode: error ? (error as any).code ?? 1 : 0, + }); + }); + }); + } + + it("emulator help shows subcommands", async ({ expect }) => { + const { stdout, exitCode } = await runCliBare(["emulator", "--help"]); + expect(exitCode).toBe(0); + expect(stdout).toContain("pull"); + expect(stdout).toContain("run"); + expect(stdout).toContain("stop"); + expect(stdout).toContain("reset"); + expect(stdout).toContain("status"); + expect(stdout).toContain("build"); + expect(stdout).toContain("list-releases"); + }); + + it("emulator pull help shows options", async ({ expect }) => { + const { stdout, exitCode } = await runCliBare(["emulator", "pull", "--help"]); + expect(exitCode).toBe(0); + expect(stdout).toContain("--arch"); + expect(stdout).toContain("--branch"); + expect(stdout).toContain("--tag"); + expect(stdout).toContain("--repo"); + }); + + it("emulator build help shows arch option", async ({ expect }) => { + const { stdout, exitCode } = await runCliBare(["emulator", "build", "--help"]); + expect(exitCode).toBe(0); + expect(stdout).toContain("--arch"); + }); + + it("emulator list-releases help shows repo option", async ({ expect }) => { + const { stdout, exitCode } = await runCliBare(["emulator", "list-releases", "--help"]); + expect(exitCode).toBe(0); + expect(stdout).toContain("--repo"); + }); +}); diff --git a/claude/CLAUDE-KNOWLEDGE.md b/claude/CLAUDE-KNOWLEDGE.md index 5a5e03b213..8ad032bef2 100644 --- a/claude/CLAUDE-KNOWLEDGE.md +++ b/claude/CLAUDE-KNOWLEDGE.md @@ -136,14 +136,8 @@ A: The app container runs `docker/server/entrypoint.sh` which by default: (1) ru Q: Why can't STACK_SKIP_MIGRATIONS be set in the QEMU cloud-init user-data? A: During the QEMU image build, cloud-init provisions the VM using the same `render-stack-env` script as runtime. If `STACK_SKIP_MIGRATIONS=true` is hardcoded there, the build's container start also skips migrations (when the DB is actually empty). Since there's no STACKCFG ISO during build, the render-stack-env script fails anyway, but if it were fixed, the skip flag would prevent DB setup. Runtime-only flags should go in the runtime.env on the STACKCFG ISO (created by `run-emulator.sh`'s `prepare_runtime_config_iso`). -Q: How does the QEMU emulator snapshot restore work? -A: After a successful cold boot where all services are green, `run-emulator.sh` saves a QEMU `savevm` snapshot (named "ready") for both the deps and dev-server VMs via QMP. The snapshot includes full CPU/RAM/device state. On subsequent starts, if both overlays contain a "ready" snapshot, QEMU is launched with `-loadvm ready` which restores the entire VM state instantly (no Linux boot, no Docker start, no migrations). This reduces restart from ~62s to ~4s. If snapshot restore fails (services don't come up within 30s), the script automatically falls back to a fresh boot. Use `pnpm emulator-qemu:reset` to clear snapshots and force a fresh boot. - -Q: Why does the QEMU emulator use a deterministic file bridge token instead of a random one? -A: The file bridge token is baked into the VM's environment when the container starts. When restoring from a snapshot, the VM resumes with the old token. If the host bridge generates a new random token each time, the VM's token won't match and file bridge requests will fail. Using a deterministic token derived from the port prefix (`shasum -a 256` of `stack-local-emulator-$PORT_PREFIX`) ensures the same token on every start, making snapshot restore work seamlessly. - -Q: Why can't `qemu-img snapshot -l` be used to check snapshots while QEMU is running? -A: QEMU holds an exclusive write lock on the qcow2 file. `qemu-img` commands (including `snapshot -l`) fail with "Failed to get shared write lock". To check snapshots while QEMU runs, use QMP: `{"execute":"human-monitor-command","arguments":{"command-line":"info snapshots"}}`. To check snapshots when QEMU is stopped (e.g., in `vm_has_snapshot`), `qemu-img snapshot -l` works fine. +Q: How does the QEMU emulator persist data across restarts? +A: The emulator uses a qcow2 overlay disk backed by the base image. All writes go to the overlay, which is reused across restarts. This means DB changes, config, and other state persist without snapshots. The overlay is only recreated if the base image changes (detected via fingerprint). Use `pnpm emulator-qemu:reset` to discard the overlay and force a completely fresh boot. The host filesystem is mounted into the VM via virtfs (9p) at `/host`, giving the emulator read/write access to project files like `stack.config.ts`. Q: Why was the QEMU emulator consolidated from two VMs to one? A: Originally the emulator used two VMs: one for deps (postgres, redis, clickhouse, etc.) and one for the app (backend + dashboard). This was unnecessarily complex — both containers use `--network host` and can coexist in a single VM. The unified approach uses one VM with both Docker containers, all port forwards in a single QEMU netdev, one qcow2 overlay, and one snapshot. The cloud-init is in `docker/local-emulator/qemu/cloud-init/emulator/`. During the build phase, only the deps container starts (the app container fails because no STACKCFG ISO exists). At runtime, both containers start — deps first, then the app renders its env from STACKCFG and runs migrations. The deps host in the app's env is `127.0.0.1` (localhost within the same VM), while the file bridge URL still points to `10.0.2.2` (QEMU user-mode host gateway). Default resources: 8GB RAM / 6 CPUs (configurable via `EMULATOR_RAM` / `EMULATOR_CPUS`). diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index ce5e8b6dd4..58902363cf 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -59,11 +59,8 @@ runtime_iso_path() { echo "$VM_DIR/runtime-config.iso" } -SNAPSHOT_NAME="ready" -IS_SNAPSHOT_RESTORE=false - # Returns a fast fingerprint (size:mtime) of the base QEMU image. -# Used to detect whether the image has changed since the snapshot was saved. +# Used to detect whether the image has changed since the overlay was created. base_image_fingerprint() { local img="$1" case "$HOST_OS" in @@ -73,55 +70,6 @@ base_image_fingerprint() { esac } -vm_has_snapshot() { - [ -f "$VM_DIR/disk.qcow2" ] && - qemu-img snapshot -l "$VM_DIR/disk.qcow2" 2>/dev/null | grep -q "$SNAPSHOT_NAME" -} - -save_snapshot() { - if [ ! -S "$VM_DIR/monitor.sock" ]; then - warn "No monitor socket, skipping snapshot" - return 1 - fi - - log "Saving VM snapshot..." - local out_file="$VM_DIR/savevm.out" - rm -f "$out_file" - - ( - printf '{"execute":"qmp_capabilities"}\n' - sleep 0.3 - printf '{"execute":"human-monitor-command","arguments":{"command-line":"savevm %s"}}\n' "$SNAPSHOT_NAME" - sleep 180 - ) | socat -t 180 - "UNIX-CONNECT:$VM_DIR/monitor.sock" > "$out_file" 2>/dev/null & - local pid=$! - - local elapsed=0 - local snap_start=$SECONDS - while [ "$elapsed" -lt 120 ]; do - sleep 2 - elapsed=$((SECONDS - snap_start)) - printf "\r [%3ds] saving snapshot..." "$elapsed" - if [ -f "$out_file" ] && [ "$(grep -c '"return"' "$out_file" 2>/dev/null)" -ge 2 ]; then - echo "" - kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null || true - rm -f "$out_file" - base_image_fingerprint "$(image_path)" > "$VM_DIR/base-image.fingerprint" - log "Snapshot saved." - return 0 - fi - if ! kill -0 "$pid" 2>/dev/null; then - break - fi - done - echo "" - - kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null || true - rm -f "$out_file" - warn "Snapshot save timed out" - return 1 -} - prepare_runtime_config_iso() { local cfg_dir="$VM_DIR/runtime-config" local cfg_iso @@ -209,21 +157,20 @@ build_qemu_cmd() { fi mkdir -p "$VM_DIR" - local has_snapshot=false local fingerprint_file="$VM_DIR/base-image.fingerprint" local current_fp current_fp="$(base_image_fingerprint "$base_img")" - if [ -f "$VM_DIR/disk.qcow2" ] && vm_has_snapshot; then + if [ -f "$VM_DIR/disk.qcow2" ]; then if [ -f "$fingerprint_file" ] && [ "$(cat "$fingerprint_file")" = "$current_fp" ]; then - has_snapshot=true + log "Reusing existing overlay disk (changes persist)" else - warn "QEMU base image has changed — discarding stale snapshot." + warn "QEMU base image has changed — recreating overlay." rm -f "$VM_DIR/disk.qcow2" "$fingerprint_file" fi fi - if [ "$has_snapshot" = "false" ]; then - rm -f "$VM_DIR/disk.qcow2" + if [ ! -f "$VM_DIR/disk.qcow2" ]; then qemu-img create -f qcow2 -b "$base_img" -F qcow2 "$VM_DIR/disk.qcow2" >/dev/null + base_image_fingerprint "$base_img" > "$fingerprint_file" fi local qemu_bin machine cpu firmware_args=() @@ -284,10 +231,6 @@ build_qemu_cmd() { -pidfile "$VM_DIR/qemu.pid" ) - if [ "$has_snapshot" = "true" ]; then - QEMU_CMD+=(-loadvm "$SNAPSHOT_NAME") - IS_SNAPSHOT_RESTORE=true - fi } is_running() { @@ -353,26 +296,11 @@ cmd_start() { ensure_ports_free mkdir -p "$RUN_DIR" - IS_SNAPSHOT_RESTORE=false - info "Starting QEMU local emulator" info "Arch: $ARCH | Accel: $ACCEL | Prefix: $PORT_PREFIX" start_vm - if [ "$IS_SNAPSHOT_RESTORE" = "true" ]; then - info "Restoring from snapshot..." - if wait_for_condition "services (snapshot)" 30 all_ready; then - log "All services are green (restored from snapshot)." - return 0 - fi - warn "Snapshot restore failed. Resetting and doing fresh boot..." - stop_vm - rm -rf "$VM_DIR" - IS_SNAPSHOT_RESTORE=false - start_vm - fi - info "VM: ${VM_RAM}MB / ${VM_CPUS} CPUs" if ! wait_for_condition "deps services" "$READY_TIMEOUT" deps_ready; then @@ -385,8 +313,7 @@ cmd_start() { exit 1 fi - save_snapshot - log "All services are green. Snapshot saved for fast restart." + log "All services are green. Overlay disk preserves changes across restarts." } cmd_stop() { diff --git a/packages/stack-cli/src/commands/emulator.ts b/packages/stack-cli/src/commands/emulator.ts index c195c7b360..26d3339d4a 100644 --- a/packages/stack-cli/src/commands/emulator.ts +++ b/packages/stack-cli/src/commands/emulator.ts @@ -79,8 +79,6 @@ export function registerEmulatorCommand(program: Command) { emulator .command("pull") .description("Download the latest emulator image from GitHub Releases") - .option("--snapshot", "Download snapshot image (instant boot)", true) - .option("--base", "Download base image (clean boot)") .option("--arch ", "Target architecture (arm64 or amd64)") .option("--branch ", `Release branch (default: ${DEFAULT_BRANCH})`) .option("--tag ", "Specific release tag") @@ -89,9 +87,8 @@ export function registerEmulatorCommand(program: Command) { const arch = opts.arch || detectArch(); const repo = opts.repo || DEFAULT_REPO; const branch = opts.branch || DEFAULT_BRANCH; - const variant = opts.base ? "base" : "snapshot"; const tag = opts.tag || `emulator-${branch}-latest`; - const asset = `stack-emulator-${arch}-${variant}.qcow2`; + const asset = `stack-emulator-${arch}.qcow2`; const qemuDir = findQemuDir(); const imageDir = join(qemuDir, "images"); @@ -100,7 +97,7 @@ export function registerEmulatorCommand(program: Command) { const dest = join(imageDir, `stack-emulator-${arch}.qcow2`); const tmpDest = `${dest}.download`; - console.log(`Pulling ${variant} image for ${arch} from release ${tag}...`); + console.log(`Pulling image for ${arch} from release ${tag}...`); try { execSync( @@ -116,13 +113,6 @@ export function registerEmulatorCommand(program: Command) { renameSync(tmpDest, dest); console.log(`Downloaded: ${dest}`); - - if (variant === "snapshot") { - const runDir = join(qemuDir, "run", "vm"); - mkdirSync(runDir, { recursive: true }); - execSync(`cp ${JSON.stringify(dest)} ${JSON.stringify(join(runDir, "disk.qcow2"))}`); - console.log("Snapshot image installed — next start will restore instantly."); - } }); emulator @@ -135,7 +125,7 @@ export function registerEmulatorCommand(program: Command) { const img = join(qemuDir, "images", `stack-emulator-${arch}.qcow2`); if (!existsSync(img)) { - console.log("No emulator image found. Pulling latest snapshot..."); + console.log("No emulator image found. Pulling latest..."); // Re-invoke pull via the same program await program.parseAsync(["node", "stack", "emulator", "pull", "--arch", arch], { from: "user" }); } @@ -145,15 +135,10 @@ export function registerEmulatorCommand(program: Command) { emulator .command("stop") - .description("Stop the emulator (saves snapshot)") - .option("--no-snapshot", "Skip saving snapshot before stopping") - .action(async (opts) => { + .description("Stop the emulator") + .action(async () => { const qemuDir = findQemuDir(); - const env: Record = {}; - if (opts.snapshot === false) { - env.SKIP_SNAPSHOT = "true"; - } - await runScript(qemuDir, "run-emulator.sh", ["stop"], env); + await runScript(qemuDir, "run-emulator.sh", ["stop"]); }); emulator @@ -172,14 +157,6 @@ export function registerEmulatorCommand(program: Command) { await runScript(qemuDir, "run-emulator.sh", ["status"]); }); - emulator - .command("snapshot") - .description("Save a snapshot of the running emulator") - .action(async () => { - const qemuDir = findQemuDir(); - await runScript(qemuDir, "run-emulator.sh", ["snapshot"]); - }); - emulator .command("build") .description("Build the QEMU emulator image locally") diff --git a/stack.config.ts b/stack.config.ts index e5d702ec5a..4e9872c76e 100644 --- a/stack.config.ts +++ b/stack.config.ts @@ -6,5 +6,8 @@ export const config = { "apps.installed.api-keys.enabled": true, "apps.installed.webhooks.enabled": true, "apps.installed.launch-checklist.enabled": true, - "auth.password.allowSignIn": true + "auth.otp.allowSignIn": true, + "onboarding.requireEmailVerification": true, + "auth.passkey.allowSignIn": true, + "auth.password.allowSignIn": false }; From 5d98b44c34b608eaaf147a92be48b2b824143eb6 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 12:51:57 -0700 Subject: [PATCH 12/30] Remove deprecated configuration files and streamline emulator CLI commands - Deleted `stack.config.ts` and `CLAUDE-KNOWLEDGE.md` as they are no longer needed. - Refactored the `local-emulator.ts` to improve error handling and type definitions. - Updated the `header.tsx` component to remove the Emulator Status link, simplifying the navigation. - Enhanced the `emulator.ts` CLI commands for better readability and maintainability. These changes improve the overall structure and usability of the emulator and related components. --- apps/backend/src/lib/local-emulator.ts | 6 +- claude/CLAUDE-KNOWLEDGE.md | 143 -------------------- examples/demo/src/components/header.tsx | 3 - packages/stack-cli/src/commands/emulator.ts | 101 ++++++-------- stack.config.ts | 13 -- 5 files changed, 44 insertions(+), 222 deletions(-) delete mode 100644 claude/CLAUDE-KNOWLEDGE.md delete mode 100644 stack.config.ts diff --git a/apps/backend/src/lib/local-emulator.ts b/apps/backend/src/lib/local-emulator.ts index 08e45baf22..388fedcd51 100644 --- a/apps/backend/src/lib/local-emulator.ts +++ b/apps/backend/src/lib/local-emulator.ts @@ -56,8 +56,8 @@ async function resolveConfigFilePath(filePath: string): Promise { export async function readConfigFromFile(filePath: string): Promise> { const resolvedPath = await resolveConfigFilePath(filePath); - const content = await fs.readFile(resolvedPath, "utf-8").catch((error: unknown) => { - if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; + const content = await fs.readFile(resolvedPath, "utf-8").catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; throw error; }); @@ -66,7 +66,7 @@ export async function readConfigFromFile(filePath: string): Promise; + const mod: Record = jiti.evalModule(content, { filename: resolvedPath }); const config = mod.config; if (!isValidConfig(config)) { throw new StatusError(StatusError.BadRequest, `Invalid config in ${filePath}. The file must export a 'config' object.`); diff --git a/claude/CLAUDE-KNOWLEDGE.md b/claude/CLAUDE-KNOWLEDGE.md deleted file mode 100644 index 8ad032bef2..0000000000 --- a/claude/CLAUDE-KNOWLEDGE.md +++ /dev/null @@ -1,143 +0,0 @@ -# CLAUDE Knowledge Base - -Q: How are the development ports derived now that NEXT_PUBLIC_STACK_PORT_PREFIX exists? -A: Host ports use `${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}` plus the two-digit suffix (e.g., Postgres is `${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28`, Inbucket SMTP `${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}29`, POP3 `${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}30`, and OTLP `${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}31` by default). - -Q: How can I show helper text beneath metadata text areas in the dashboard? -A: Use the shared `TextAreaField` component's `helperText` prop in `apps/dashboard/src/components/form-fields.tsx`; it now renders the helper content in a secondary Typography line under the textarea. - -Q: How is the Email Template Editor structured? -A: It uses a hero-preview layout in `VibeCodeLayout` where the preview area dominates the screen. The code editor is hidden by default and accessible via a modal, while the AI assistant chat resides in a resizable right panel. Device viewport switching (Desktop/Tablet/Mobile) is integrated into the top toolbar. - -Q: How can I improve AI design generations for emails? -A: Update the system prompts in the backend's chat adapters (e.g., `apps/backend/src/lib/ai-chat/email-template-adapter.ts`). Providing explicit design principles, Tailwind CSS best practices, and structured technical rules helps the AI generate more polished and consistent designs. - -Q: What endpoint does the local Freestyle mock expose for script execution? -A: The mock server responds on `/execute/v1/script` and `/execute/v2/script` when built from `docker/dependencies/freestyle-mock/Dockerfile`; if the running image is older and only supports v1, backend dev can post to `/execute/v1/script` for email rendering. - -Q: How can I add a small Vitest check inside a client-only file? -A: Use `import.meta.vitest?.test(...)` at the bottom of the file for lightweight, in-file tests without adding a separate test file. -Q: Why did `pnpm typecheck` fail after deleting a Next.js route? -A: The generated `.next/types/validator.ts` can keep stale imports for removed routes. Deleting that file (or regenerating Next build output) clears the outdated references so `pnpm typecheck` succeeds again. - -Q: Why can auto-migrations time out and how should I mitigate it? -A: Auto-migrations run each migration inside a Prisma interactive transaction with an 80s timeout. Long-running statements (even if marked RUN_OUTSIDE_TRANSACTION_SENTINEL) still consume that time, so keep each iteration small using CONDITIONALLY_REPEAT_MIGRATION_SENTINEL and reduce batch sizes (e.g., lower LIMIT) so each transaction finishes under 80s. - -Q: How should `restricted_by_admin` updates handle reason fields? -A: When setting `restricted_by_admin` to false, explicitly clear `restricted_by_admin_reason` and `restricted_by_admin_private_details` to null (even if omitted in the PATCH) to satisfy the database constraint. - -Q: Where should `stackAppInternalsSymbol` be imported from in the dashboard? -A: Use the shared `apps/dashboard/src/lib/stack-app-internals.ts` export to avoid duplicating the Symbol.for definition across files. - -Q: How do we control whether a project requires publishable client keys? -A: Use the project-level config override field `project.requirePublishableClientKey` via `/api/v1/internal/config/override/project` or `AdminProject.update({ requirePublishableClientKey: ... })`. It defaults to false for new projects and is set true for existing projects via DB migration. - -Q: When adding new config fields, what else should be updated? -A: Update the config schema fuzzer configs in `packages/stack-shared/src/config/schema-fuzzer.test.ts` (for example, add the new field under `projectSchemaFuzzerConfig`/`branchSchemaFuzzerConfig`). - -Q: Why can't `canNoLongerBeOverridden` accept dotted paths? -A: It uses `schema.getNested`, which only allows keys with alphanumerics, `_`, `$`, `:`, or `-`. Dots are rejected, so mark the parent object key (e.g., `project`) as non-overridable instead. - -Q: Where is the editable-grid preview spacing controlled in the dashboard playground? -A: In `apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/playground/page-client.tsx`, the `selected === "editable-grid"` branch controls the card width/padding and the main preview container now uses `isExpandedPreview` to reduce outer gray padding only for editable-grid. - -Q: Why do editable-grid dropdown/boolean values sometimes not fill the full value column width? -A: In `apps/dashboard/src/components/design-components/editable-grid.tsx`, the value wrappers must be explicitly full-width (`w-full`) for boolean and dropdown fields, and the grid value cell container should also include `w-full`; otherwise controls shrink to content width. - -Q: How should dashboard inline editable text fields match the new design-components style? -A: Use `DesignInput` and `DesignButton` in `apps/dashboard/src/components/editable-input.tsx` (instead of legacy `Input`/`Button`) and style accept/reject actions as subtle glassy icon buttons with muted ring/border plus semantic hover tints. - -Q: What should dashboard email/project pages prefer for UI primitives? -A: Prefer `apps/dashboard/src/components/design-components/*` components (`DesignCard`, `DesignAlert`, `DesignBadge`, `DesignButton`, `DesignPillToggle`, `DesignCategoryTabs`, etc.) over page-local wrappers or repeated inline class patterns; current email surfaces still contain local patterns like custom GlassCard/SectionHeader/ViewportSelector that should be standardized. - -Q: What sections are expected in the dashboard design guide beyond component mapping? -A: Include explicit best-practices plus dedicated guidance for animation, typography, light/dark color system, micro-interactions, and spacing/layout rules so the guide is actionable for both humans and AI agents. - -Q: How should the project emails page cards align with the design system? -A: In `apps/dashboard/src/app/(main)/(protected)/projects/[projectId]/emails/page-client.tsx`, wrap the major sections with `DesignCard` from `@/components/design-components` (for example `gradient="default"`/`"purple"` with `glassmorphic`) instead of maintaining a local page-specific glass card wrapper. - -Q: Where is the default inner spacing for shared design cards controlled? -A: `apps/dashboard/src/components/design-components/card.tsx` sets the default content padding via `bodyPaddingClass` (currently `p-5`), and compact cards use `p-5` header plus `px-5 py-4` body spacing. - -Q: Why can two `DesignCard` surfaces look like they have different padding? -A: Pages can add extra local wrappers inside `DesignCard` (for example `p-5`, `px-5`, `pb-5`) which stack on top of `DesignCard` defaults; in the emails page, removing those local wrappers (`p-0`, `px-0`, `pb-0`) makes spacing match playground behavior. - -Q: How can a split section inside body-only `DesignCard` match header/content card borders? -A: Inside body-only cards (which already apply `p-5`), use a second section with `-mx-5 px-5` and `border-t border-black/[0.12] dark:border-white/[0.06]` so the divider spans full card width while content alignment matches `DesignCard` header/content layout. - -Q: How should cards handle header action buttons when using `title` + `subtitle`? -A: `DesignCard` now supports an `actions` prop when title/icon are provided; use `title`, `subtitle`, `icon`, and `actions` in pages like emails so header spacing and subtitle-bottom spacing exactly match playground/header variant styles without custom section-header workarounds. - -Q: What should we do after changing props in a core dashboard design component? -A: Update the playground implementation (`apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/playground/page-client.tsx`) in the same change so the component controls/examples reflect the new or changed props immediately. - -Q: How is the new `DesignCard` `actions` prop represented in playground? -A: The card playground now includes a `Header Actions` toggle that injects a sample `actions` slot (`DesignButton` with `Sliders` icon and "Configure") into `DesignCard` preview and generated code, only when `title` is present. - -Q: What is the reliable way to lint a single dashboard file in this monorepo? -A: Run lint from `apps/dashboard` directly (for example `pnpm lint -- "src/app/(main)/(protected)/projects/[projectId]/(overview)/line-chart.tsx"`), because running root `pnpm lint -- ` fans out through Turbo packages where that path does not exist. -Q: How should unsubscribe-link e2e tests avoid breakage from email theme/layout changes? -A: In `apps/e2e/tests/backend/endpoints/api/v1/unsubscribe-link.test.ts`, avoid snapshotting the entire rendered HTML for transactional emails; assert stable behavior instead (email content present and `/api/v1/emails/unsubscribe-link` absent) so cosmetic wrapper/style changes do not fail the test. - -Q: How should dashboard pages update project config values? -A: Do not call `project.updateConfig(...)` directly from dashboard pages; lint enforces using `useUpdateConfig()` from `apps/dashboard/src/lib/config-update.tsx` so pushable-config confirmation flows are handled consistently. - -Q: How can the dashboard find resumable onboarding state without SDK type changes? -A: Query `/internal/projects` via `stackAppInternalsSymbol` and read each project's `onboarding_status`; this avoids relying on `AdminOwnedProject` fields that may lag until generated package copies are rebuilt. - -Q: How should the new-project onboarding page avoid React's "Cannot update a component while rendering a different component" router error? -A: In `apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/new-project/page-client.tsx`, never call `router.replace(...)` during render when an onboarding project is already completed; move that redirect into a `useEffect` and render a plain spinner while the redirect is in progress. - -Q: What is the expected lightweight loading state when reopening an in-progress onboarding project? -A: On `apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/new-project/page-client.tsx`, the "loading onboarding" state should be just a centered `Spinner` with no card chrome or explanatory copy. - -Q: How should dashboard project onboarding status responses be handled to avoid silently bypassing onboarding? -A: Import `ProjectOnboardingStatus`/`projectOnboardingStatusValues` from `@stackframe/stack-shared/dist/schema-fields`, validate every `onboarding_status` from `/internal/projects`, and throw on invalid/missing values instead of defaulting to `"completed"`. - -Q: What E2E updates are required after adding `onboarding_status` to project API responses? -A: Update affected inline snapshots in `apps/e2e/tests/backend/endpoints/api/v1/**` to include `"onboarding_status": "completed"` in project payloads (for example projects, permissions, and integration provisioning/current endpoints), otherwise CI setup/restart E2E jobs fail with snapshot mismatches. - -Q: How should `createOrUpdateProjectWithLegacyConfig` handle `onboardingStatus` for forward-compat checks? -A: Only write `onboardingStatus` when the `Project.onboardingStatus` column exists (for example by checking `information_schema.columns` in-transaction) so current code can still run against older schemas where that column is absent. - -Q: How does the Docker local emulator make generated config files visible on the host filesystem? -A: In `docker/local-emulator/docker-compose.yaml`, the `stack-app` service now bind-mounts `"${HOME}:${HOME}"` and `"/tmp:/tmp"`, so local-emulator config paths under the user's home directory or `/tmp` resolve to the same absolute path inside and outside the container. - -Q: Why shouldn't the Docker app entrypoint use `/tmp/processed` as its runtime working directory when `/tmp` is bind-mounted? -A: With `/tmp` bind-mounted for host-visible config files, copying the full runtime tree into `/tmp/processed` pushes that heavy startup copy onto the host filesystem and makes boot much slower. `docker/server/entrypoint.sh` should keep its scratch runtime under a non-mounted path like `/var/tmp/stack-runtime` instead. - -Q: How can we verify the Docker local-emulator config-generation flow end to end? -A: POST `http://127.0.0.1:8102/api/v1/internal/local-emulator/project` with admin headers for the internal project and a body like `{"absolute_file_path":"/tmp/stack-auth-test-config-internal/stack.config.ts"}`. A successful `200` response should create `/tmp/stack-auth-test-config-internal/stack.config.ts` on the host containing `export const config = {};`. - -Q: What is the measured footprint of the Docker local emulator on an arm Mac once the stack is healthy? -A: With `pnpm run start-emulator` green on port prefix `81`, `docker stats --no-stream` showed about `578.6MiB` for `stack-deps` and `552.9MiB` for `stack-app`, for roughly `1.13GiB` RAM total. `docker image inspect` showed image sizes of about `1.44GB` (`stack-local-emulator-deps`) and `2.79GB` (`stack-local-emulator-app`), roughly `3.94GiB` combined image footprint, and `docker system df -v` showed another ~`77.7MiB` across the emulator's named volumes right after startup. - -Q: What made the split QEMU local-emulator build reliable on arm Macs? -A: The working path provisions two Debian arm64 guests that run the already-built `stack-local-emulator-deps` and `stack-local-emulator-app` Docker images inside the VM, instead of re-implementing the full service stack twice. The build script caches the Debian base image, reuses gzipped `docker save` bundles, and then provisions the `deps` and `dev-server` qcow images in parallel. - -Q: What subtle issues mattered for the QEMU image-bundle path? -A: Two details were critical: use a short ISO-safe bundle filename like `img.tgz` instead of a longer name such as `image.tar.gz`, and use Docker volumes inside the guest for the deps container rather than bind-mounting empty guest directories into `/data/*`. The short name avoids missing-file issues after mounting the ISO in the guest, and Docker volumes preserve the ownership expectations that the deps image's PostgreSQL initialization relies on. - -Q: How can we verify that the QEMU-backed local emulator is already seeded correctly? -A: Query the `stackframe` Postgres on host port `8128` and check for the local-emulator seed records directly: `ContactChannel.value='local-emulator@stack-auth.com'`, `ProjectUser.projectUserId='63abbc96-5329-454a-ba56-e0460173c6c1'` with display name `Local Emulator User`, `Team.teamId='5a0c858b-d9e9-49d4-9943-8ce385d86428'` with display name `Emulator Team`, and the matching `TeamMember` row. On the working QEMU stack these rows were all present under tenancy `3c69b8d4-55c0-4417-8a0b-2f1923d745f6`, confirming the app guest had already run migrations and seed on boot. - -Q: How should the QEMU local-emulator access host `stack.config.ts` paths reliably? -A: Use a host-side file bridge plus backend helper support rather than assuming the guest can read macOS host paths directly. In this repo that means `docker/local-emulator/qemu/host-file-bridge.mjs` running on the host, `apps/backend/src/lib/local-emulator.ts` reading/writing through `STACK_LOCAL_EMULATOR_FILE_BRIDGE_URL` and `STACK_LOCAL_EMULATOR_FILE_BRIDGE_TOKEN`, and `docker/local-emulator/qemu/run-emulator.sh` injecting those values into the dev-server guest runtime config. - -Q: What was the subtle process-lifecycle bug with the QEMU host file bridge on macOS? -A: Starting the bridge with a plain background shell job (even with `nohup`) was not reliable; the process printed its startup line and then died after the launcher shell exited. The durable fix was to spawn it in a new session from `docker/local-emulator/qemu/run-emulator.sh` using Python's `subprocess.Popen(..., start_new_session=True)` and then wait for `http://127.0.0.1:${PORT_PREFIX}16/health` before booting the app guest. - -Q: How should the QEMU image build decide whether to reuse a cached Docker image bundle? -A: Reusing `docker/local-emulator/qemu/images/*-docker-image.tar.gz` blindly causes stale guest images after the app Docker image changes. `docker/local-emulator/qemu/build-image.sh` should compare the current Docker image ID to a sidecar metadata file like `*.image-id` and only reuse the cached tarball when the IDs match; otherwise it must regenerate the bundle before provisioning the qcow image. - -Q: Why does the QEMU emulator's app container take so long to start, and what optimizations help? -A: The app container runs `docker/server/entrypoint.sh` which by default: (1) runs DB migrations, (2) runs seed, (3) copies the entire /app to a working directory (`cp -r /app/. /var/tmp/stack-runtime/.`), and (4) does find+sed sentinel replacement on all files. Migrations/seed cannot be skipped because they're never pre-run during the QEMU build (the STACKCFG ISO isn't present during build, so the app container fails to start during provisioning). Two optimizations cut startup from ~92s to ~62s: (a) use qcow2 backing files (`qemu-img create -f qcow2 -b base -F qcow2 overlay`) instead of copying the full 2.2GB base image, and (b) set `STACK_RUNTIME_WORK_DIR=/app` in the emulator env so the entrypoint skips the ~2.6GB app copy and does sentinel replacement in-place (safe since the container is ephemeral with `--rm`). - -Q: Why can't STACK_SKIP_MIGRATIONS be set in the QEMU cloud-init user-data? -A: During the QEMU image build, cloud-init provisions the VM using the same `render-stack-env` script as runtime. If `STACK_SKIP_MIGRATIONS=true` is hardcoded there, the build's container start also skips migrations (when the DB is actually empty). Since there's no STACKCFG ISO during build, the render-stack-env script fails anyway, but if it were fixed, the skip flag would prevent DB setup. Runtime-only flags should go in the runtime.env on the STACKCFG ISO (created by `run-emulator.sh`'s `prepare_runtime_config_iso`). - -Q: How does the QEMU emulator persist data across restarts? -A: The emulator uses a qcow2 overlay disk backed by the base image. All writes go to the overlay, which is reused across restarts. This means DB changes, config, and other state persist without snapshots. The overlay is only recreated if the base image changes (detected via fingerprint). Use `pnpm emulator-qemu:reset` to discard the overlay and force a completely fresh boot. The host filesystem is mounted into the VM via virtfs (9p) at `/host`, giving the emulator read/write access to project files like `stack.config.ts`. - -Q: Why was the QEMU emulator consolidated from two VMs to one? -A: Originally the emulator used two VMs: one for deps (postgres, redis, clickhouse, etc.) and one for the app (backend + dashboard). This was unnecessarily complex — both containers use `--network host` and can coexist in a single VM. The unified approach uses one VM with both Docker containers, all port forwards in a single QEMU netdev, one qcow2 overlay, and one snapshot. The cloud-init is in `docker/local-emulator/qemu/cloud-init/emulator/`. During the build phase, only the deps container starts (the app container fails because no STACKCFG ISO exists). At runtime, both containers start — deps first, then the app renders its env from STACKCFG and runs migrations. The deps host in the app's env is `127.0.0.1` (localhost within the same VM), while the file bridge URL still points to `10.0.2.2` (QEMU user-mode host gateway). Default resources: 8GB RAM / 6 CPUs (configurable via `EMULATOR_RAM` / `EMULATOR_CPUS`). diff --git a/examples/demo/src/components/header.tsx b/examples/demo/src/components/header.tsx index 989cce998e..8957fbb076 100644 --- a/examples/demo/src/components/header.tsx +++ b/examples/demo/src/components/header.tsx @@ -27,9 +27,6 @@ export default function Header() { Anonymous Test - - Emulator Status -
diff --git a/packages/stack-cli/src/commands/emulator.ts b/packages/stack-cli/src/commands/emulator.ts index 26d3339d4a..b05b1aa000 100644 --- a/packages/stack-cli/src/commands/emulator.ts +++ b/packages/stack-cli/src/commands/emulator.ts @@ -1,27 +1,23 @@ import { Command } from "commander"; -import { execSync, spawn } from "child_process"; -import { existsSync, mkdirSync, renameSync, unlinkSync, createWriteStream } from "fs"; +import { execFileSync, execSync, spawn } from "child_process"; +import { existsSync, mkdirSync, renameSync, unlinkSync } from "fs"; import { join, resolve } from "path"; import { CliError } from "../lib/errors.js"; const DEFAULT_REPO = "stack-auth/stack-auth"; const DEFAULT_BRANCH = "dev"; -function detectArch(): string { - const arch = process.arch; - if (arch === "arm64") { - return "arm64"; - } else if (arch === "x64") { - return "amd64"; - } else { - throw new CliError(`Unsupported architecture: ${arch}`); +type EmulatorArch = "arm64" | "amd64"; + +function detectArch(): EmulatorArch { + switch (process.arch) { + case "arm64": return "arm64"; + case "x64": return "amd64"; + default: throw new CliError(`Unsupported architecture: ${process.arch}`); } } function findQemuDir(): string { - // Walk up from this file to find the repo root, then locate qemu dir - // When running from the repo, it's relative to the monorepo root - // Try common locations const candidates = [ resolve(process.cwd(), "docker/local-emulator/qemu"), resolve(process.cwd(), "../docker/local-emulator/qemu"), @@ -38,34 +34,35 @@ function findQemuDir(): string { ); } -function runScript(qemuDir: string, script: string, args: string[], env?: Record) { - const scriptPath = join(qemuDir, script); - const result = spawn(scriptPath, args, { - stdio: "inherit", - env: { ...process.env, ...env }, - cwd: qemuDir, - }); +function runScript(qemuDir: string, script: string, args: string[], env?: Record): Promise { + return new Promise((resolve, reject) => { + const child = spawn(join(qemuDir, script), args, { + stdio: "inherit", + env: { ...process.env, ...env }, + cwd: qemuDir, + }); - return new Promise((resolve, reject) => { - result.on("close", (code) => { - if (code === 0) { - resolve(); - } else { - reject(new CliError(`${script} exited with code ${code}`)); - } + child.on("close", (code) => { + if (code === 0) resolve(); + else reject(new CliError(`${script} exited with code ${code}`)); }); - result.on("error", (err) => { + child.on("error", (err) => { reject(new CliError(`Failed to run ${script}: ${err.message}`)); }); }); } +function runEmulatorAction(action: string, env?: Record): Promise { + return runScript(findQemuDir(), "run-emulator.sh", [action], env); +} + function ghRelease(args: string[]): string { try { - return execSync(`gh ${args.join(" ")}`, { encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"] }).trim(); + return execFileSync("gh", args, { encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"] }).trim(); } catch (err: unknown) { if (err instanceof Error && "stderr" in err) { - throw new CliError(`GitHub CLI error: ${(err as { stderr: string }).stderr}`); + const { stderr } = err as Error & { stderr: string }; + throw new CliError(`GitHub CLI error: ${stderr}`); } throw new CliError("GitHub CLI (gh) is required. Install: https://cli.github.com/"); } @@ -84,7 +81,7 @@ export function registerEmulatorCommand(program: Command) { .option("--tag ", "Specific release tag") .option("--repo ", `GitHub repository (default: ${DEFAULT_REPO})`) .action(async (opts) => { - const arch = opts.arch || detectArch(); + const arch: EmulatorArch = opts.arch || detectArch(); const repo = opts.repo || DEFAULT_REPO; const branch = opts.branch || DEFAULT_BRANCH; const tag = opts.tag || `emulator-${branch}-latest`; @@ -94,7 +91,7 @@ export function registerEmulatorCommand(program: Command) { const imageDir = join(qemuDir, "images"); mkdirSync(imageDir, { recursive: true }); - const dest = join(imageDir, `stack-emulator-${arch}.qcow2`); + const dest = join(imageDir, asset); const tmpDest = `${dest}.download`; console.log(`Pulling image for ${arch} from release ${tag}...`); @@ -120,13 +117,12 @@ export function registerEmulatorCommand(program: Command) { .description("Start the emulator (auto-pulls if no image exists)") .option("--arch ", "Target architecture") .action(async (opts) => { - const arch = opts.arch || detectArch(); + const arch: EmulatorArch = opts.arch || detectArch(); const qemuDir = findQemuDir(); const img = join(qemuDir, "images", `stack-emulator-${arch}.qcow2`); if (!existsSync(img)) { console.log("No emulator image found. Pulling latest..."); - // Re-invoke pull via the same program await program.parseAsync(["node", "stack", "emulator", "pull", "--arch", arch], { from: "user" }); } @@ -136,26 +132,17 @@ export function registerEmulatorCommand(program: Command) { emulator .command("stop") .description("Stop the emulator") - .action(async () => { - const qemuDir = findQemuDir(); - await runScript(qemuDir, "run-emulator.sh", ["stop"]); - }); + .action(() => runEmulatorAction("stop")); emulator .command("reset") .description("Reset emulator state for a fresh boot") - .action(async () => { - const qemuDir = findQemuDir(); - await runScript(qemuDir, "run-emulator.sh", ["reset"]); - }); + .action(() => runEmulatorAction("reset")); emulator .command("status") .description("Show emulator and service health") - .action(async () => { - const qemuDir = findQemuDir(); - await runScript(qemuDir, "run-emulator.sh", ["status"]); - }); + .action(() => runEmulatorAction("status")); emulator .command("build") @@ -163,8 +150,7 @@ export function registerEmulatorCommand(program: Command) { .option("--arch ", "Target architecture (arm64, amd64, or both)") .action(async (opts) => { const arch = opts.arch || detectArch(); - const qemuDir = findQemuDir(); - await runScript(qemuDir, "build-image.sh", [arch]); + await runScript(findQemuDir(), "build-image.sh", [arch]); }); emulator @@ -174,19 +160,14 @@ export function registerEmulatorCommand(program: Command) { .action(async (opts) => { const repo = opts.repo || DEFAULT_REPO; console.log(`Available emulator releases from ${repo}:\n`); - try { - const output = ghRelease(["release", "list", "--repo", repo, "--limit", "20"]); - const lines = output.split("\n").filter((l) => l.toLowerCase().includes("emulator")); - if (lines.length === 0) { - console.log("No emulator releases found."); - } else { - for (const line of lines) { - console.log(line); - } + const output = ghRelease(["release", "list", "--repo", repo, "--limit", "20"]); + const lines = output.split("\n").filter((l) => l.toLowerCase().includes("emulator")); + if (lines.length === 0) { + console.log("No emulator releases found."); + } else { + for (const line of lines) { + console.log(line); } - } catch (err) { - if (err instanceof CliError) throw err; - throw new CliError("Failed to list releases. Ensure GitHub CLI (gh) is installed and authenticated."); } }); } diff --git a/stack.config.ts b/stack.config.ts deleted file mode 100644 index 4e9872c76e..0000000000 --- a/stack.config.ts +++ /dev/null @@ -1,13 +0,0 @@ -export const config = { - "apps.installed.authentication.enabled": true, - "apps.installed.emails.enabled": true, - "apps.installed.teams.enabled": true, - "apps.installed.rbac.enabled": true, - "apps.installed.api-keys.enabled": true, - "apps.installed.webhooks.enabled": true, - "apps.installed.launch-checklist.enabled": true, - "auth.otp.allowSignIn": true, - "onboarding.requireEmailVerification": true, - "auth.passkey.allowSignIn": true, - "auth.password.allowSignIn": false -}; From 3321e5c466f3d57c2c35085a3f57773d22902564 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 12:52:41 -0700 Subject: [PATCH 13/30] Update QEMU emulator scripts to improve KVM detection and service status reporting - Modified `build-image.sh` and `run-emulator.sh` to check for write permissions on `/dev/kvm` instead of existence, enhancing compatibility with various environments. - Added status tracking for services in `run-emulator.sh`, allowing for better error handling and reporting during emulator status checks. These changes enhance the reliability and usability of the QEMU emulator setup. --- docker/local-emulator/qemu/build-image.sh | 4 ++-- docker/local-emulator/qemu/run-emulator.sh | 8 +++++++- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/docker/local-emulator/qemu/build-image.sh b/docker/local-emulator/qemu/build-image.sh index feca447fad..302403bf01 100755 --- a/docker/local-emulator/qemu/build-image.sh +++ b/docker/local-emulator/qemu/build-image.sh @@ -91,7 +91,7 @@ qemu_cmd_prefix_for_arch() { if [ "$HOST_ARCH" = "arm64" ]; then case "$HOST_OS" in darwin) accel="hvf" ;; - linux) [ -e /dev/kvm ] && accel="kvm" ;; + linux) [ -w /dev/kvm ] && accel="kvm" ;; esac fi local firmware @@ -104,7 +104,7 @@ qemu_cmd_prefix_for_arch() { if [ "$HOST_ARCH" = "amd64" ]; then case "$HOST_OS" in darwin) accel="hvf" ;; - linux) [ -e /dev/kvm ] && accel="kvm" ;; + linux) [ -w /dev/kvm ] && accel="kvm" ;; esac else cpu="qemu64" diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index 58902363cf..24461976bf 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -38,7 +38,7 @@ select_accelerator() { fi ;; linux) - if [ -e /dev/kvm ]; then + if [ -w /dev/kvm ]; then accel="kvm" fi ;; @@ -327,6 +327,8 @@ cmd_reset() { log "Emulator state reset. Next start will be a fresh boot." } +STATUS_FAILED=0 + print_service_status() { local name="$1" local port="$2" @@ -337,15 +339,18 @@ print_service_status() { echo -e " ${GREEN}●${NC} $name (:$port)" else echo -e " ${RED}●${NC} $name (:$port)" + STATUS_FAILED=1 fi } cmd_status() { + STATUS_FAILED=0 echo "VM:" if is_running; then echo -e " ${GREEN}●${NC} emulator" else echo -e " ${RED}●${NC} emulator" + STATUS_FAILED=1 fi echo "" echo "Services:" @@ -357,6 +362,7 @@ cmd_status() { print_service_status "MinIO" "${PORT_PREFIX}21" http /minio/health/live print_service_status "QStash" "${PORT_PREFIX}25" http / 401 print_service_status "ClickHouse" "${PORT_PREFIX}36" http /ping + exit "$STATUS_FAILED" } cmd_bench() { From 8336d4821d5c984d2aaabb7a7f21357ede99943d Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 12:55:44 -0700 Subject: [PATCH 14/30] Refactor local emulator type handling and architecture detection - Updated `local-emulator.ts` to explicitly cast the module evaluation result to `Record`, improving type safety. - Enhanced `emulator.ts` architecture detection with clearer case structures for better readability and maintainability. These changes improve code clarity and type management in the local emulator setup. --- apps/backend/src/lib/local-emulator.ts | 2 +- packages/stack-cli/src/commands/emulator.ts | 12 +++++++++--- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/apps/backend/src/lib/local-emulator.ts b/apps/backend/src/lib/local-emulator.ts index 388fedcd51..6caf098797 100644 --- a/apps/backend/src/lib/local-emulator.ts +++ b/apps/backend/src/lib/local-emulator.ts @@ -66,7 +66,7 @@ export async function readConfigFromFile(filePath: string): Promise = jiti.evalModule(content, { filename: resolvedPath }); + const mod = jiti.evalModule(content, { filename: resolvedPath }) as Record; const config = mod.config; if (!isValidConfig(config)) { throw new StatusError(StatusError.BadRequest, `Invalid config in ${filePath}. The file must export a 'config' object.`); diff --git a/packages/stack-cli/src/commands/emulator.ts b/packages/stack-cli/src/commands/emulator.ts index b05b1aa000..de5652f83d 100644 --- a/packages/stack-cli/src/commands/emulator.ts +++ b/packages/stack-cli/src/commands/emulator.ts @@ -11,9 +11,15 @@ type EmulatorArch = "arm64" | "amd64"; function detectArch(): EmulatorArch { switch (process.arch) { - case "arm64": return "arm64"; - case "x64": return "amd64"; - default: throw new CliError(`Unsupported architecture: ${process.arch}`); + case "arm64": { + return "arm64"; + } + case "x64": { + return "amd64"; + } + default: { + throw new CliError(`Unsupported architecture: ${process.arch}`); + } } } From 298062a8954b95d8396ad27615eae7ecb2c01e88 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 13:26:52 -0700 Subject: [PATCH 15/30] Enhance local emulator configuration handling and file management - Added functionality to write an empty configuration file if none exists when processing a project in `route.tsx`. - Updated `readConfigFromFile` in `local-emulator.ts` to support reading configuration from an environment variable, improving flexibility in configuration management. - Ensured the directory structure is created when writing configuration files, enhancing file handling robustness. These changes improve the local emulator's configuration management and usability for developers. --- .../internal/local-emulator/project/route.tsx | 5 +++++ apps/backend/src/lib/local-emulator.ts | 14 +++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx b/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx index 0635cd388d..49c0efd451 100644 --- a/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx +++ b/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx @@ -5,6 +5,7 @@ import { LOCAL_EMULATOR_OWNER_TEAM_ID, isLocalEmulatorEnabled, readConfigFromFile, + writeConfigToFile, } from "@/lib/local-emulator"; import { DEFAULT_BRANCH_ID, getSoleTenancyFromProjectBranch } from "@/lib/tenancies"; import { getPrismaClientForTenancy, globalPrismaClient } from "@/prisma-client"; @@ -197,6 +198,10 @@ export const POST = createSmartRouteHandler({ const credentials = await getOrCreateCredentials(projectId); const fileConfig = await readConfigFromFile(absoluteFilePath); + if (Object.keys(fileConfig).length === 0) { + await writeConfigToFile(absoluteFilePath, {}); + } + return { statusCode: 200 as const, bodyType: "json" as const, diff --git a/apps/backend/src/lib/local-emulator.ts b/apps/backend/src/lib/local-emulator.ts index 6caf098797..67137cbf31 100644 --- a/apps/backend/src/lib/local-emulator.ts +++ b/apps/backend/src/lib/local-emulator.ts @@ -55,11 +55,14 @@ async function resolveConfigFilePath(filePath: string): Promise { } export async function readConfigFromFile(filePath: string): Promise> { - const resolvedPath = await resolveConfigFilePath(filePath); - const content = await fs.readFile(resolvedPath, "utf-8").catch((error: NodeJS.ErrnoException) => { - if (error.code === "ENOENT") return null; - throw error; - }); + const envContent = getEnvVariable("STACK_LOCAL_EMULATOR_CONFIG_CONTENT", ""); + const resolvedPath = envContent ? filePath : await resolveConfigFilePath(filePath); + const content = envContent + ? Buffer.from(envContent, "base64").toString("utf-8") + : await fs.readFile(resolvedPath, "utf-8").catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); if (content === null || content.trim() === "") { return {}; @@ -76,6 +79,7 @@ export async function readConfigFromFile(filePath: string): Promise): Promise { const resolvedPath = await resolveConfigFilePath(filePath); + await fs.mkdir(path.dirname(resolvedPath), { recursive: true }); const configString = JSON.stringify(config, null, 2); const content = `export const config = ${configString};\n`; await fs.writeFile(resolvedPath, content, "utf-8"); From 578593a6da84e96085055513a26e27ef7dbd7c92 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 15:09:04 -0700 Subject: [PATCH 16/30] Enhance QEMU emulator build workflow and local emulator configuration - Updated the GitHub Actions workflow to improve the release process for QEMU emulator images, including checks for existing releases and editing them if necessary. - Refactored local emulator configuration handling to ensure proper file writing and directory creation, enhancing robustness. - Added tests for writing configuration files to validate functionality and improve reliability. These changes improve the automation of the QEMU emulator build process and enhance the local emulator's configuration management for developers. --- .github/workflows/qemu-emulator-build.yaml | 37 ++++++++++++++----- apps/backend/src/lib/config.tsx | 2 +- apps/backend/src/lib/local-emulator.test.ts | 22 ++++++++++- apps/backend/src/lib/local-emulator.ts | 7 +++- claude/CLAUDE-KNOWLEDGE.md | 4 ++ docker/local-emulator/Dockerfile | 6 +-- docker/local-emulator/base.env | 12 ++++++ docker/local-emulator/docker-compose.yaml | 12 +----- docker/local-emulator/init-services.sh | 9 +++++ docker/local-emulator/qemu/build-image.sh | 2 +- .../qemu/cloud-init/emulator/user-data | 7 +++- docker/local-emulator/start-app.sh | 11 ++++++ packages/stack-cli/src/commands/emulator.ts | 7 +++- 13 files changed, 106 insertions(+), 32 deletions(-) create mode 100644 claude/CLAUDE-KNOWLEDGE.md diff --git a/.github/workflows/qemu-emulator-build.yaml b/.github/workflows/qemu-emulator-build.yaml index c4042d60ad..07f65b5d8c 100644 --- a/.github/workflows/qemu-emulator-build.yaml +++ b/.github/workflows/qemu-emulator-build.yaml @@ -159,13 +159,30 @@ jobs: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | LATEST_TAG="emulator-${{ github.ref_name }}-latest" - # Delete existing latest release if it exists - gh release delete "$LATEST_TAG" --yes 2>/dev/null || true - git tag -d "$LATEST_TAG" 2>/dev/null || true - git push origin ":refs/tags/$LATEST_TAG" 2>/dev/null || true - - gh release create "$LATEST_TAG" \ - --title "QEMU Emulator — ${{ github.ref_name }} (latest)" \ - --notes "Latest emulator images from \`${{ github.ref_name }}\`. Auto-updated on each build." \ - --prerelease \ - release/* + TITLE="QEMU Emulator — ${{ github.ref_name }} (latest)" + NOTES="Latest emulator images from \`${{ github.ref_name }}\`. Auto-updated on each build." + + if gh release view "$LATEST_TAG" >/dev/null 2>&1; then + gh release edit "$LATEST_TAG" \ + --draft \ + --prerelease \ + --target "${{ github.sha }}" \ + --title "$TITLE" \ + --notes "$NOTES" + else + gh release create "$LATEST_TAG" \ + --draft \ + --prerelease \ + --target "${{ github.sha }}" \ + --title "$TITLE" \ + --notes "$NOTES" \ + || gh release edit "$LATEST_TAG" \ + --draft \ + --prerelease \ + --target "${{ github.sha }}" \ + --title "$TITLE" \ + --notes "$NOTES" + fi + + gh release upload "$LATEST_TAG" release/* --clobber + gh release edit "$LATEST_TAG" --draft=false --prerelease diff --git a/apps/backend/src/lib/config.tsx b/apps/backend/src/lib/config.tsx index c73c09a1af..995829cbd6 100644 --- a/apps/backend/src/lib/config.tsx +++ b/apps/backend/src/lib/config.tsx @@ -299,7 +299,7 @@ export async function setBranchConfigOverride(options: { // In the local emulator, write config changes back to the config file if (isLocalEmulatorEnabled()) { const filePath = await getLocalEmulatorFilePath(options.projectId); - if (filePath) { + if (filePath != null) { await writeConfigToFile(filePath, newConfig); } } diff --git a/apps/backend/src/lib/local-emulator.test.ts b/apps/backend/src/lib/local-emulator.test.ts index 028c5000ba..aff12f2b28 100644 --- a/apps/backend/src/lib/local-emulator.test.ts +++ b/apps/backend/src/lib/local-emulator.test.ts @@ -1,5 +1,6 @@ +import fs from "fs/promises"; import { afterEach, describe, expect, it, vi } from "vitest"; -import { readConfigFromFile } from "./local-emulator"; +import { readConfigFromFile, writeConfigToFile } from "./local-emulator"; describe("local emulator config", () => { afterEach(() => { @@ -30,4 +31,23 @@ describe("local emulator config", () => { await expect(readConfigFromFile("/irrelevant/path/stack.config.ts")).resolves.toEqual({}); }); + + it("writes new config files to the host mount when the mounted parent directory exists", async () => { + const accessSpy = vi.spyOn(fs, "access") + .mockRejectedValueOnce(Object.assign(new Error("missing file"), { code: "ENOENT" })) + .mockResolvedValueOnce(undefined); + const mkdirSpy = vi.spyOn(fs, "mkdir").mockResolvedValueOnce("/host/Users/foo/project"); + const writeFileSpy = vi.spyOn(fs, "writeFile").mockResolvedValueOnce(); + + await writeConfigToFile("/Users/foo/project/stack.config.ts", { auth: { allowLocalhost: true } }); + + expect(accessSpy).toHaveBeenNthCalledWith(1, "/host/Users/foo/project/stack.config.ts"); + expect(accessSpy).toHaveBeenNthCalledWith(2, "/host/Users/foo/project"); + expect(mkdirSpy).toHaveBeenCalledWith("/host/Users/foo/project", { recursive: true }); + expect(writeFileSpy).toHaveBeenCalledWith( + "/host/Users/foo/project/stack.config.ts", + `export const config = {\n "auth": {\n "allowLocalhost": true\n }\n};\n`, + "utf-8", + ); + }); }); diff --git a/apps/backend/src/lib/local-emulator.ts b/apps/backend/src/lib/local-emulator.ts index 67137cbf31..dd2e46dd84 100644 --- a/apps/backend/src/lib/local-emulator.ts +++ b/apps/backend/src/lib/local-emulator.ts @@ -50,7 +50,12 @@ async function resolveConfigFilePath(filePath: string): Promise { await fs.access(hostMountedPath); return hostMountedPath; } catch { - return filePath; + try { + await fs.access(path.dirname(hostMountedPath)); + return hostMountedPath; + } catch { + return filePath; + } } } diff --git a/claude/CLAUDE-KNOWLEDGE.md b/claude/CLAUDE-KNOWLEDGE.md new file mode 100644 index 0000000000..fbefcdf2d2 --- /dev/null +++ b/claude/CLAUDE-KNOWLEDGE.md @@ -0,0 +1,4 @@ +# CLAUDE-KNOWLEDGE + +## Q: How should QEMU local-emulator containers reach host-only dev services like the OAuth mock server? +A: Keep container-local dependencies on `127.0.0.1`, but point host-only services at QEMU's user-network host alias `10.0.2.2`. Using `127.0.0.1:${PORT}` for host services from inside the guest container points back at the container itself. diff --git a/docker/local-emulator/Dockerfile b/docker/local-emulator/Dockerfile index 52eab95b68..7f9e6d45a3 100644 --- a/docker/local-emulator/Dockerfile +++ b/docker/local-emulator/Dockerfile @@ -92,10 +92,10 @@ RUN cp -a /app/node_modules /pruned-node_modules && \ # ── Service binary stages ───────────────────────────────────────────────────── FROM inbucket/inbucket:3.1.0 AS inbucket-bin -FROM svix/svix-server:latest AS svix-bin +FROM svix/svix-server:v1.88.0 AS svix-bin FROM clickhouse/clickhouse-server:25.10 AS clickhouse-bin -FROM minio/minio:latest AS minio-bin -FROM minio/mc:latest AS mc-bin +FROM minio/minio:RELEASE.2025-09-07T16-13-09Z AS minio-bin +FROM minio/mc:RELEASE.2025-02-21T16-00-46Z AS mc-bin FROM bgodil/qstash:latest AS qstash-bin RUN cp $(which qstash) /qstash-binary 2>/dev/null || \ diff --git a/docker/local-emulator/base.env b/docker/local-emulator/base.env index b5bca9b838..8d356cfc9a 100644 --- a/docker/local-emulator/base.env +++ b/docker/local-emulator/base.env @@ -64,3 +64,15 @@ STACK_MICROSOFT_CLIENT_SECRET=MOCK STACK_SPOTIFY_CLIENT_ID=MOCK STACK_SPOTIFY_CLIENT_SECRET=MOCK STACK_ALLOW_SHARED_OAUTH_ACCESS_TOKENS=true + +# Internal service endpoints (defaults for docker-compose; overridden in QEMU) +STACK_DATABASE_CONNECTION_STRING=postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@127.0.0.1:5432/stackframe +STACK_EMAIL_HOST=127.0.0.1 +STACK_SVIX_SERVER_URL=http://127.0.0.1:8071 +STACK_S3_ENDPOINT=http://127.0.0.1:9090 +STACK_QSTASH_URL=http://127.0.0.1:8080 +STACK_CLICKHOUSE_URL=http://127.0.0.1:8123 +STACK_CLICKHOUSE_DATABASE=analytics +STACK_EMAIL_MONITOR_INBUCKET_API_URL=http://127.0.0.1:9001 +BACKEND_PORT=8102 +DASHBOARD_PORT=8101 diff --git a/docker/local-emulator/docker-compose.yaml b/docker/local-emulator/docker-compose.yaml index 82696d5d95..0a434c5271 100644 --- a/docker/local-emulator/docker-compose.yaml +++ b/docker/local-emulator/docker-compose.yaml @@ -29,7 +29,7 @@ services: - "/tmp:/tmp" env_file: ./base.env environment: - # Port-prefixed URLs (host-specific) + # Port-prefixed URLs — need shell interpolation, can't go in env_file NEXT_PUBLIC_STACK_PORT_PREFIX: "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}" NEXT_PUBLIC_STACK_API_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02" NEXT_PUBLIC_STACK_DASHBOARD_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01" @@ -37,16 +37,6 @@ services: STACK_S3_PUBLIC_ENDPOINT: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}21/stack-storage" STACK_EMAIL_MONITOR_VERIFICATION_CALLBACK_URL: "http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01/handler/email-verification" STACK_OAUTH_MOCK_URL: "http://host.docker.internal:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}14" - BACKEND_PORT: "8102" - DASHBOARD_PORT: "8101" - # Everything is in one container, so deps are at localhost - STACK_DATABASE_CONNECTION_STRING: "postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@127.0.0.1:5432/stackframe" - STACK_EMAIL_HOST: "127.0.0.1" - STACK_SVIX_SERVER_URL: "http://127.0.0.1:8071" - STACK_S3_ENDPOINT: "http://127.0.0.1:9090" - STACK_QSTASH_URL: "http://127.0.0.1:8080" - STACK_CLICKHOUSE_URL: "http://127.0.0.1:8123" - STACK_EMAIL_MONITOR_INBUCKET_API_URL: "http://127.0.0.1:9001" extra_hosts: - "host.docker.internal:host-gateway" healthcheck: diff --git a/docker/local-emulator/init-services.sh b/docker/local-emulator/init-services.sh index bc1af81654..8fd1f7de32 100644 --- a/docker/local-emulator/init-services.sh +++ b/docker/local-emulator/init-services.sh @@ -1,6 +1,12 @@ #!/bin/bash set -e +INIT_SERVICES_DONE_FILE=/var/run/stack-local-init-services.done +INIT_SERVICES_FAILED_FILE=/var/run/stack-local-init-services.failed + +rm -f "$INIT_SERVICES_DONE_FILE" "$INIT_SERVICES_FAILED_FILE" +trap 'touch "$INIT_SERVICES_FAILED_FILE"' ERR + wait_for_http() { local url="$1" attempts=0 while [ "$attempts" -lt 60 ]; do @@ -19,3 +25,6 @@ mc mb --ignore-existing local/stack-storage-private wait_for_http http://127.0.0.1:8123/ping curl -s "http://127.0.0.1:8123/?user=default" --data "CREATE DATABASE IF NOT EXISTS analytics" + +rm -f "$INIT_SERVICES_FAILED_FILE" +touch "$INIT_SERVICES_DONE_FILE" diff --git a/docker/local-emulator/qemu/build-image.sh b/docker/local-emulator/qemu/build-image.sh index 302403bf01..0e8fb70116 100755 --- a/docker/local-emulator/qemu/build-image.sh +++ b/docker/local-emulator/qemu/build-image.sh @@ -12,7 +12,7 @@ DEBIAN_VERSION="${DEBIAN_VERSION:-13}" DISK_SIZE="${EMULATOR_DISK_SIZE:-12G}" RAM="${EMULATOR_BUILD_RAM:-4096}" CPUS="${EMULATOR_BUILD_CPUS:-$(sysctl -n hw.ncpu 2>/dev/null || nproc 2>/dev/null || echo 4)}" -PROVISION_TIMEOUT="${EMULATOR_PROVISION_TIMEOUT:-1800}" +PROVISION_TIMEOUT="${EMULATOR_PROVISION_TIMEOUT:-3200}" RED='\033[0;31m' GREEN='\033[0;32m' diff --git a/docker/local-emulator/qemu/cloud-init/emulator/user-data b/docker/local-emulator/qemu/cloud-init/emulator/user-data index 953173101c..3a45e03e3b 100644 --- a/docker/local-emulator/qemu/cloud-init/emulator/user-data +++ b/docker/local-emulator/qemu/cloud-init/emulator/user-data @@ -58,8 +58,11 @@ write_files: source /mnt/stack-runtime/base.env set +a - # Everything runs in the same container, so always localhost + # Container-local dependencies run on localhost. Host-only development + # services (such as the OAuth mock server) are reachable via the QEMU + # user-network host alias. DEPS_HOST=127.0.0.1 + HOST_SERVICES_HOST=10.0.2.2 P="$STACK_EMULATOR_PORT_PREFIX" { @@ -87,7 +90,7 @@ write_files: STACK_CLICKHOUSE_URL=http://${DEPS_HOST}:8123 STACK_EMAIL_MONITOR_VERIFICATION_CALLBACK_URL=http://localhost:${P}01/handler/email-verification STACK_EMAIL_MONITOR_INBUCKET_API_URL=http://${DEPS_HOST}:9001 - STACK_OAUTH_MOCK_URL=http://${DEPS_HOST}:${P}14 + STACK_OAUTH_MOCK_URL=http://${HOST_SERVICES_HOST}:${P}14 BACKEND_PORT=${P}02 DASHBOARD_PORT=${P}01 COMPUTED diff --git a/docker/local-emulator/start-app.sh b/docker/local-emulator/start-app.sh index 4b2e610c31..ad7472732d 100644 --- a/docker/local-emulator/start-app.sh +++ b/docker/local-emulator/start-app.sh @@ -11,10 +11,21 @@ fi # Wait for all infrastructure services to be ready before running migrations # and starting the backend/dashboard. +INIT_SERVICES_DONE_FILE=/var/run/stack-local-init-services.done +INIT_SERVICES_FAILED_FILE=/var/run/stack-local-init-services.failed + until pg_isready -h 127.0.0.1 -p 5432 -U postgres >/dev/null 2>&1; do sleep 2; done until curl -sf http://127.0.0.1:8123/ping >/dev/null 2>&1; do sleep 2; done until curl -sf http://127.0.0.1:8071/api/v1/health/ >/dev/null 2>&1; do sleep 2; done until curl -sf http://127.0.0.1:9090/minio/health/live >/dev/null 2>&1; do sleep 2; done until [ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/ 2>/dev/null || true)" = "401" ]; do sleep 2; done +until [ -f "$INIT_SERVICES_DONE_FILE" ]; do + if [ -f "$INIT_SERVICES_FAILED_FILE" ]; then + echo "init-services.sh failed; refusing to start the app." >&2 + exit 1 + fi + sleep 1 +done + exec /app-entrypoint.sh diff --git a/packages/stack-cli/src/commands/emulator.ts b/packages/stack-cli/src/commands/emulator.ts index de5652f83d..65b9a161a2 100644 --- a/packages/stack-cli/src/commands/emulator.ts +++ b/packages/stack-cli/src/commands/emulator.ts @@ -107,10 +107,13 @@ export function registerEmulatorCommand(program: Command) { `gh release download ${JSON.stringify(tag)} --repo ${JSON.stringify(repo)} --pattern ${JSON.stringify(asset)} --output ${JSON.stringify(tmpDest)} --clobber`, { stdio: "inherit" } ); - } catch { + } catch (err) { if (existsSync(tmpDest)) unlinkSync(tmpDest); + const reason = err instanceof Error + ? (err.stack ?? err.message) + : String(err); throw new CliError( - `Failed to download ${asset} from release ${tag}.\nRun 'stack emulator list-releases' to see available releases.` + `Failed to download ${asset} from release ${tag}: ${reason}\nRun 'stack emulator list-releases' to see available releases.` ); } From 42bc12587cc0b5d0100862ec2d111efa706d0e82 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 15:31:17 -0700 Subject: [PATCH 17/30] Enhance setBranchConfigOverride for local emulator file handling - Updated the `setBranchConfigOverride` function to write branch configuration directly to a file when the local emulator is enabled, ensuring the file serves as the single source of truth. - Added tests to verify that configuration writes to the local emulator file are handled correctly and that database operations are skipped when writing to the file. - Improved error handling to surface file write failures before attempting database updates. These changes enhance the local emulator's configuration management and improve reliability for developers working with branch configurations. --- apps/backend/src/lib/config.tsx | 86 +++++++++++++++++++-- packages/stack-cli/src/commands/emulator.ts | 18 +++-- 2 files changed, 91 insertions(+), 13 deletions(-) diff --git a/apps/backend/src/lib/config.tsx b/apps/backend/src/lib/config.tsx index 995829cbd6..5e58e4bb70 100644 --- a/apps/backend/src/lib/config.tsx +++ b/apps/backend/src/lib/config.tsx @@ -279,6 +279,17 @@ export async function setBranchConfigOverride(options: { if (overrideErrors.status === "error") { captureError("setBranchConfigOverride", new StackAssertionError(`Config override is invalid — at a place where it should have already been validated! ${overrideErrors.error}`, { projectId: options.projectId, branchId: options.branchId })); } + + if (isLocalEmulatorEnabled()) { + const filePath = await getLocalEmulatorFilePath(options.projectId); + if (filePath != null) { + // Local emulator projects read branch config directly from the host config file, + // so the file stays the single source of truth to avoid split DB/file state. + await writeConfigToFile(filePath, newConfig); + return; + } + } + await globalPrismaClient.branchConfigOverride.upsert({ where: { projectId_branchId: { @@ -295,14 +306,6 @@ export async function setBranchConfigOverride(options: { config: newConfig, }, }); - - // In the local emulator, write config changes back to the config file - if (isLocalEmulatorEnabled()) { - const filePath = await getLocalEmulatorFilePath(options.projectId); - if (filePath != null) { - await writeConfigToFile(filePath, newConfig); - } - } } /** @@ -1104,6 +1107,73 @@ import.meta.vitest?.test('setEnvironmentConfigOverride blocks writes in local em } }); +import.meta.vitest?.test('setBranchConfigOverride writes local emulator config to the file instead of the DB', async ({ expect }) => { + const vi = import.meta.vitest?.vi; + if (!vi) { + throw new StackAssertionError("Vitest context is required for in-source tests."); + } + + const localEmulator = await import("./local-emulator"); + const isLocalEmulatorEnabledSpy = vi.spyOn(localEmulator, "isLocalEmulatorEnabled").mockReturnValue(true); + const getLocalEmulatorFilePathSpy = vi.spyOn(localEmulator, "getLocalEmulatorFilePath").mockResolvedValue("/Users/foo/project/stack.config.ts"); + const writeConfigToFileSpy = vi.spyOn(localEmulator, "writeConfigToFile").mockResolvedValue(undefined); + const upsertSpy = vi.spyOn(globalPrismaClient.branchConfigOverride, "upsert").mockImplementation(() => { + throw new StackAssertionError("DB upsert should not run for local emulator branch config writes."); + }); + + try { + await setBranchConfigOverride({ + projectId: "project-id", + branchId: "branch-id", + branchConfigOverride: { + "teams.allowClientTeamCreation": true, + }, + }); + + expect(writeConfigToFileSpy).toHaveBeenCalledWith("/Users/foo/project/stack.config.ts", { + "teams.allowClientTeamCreation": true, + }); + expect(upsertSpy).not.toHaveBeenCalled(); + } finally { + upsertSpy.mockRestore(); + writeConfigToFileSpy.mockRestore(); + getLocalEmulatorFilePathSpy.mockRestore(); + isLocalEmulatorEnabledSpy.mockRestore(); + } +}); + +import.meta.vitest?.test('setBranchConfigOverride surfaces local emulator file write failures before touching the DB', async ({ expect }) => { + const vi = import.meta.vitest?.vi; + if (!vi) { + throw new StackAssertionError("Vitest context is required for in-source tests."); + } + + const localEmulator = await import("./local-emulator"); + const isLocalEmulatorEnabledSpy = vi.spyOn(localEmulator, "isLocalEmulatorEnabled").mockReturnValue(true); + const getLocalEmulatorFilePathSpy = vi.spyOn(localEmulator, "getLocalEmulatorFilePath").mockResolvedValue("/Users/foo/project/stack.config.ts"); + const writeConfigToFileSpy = vi.spyOn(localEmulator, "writeConfigToFile").mockRejectedValue(new Error("virtio-9p timeout")); + const upsertSpy = vi.spyOn(globalPrismaClient.branchConfigOverride, "upsert").mockImplementation(() => { + throw new StackAssertionError("DB upsert should not run when the local emulator file write fails."); + }); + + try { + await expect(setBranchConfigOverride({ + projectId: "project-id", + branchId: "branch-id", + branchConfigOverride: { + "teams.allowClientTeamCreation": true, + }, + })).rejects.toThrow("virtio-9p timeout"); + + expect(upsertSpy).not.toHaveBeenCalled(); + } finally { + upsertSpy.mockRestore(); + writeConfigToFileSpy.mockRestore(); + getLocalEmulatorFilePathSpy.mockRestore(); + isLocalEmulatorEnabledSpy.mockRestore(); + } +}); + // --------------------------------------------------------------------------------------------------------------------- // Conversions // --------------------------------------------------------------------------------------------------------------------- diff --git a/packages/stack-cli/src/commands/emulator.ts b/packages/stack-cli/src/commands/emulator.ts index 65b9a161a2..1bd74b80f4 100644 --- a/packages/stack-cli/src/commands/emulator.ts +++ b/packages/stack-cli/src/commands/emulator.ts @@ -1,5 +1,5 @@ import { Command } from "commander"; -import { execFileSync, execSync, spawn } from "child_process"; +import { execFileSync, spawn } from "child_process"; import { existsSync, mkdirSync, renameSync, unlinkSync } from "fs"; import { join, resolve } from "path"; import { CliError } from "../lib/errors.js"; @@ -103,10 +103,18 @@ export function registerEmulatorCommand(program: Command) { console.log(`Pulling image for ${arch} from release ${tag}...`); try { - execSync( - `gh release download ${JSON.stringify(tag)} --repo ${JSON.stringify(repo)} --pattern ${JSON.stringify(asset)} --output ${JSON.stringify(tmpDest)} --clobber`, - { stdio: "inherit" } - ); + execFileSync("gh", [ + "release", + "download", + tag, + "--repo", + repo, + "--pattern", + asset, + "--output", + tmpDest, + "--clobber", + ], { stdio: "inherit" }); } catch (err) { if (existsSync(tmpDest)) unlinkSync(tmpDest); const reason = err instanceof Error From 4be8a8299b6070b15a2e40bba99cad707246af54 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 16:41:53 -0700 Subject: [PATCH 18/30] Update QEMU emulator build workflow for improved performance and reliability - Changed the runner to `ubicloud-standard-8` and increased the timeout to 120 minutes for the build job, enhancing resource allocation. - Added a step to set up QEMU user-mode emulation, improving compatibility for different architectures. - Adjusted environment variables for the build and emulator run scripts to optimize execution time and reliability. These changes enhance the QEMU emulator build process and ensure better performance during image creation and testing. --- .github/workflows/qemu-emulator-build.yaml | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/.github/workflows/qemu-emulator-build.yaml b/.github/workflows/qemu-emulator-build.yaml index 07f65b5d8c..c12ad6d834 100644 --- a/.github/workflows/qemu-emulator-build.yaml +++ b/.github/workflows/qemu-emulator-build.yaml @@ -27,22 +27,23 @@ env: jobs: build: name: Build QEMU Image (${{ matrix.arch }}) - runs-on: ${{ matrix.runner }} - timeout-minutes: 60 + runs-on: ubicloud-standard-8 + timeout-minutes: 120 strategy: fail-fast: false matrix: include: - arch: amd64 - runner: ubicloud-standard-8 docker_platform: linux/amd64 - arch: arm64 - runner: ubicloud-standard-8-arm docker_platform: linux/arm64 steps: - uses: actions/checkout@v6 + - name: Set up QEMU user-mode emulation + uses: docker/setup-qemu-action@v3 + - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 @@ -63,13 +64,14 @@ jobs: - name: Build QEMU image run: | chmod +x docker/local-emulator/qemu/build-image.sh - docker/local-emulator/qemu/build-image.sh ${{ matrix.arch }} + EMULATOR_PROVISION_TIMEOUT=6000 \ + docker/local-emulator/qemu/build-image.sh ${{ matrix.arch }} - name: Start emulator and verify run: | chmod +x docker/local-emulator/qemu/run-emulator.sh EMULATOR_ARCH=${{ matrix.arch }} \ - EMULATOR_READY_TIMEOUT=300 \ + EMULATOR_READY_TIMEOUT=600 \ docker/local-emulator/qemu/run-emulator.sh start - name: Verify services are healthy From f3514f8fdd2490593e849efd9719022e2becc42e Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 17:49:05 -0700 Subject: [PATCH 19/30] Refactor QEMU emulator build and local emulator configuration - Updated the GitHub Actions workflow to streamline the QEMU emulator image build process, including improved handling of existing releases and enhanced error reporting. - Refactored local emulator configuration management to support reading from host-mounted paths, ensuring robust file handling and error reporting for configuration reads and writes. - Added tests to validate new configuration handling features, improving reliability for developers working with local emulator setups. These changes enhance the overall performance and usability of the QEMU emulator and local emulator configuration management. --- .github/workflows/qemu-emulator-build.yaml | 49 ++++---- apps/backend/src/lib/local-emulator.test.ts | 71 ++++++++--- apps/backend/src/lib/local-emulator.ts | 55 +++++--- claude/CLAUDE-KNOWLEDGE.md | 103 ++++++++++++++- docker/local-emulator/qemu/build-image.sh | 27 +++- .../qemu/cloud-init/emulator/user-data | 6 +- docker/local-emulator/qemu/run-emulator.sh | 2 +- packages/stack-cli/src/commands/emulator.ts | 118 ++++++++++-------- 8 files changed, 320 insertions(+), 111 deletions(-) diff --git a/.github/workflows/qemu-emulator-build.yaml b/.github/workflows/qemu-emulator-build.yaml index c12ad6d834..66b5651b56 100644 --- a/.github/workflows/qemu-emulator-build.yaml +++ b/.github/workflows/qemu-emulator-build.yaml @@ -34,9 +34,7 @@ jobs: matrix: include: - arch: amd64 - docker_platform: linux/amd64 - arch: arm64 - docker_platform: linux/arm64 steps: - uses: actions/checkout@v6 @@ -52,15 +50,6 @@ jobs: sudo apt-get update sudo apt-get install -y qemu-system-x86 qemu-system-arm qemu-utils genisoimage socat qemu-efi-aarch64 - - name: Build Docker emulator image - run: | - docker buildx build \ - --platform ${{ matrix.docker_platform }} \ - --tag ${{ env.EMULATOR_IMAGE_NAME }} \ - --load \ - -f docker/local-emulator/Dockerfile \ - . - - name: Build QEMU image run: | chmod +x docker/local-emulator/qemu/build-image.sh @@ -80,6 +69,7 @@ jobs: docker/local-emulator/qemu/run-emulator.sh status - name: Stop emulator + if: ${{ always() }} run: | EMULATOR_ARCH=${{ matrix.arch }} \ docker/local-emulator/qemu/run-emulator.sh stop @@ -127,18 +117,10 @@ jobs: cp "$f" release/ done - ls -lh release/ - - - name: Create GitHub Release - env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: | - gh release create "${{ env.RELEASE_TAG }}" \ - --title "QEMU Emulator — ${{ github.ref_name }} (${{ env.SHORT_SHA }})" \ - --notes "$(cat <<'EOF' + cat > release-notes.md <<'EOF' ## QEMU Emulator Images - Built from \`${{ github.ref_name }}\` @ \`${{ github.sha }}\` + Built from `${{ github.ref_name }}` @ `${{ github.sha }}` ### Images | File | Description | @@ -152,9 +134,28 @@ jobs: stack emulator run ``` EOF - )" \ - --prerelease \ - release/* + + ls -lh release/ + + - name: Create or update GitHub Release + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + TITLE="QEMU Emulator — ${{ github.ref_name }} ($SHORT_SHA)" + + if gh release view "$RELEASE_TAG" >/dev/null 2>&1; then + gh release edit "$RELEASE_TAG" \ + --title "$TITLE" \ + --notes-file release-notes.md \ + --prerelease + gh release upload "$RELEASE_TAG" release/* --clobber + else + gh release create "$RELEASE_TAG" \ + --title "$TITLE" \ + --notes-file release-notes.md \ + --prerelease \ + release/* + fi - name: Update latest tag for branch env: diff --git a/apps/backend/src/lib/local-emulator.test.ts b/apps/backend/src/lib/local-emulator.test.ts index aff12f2b28..e5519cae31 100644 --- a/apps/backend/src/lib/local-emulator.test.ts +++ b/apps/backend/src/lib/local-emulator.test.ts @@ -1,6 +1,12 @@ import fs from "fs/promises"; +import os from "os"; +import path from "path"; import { afterEach, describe, expect, it, vi } from "vitest"; -import { readConfigFromFile, writeConfigToFile } from "./local-emulator"; +import { + LOCAL_EMULATOR_HOST_MOUNT_ROOT_ENV, + readConfigFromFile, + writeConfigToFile, +} from "./local-emulator"; describe("local emulator config", () => { afterEach(() => { @@ -32,22 +38,55 @@ describe("local emulator config", () => { await expect(readConfigFromFile("/irrelevant/path/stack.config.ts")).resolves.toEqual({}); }); + it("throws when the config module does not export config", async () => { + const content = `export default { auth: { allowLocalhost: true } };\n`; + vi.stubEnv("STACK_LOCAL_EMULATOR_CONFIG_CONTENT", Buffer.from(content).toString("base64")); + + await expect(readConfigFromFile("/irrelevant/path/stack.config.ts")).rejects.toThrow( + "Invalid config in /irrelevant/path/stack.config.ts. The file must export a 'config' object." + ); + }); + + it("reads config files from the host mount when configured", async () => { + const hostMountRoot = await fs.mkdtemp(path.join(os.tmpdir(), "stack-host-mount-")); + const absoluteFilePath = "/Users/foo/project/stack.config.ts"; + const mountedFilePath = path.join(hostMountRoot, absoluteFilePath); + await fs.mkdir(path.dirname(mountedFilePath), { recursive: true }); + await fs.writeFile(mountedFilePath, `export const config = { auth: { allowLocalhost: true } };\n`, "utf-8"); + + vi.stubEnv(LOCAL_EMULATOR_HOST_MOUNT_ROOT_ENV, hostMountRoot); + + await expect(readConfigFromFile(absoluteFilePath)).resolves.toMatchInlineSnapshot(` + { + "auth": { + "allowLocalhost": true, + }, + } + `); + }); + it("writes new config files to the host mount when the mounted parent directory exists", async () => { - const accessSpy = vi.spyOn(fs, "access") - .mockRejectedValueOnce(Object.assign(new Error("missing file"), { code: "ENOENT" })) - .mockResolvedValueOnce(undefined); - const mkdirSpy = vi.spyOn(fs, "mkdir").mockResolvedValueOnce("/host/Users/foo/project"); - const writeFileSpy = vi.spyOn(fs, "writeFile").mockResolvedValueOnce(); - - await writeConfigToFile("/Users/foo/project/stack.config.ts", { auth: { allowLocalhost: true } }); - - expect(accessSpy).toHaveBeenNthCalledWith(1, "/host/Users/foo/project/stack.config.ts"); - expect(accessSpy).toHaveBeenNthCalledWith(2, "/host/Users/foo/project"); - expect(mkdirSpy).toHaveBeenCalledWith("/host/Users/foo/project", { recursive: true }); - expect(writeFileSpy).toHaveBeenCalledWith( - "/host/Users/foo/project/stack.config.ts", - `export const config = {\n "auth": {\n "allowLocalhost": true\n }\n};\n`, - "utf-8", + const hostMountRoot = await fs.mkdtemp(path.join(os.tmpdir(), "stack-host-mount-")); + const absoluteFilePath = "/Users/foo/project/stack.config.ts"; + const mountedParentPath = path.join(hostMountRoot, "/Users/foo/project"); + const mountedFilePath = path.join(hostMountRoot, absoluteFilePath); + await fs.mkdir(mountedParentPath, { recursive: true }); + + vi.stubEnv(LOCAL_EMULATOR_HOST_MOUNT_ROOT_ENV, hostMountRoot); + + await writeConfigToFile(absoluteFilePath, { auth: { allowLocalhost: true } }); + + await expect(fs.readFile(mountedFilePath, "utf-8")).resolves.toBe( + `export const config = {\n "auth": {\n "allowLocalhost": true\n }\n};\n` + ); + }); + + it("fails loudly when the QEMU host mount root is configured but unavailable", async () => { + const hostMountRoot = await fs.mkdtemp(path.join(os.tmpdir(), "stack-host-mount-")); + vi.stubEnv(LOCAL_EMULATOR_HOST_MOUNT_ROOT_ENV, hostMountRoot); + + await expect(writeConfigToFile("/Users/foo/project/stack.config.ts", { auth: { allowLocalhost: true } })).rejects.toThrow( + `Local emulator host mount root ${hostMountRoot} is configured` ); }); }); diff --git a/apps/backend/src/lib/local-emulator.ts b/apps/backend/src/lib/local-emulator.ts index dd2e46dd84..2591460a53 100644 --- a/apps/backend/src/lib/local-emulator.ts +++ b/apps/backend/src/lib/local-emulator.ts @@ -3,7 +3,7 @@ import path from "path"; import { createJiti } from "jiti"; import { isValidConfig } from "@stackframe/stack-shared/dist/config/format"; import { getEnvVariable } from "@stackframe/stack-shared/dist/utils/env"; -import { StatusError } from "@stackframe/stack-shared/dist/utils/errors"; +import { StatusError, throwErr } from "@stackframe/stack-shared/dist/utils/errors"; import { globalPrismaClient } from "@/prisma-client"; export const LOCAL_EMULATOR_ADMIN_USER_ID = "63abbc96-5329-454a-ba56-e0460173c6c1"; @@ -15,11 +15,16 @@ export const LOCAL_EMULATOR_ENV_CONFIG_BLOCKED_MESSAGE = "Environment configuration overrides cannot be changed in the local emulator. Update this in your production deployment instead."; export const LOCAL_EMULATOR_ONLY_ENDPOINT_MESSAGE = "This endpoint is only available in local emulator mode (set NEXT_PUBLIC_STACK_IS_LOCAL_EMULATOR=true)."; +export const LOCAL_EMULATOR_HOST_MOUNT_ROOT_ENV = "STACK_LOCAL_EMULATOR_HOST_MOUNT_ROOT"; export function isLocalEmulatorEnabled() { return getEnvVariable("NEXT_PUBLIC_STACK_IS_LOCAL_EMULATOR", "") === "true"; } +function isErrnoException(error: unknown): error is NodeJS.ErrnoException { + return typeof error === "object" && error !== null && "code" in error; +} + export async function isLocalEmulatorProject(projectId: string) { if (!isLocalEmulatorEnabled()) { return false; @@ -36,27 +41,43 @@ export async function isLocalEmulatorProject(projectId: string) { return project !== null; } +async function pathExists(filePath: string): Promise { + try { + await fs.access(filePath); + return true; + } catch (error) { + if (isErrnoException(error) && error.code === "ENOENT") { + return false; + } + throw error; + } +} + /** * Resolves the file path for config files in the local emulator. * - * In the QEMU emulator, the host filesystem is mounted at /host via virtio-9p. - * The DB stores absolute host paths (e.g. /Users/foo/project/stack.config.ts), so we - * try /host/ first, then fall back to the original path for non-QEMU environments - * (e.g. Docker Compose where the path is directly accessible). + * In the QEMU emulator, the host filesystem is mounted separately at /host, outside the + * guest qcow2 overlay. The DB stores absolute host paths (for example + * /Users/foo/project/stack.config.ts), so we map them to /host/ when the host mount + * is configured. We fail loudly if the host mount root is configured but inaccessible, + * because silently writing to a guest-local lookalike path would desync the dashboard from + * the user's real stack.config.ts file. */ async function resolveConfigFilePath(filePath: string): Promise { - const hostMountedPath = path.join("/host", filePath); - try { - await fs.access(hostMountedPath); - return hostMountedPath; - } catch { - try { - await fs.access(path.dirname(hostMountedPath)); + const hostMountRoot = getEnvVariable(LOCAL_EMULATOR_HOST_MOUNT_ROOT_ENV, ""); + if (hostMountRoot !== "") { + const hostMountedPath = path.join(hostMountRoot, filePath); + if (await pathExists(hostMountedPath) || await pathExists(path.dirname(hostMountedPath))) { return hostMountedPath; - } catch { - return filePath; } + + throw new Error( + `Local emulator host mount root ${hostMountRoot} is configured, but ${hostMountedPath} is not accessible. ` + + "Restart the QEMU emulator so the host share is mounted, or choose a config path under the shared host root." + ); } + + return filePath; } export async function readConfigFromFile(filePath: string): Promise> { @@ -74,8 +95,10 @@ export async function readConfigFromFile(filePath: string): Promise; - const config = mod.config; + const mod = jiti.evalModule(content, { filename: resolvedPath }); + const config = typeof mod === "object" && mod !== null && "config" in mod + ? mod.config + : throwErr(`Invalid config in ${filePath}. The file must export a 'config' object.`); if (!isValidConfig(config)) { throw new StatusError(StatusError.BadRequest, `Invalid config in ${filePath}. The file must export a 'config' object.`); } diff --git a/claude/CLAUDE-KNOWLEDGE.md b/claude/CLAUDE-KNOWLEDGE.md index fbefcdf2d2..4d5b54ba9e 100644 --- a/claude/CLAUDE-KNOWLEDGE.md +++ b/claude/CLAUDE-KNOWLEDGE.md @@ -1,4 +1,101 @@ -# CLAUDE-KNOWLEDGE +# CLAUDE Knowledge Base -## Q: How should QEMU local-emulator containers reach host-only dev services like the OAuth mock server? -A: Keep container-local dependencies on `127.0.0.1`, but point host-only services at QEMU's user-network host alias `10.0.2.2`. Using `127.0.0.1:${PORT}` for host services from inside the guest container points back at the container itself. +Q: How are the development ports derived now that NEXT_PUBLIC_STACK_PORT_PREFIX exists? +A: Host ports use `${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}` plus the two-digit suffix (e.g., Postgres is `${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28`, Inbucket SMTP `${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}29`, POP3 `${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}30`, and OTLP `${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}31` by default). + +Q: How can I show helper text beneath metadata text areas in the dashboard? +A: Use the shared `TextAreaField` component's `helperText` prop in `apps/dashboard/src/components/form-fields.tsx`; it now renders the helper content in a secondary Typography line under the textarea. + +Q: How is the Email Template Editor structured? +A: It uses a hero-preview layout in `VibeCodeLayout` where the preview area dominates the screen. The code editor is hidden by default and accessible via a modal, while the AI assistant chat resides in a resizable right panel. Device viewport switching (Desktop/Tablet/Mobile) is integrated into the top toolbar. + +Q: How can I improve AI design generations for emails? +A: Update the system prompts in the backend's chat adapters (e.g., `apps/backend/src/lib/ai-chat/email-template-adapter.ts`). Providing explicit design principles, Tailwind CSS best practices, and structured technical rules helps the AI generate more polished and consistent designs. + +Q: What endpoint does the local Freestyle mock expose for script execution? +A: The mock server responds on `/execute/v1/script` and `/execute/v2/script` when built from `docker/dependencies/freestyle-mock/Dockerfile`; if the running image is older and only supports v1, backend dev can post to `/execute/v1/script` for email rendering. + +Q: How can I add a small Vitest check inside a client-only file? +A: Use `import.meta.vitest?.test(...)` at the bottom of the file for lightweight, in-file tests without adding a separate test file. +Q: Why did `pnpm typecheck` fail after deleting a Next.js route? +A: The generated `.next/types/validator.ts` can keep stale imports for removed routes. Deleting that file (or regenerating Next build output) clears the outdated references so `pnpm typecheck` succeeds again. + +Q: Why can auto-migrations time out and how should I mitigate it? +A: Auto-migrations run each migration inside a Prisma interactive transaction with an 80s timeout. Long-running statements (even if marked RUN_OUTSIDE_TRANSACTION_SENTINEL) still consume that time, so keep each iteration small using CONDITIONALLY_REPEAT_MIGRATION_SENTINEL and reduce batch sizes (e.g., lower LIMIT) so each transaction finishes under 80s. + +Q: How should `restricted_by_admin` updates handle reason fields? +A: When setting `restricted_by_admin` to false, explicitly clear `restricted_by_admin_reason` and `restricted_by_admin_private_details` to null (even if omitted in the PATCH) to satisfy the database constraint. + +Q: Where should `stackAppInternalsSymbol` be imported from in the dashboard? +A: Use the shared `apps/dashboard/src/lib/stack-app-internals.ts` export to avoid duplicating the Symbol.for definition across files. + +Q: How do we control whether a project requires publishable client keys? +A: Use the project-level config override field `project.requirePublishableClientKey` via `/api/v1/internal/config/override/project` or `AdminProject.update({ requirePublishableClientKey: ... })`. It defaults to false for new projects and is set true for existing projects via DB migration. + +Q: When adding new config fields, what else should be updated? +A: Update the config schema fuzzer configs in `packages/stack-shared/src/config/schema-fuzzer.test.ts` (for example, add the new field under `projectSchemaFuzzerConfig`/`branchSchemaFuzzerConfig`). + +Q: Why can't `canNoLongerBeOverridden` accept dotted paths? +A: It uses `schema.getNested`, which only allows keys with alphanumerics, `_`, `$`, `:`, or `-`. Dots are rejected, so mark the parent object key (e.g., `project`) as non-overridable instead. + +Q: Where is the editable-grid preview spacing controlled in the dashboard playground? +A: In `apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/playground/page-client.tsx`, the `selected === "editable-grid"` branch controls the card width/padding and the main preview container now uses `isExpandedPreview` to reduce outer gray padding only for editable-grid. + +Q: Why do editable-grid dropdown/boolean values sometimes not fill the full value column width? +A: In `apps/dashboard/src/components/design-components/editable-grid.tsx`, the value wrappers must be explicitly full-width (`w-full`) for boolean and dropdown fields, and the grid value cell container should also include `w-full`; otherwise controls shrink to content width. + +Q: How should dashboard inline editable text fields match the new design-components style? +A: Use `DesignInput` and `DesignButton` in `apps/dashboard/src/components/editable-input.tsx` (instead of legacy `Input`/`Button`) and style accept/reject actions as subtle glassy icon buttons with muted ring/border plus semantic hover tints. + +Q: What should dashboard email/project pages prefer for UI primitives? +A: Prefer `apps/dashboard/src/components/design-components/*` components (`DesignCard`, `DesignAlert`, `DesignBadge`, `DesignButton`, `DesignPillToggle`, `DesignCategoryTabs`, etc.) over page-local wrappers or repeated inline class patterns; current email surfaces still contain local patterns like custom GlassCard/SectionHeader/ViewportSelector that should be standardized. + +Q: What sections are expected in the dashboard design guide beyond component mapping? +A: Include explicit best-practices plus dedicated guidance for animation, typography, light/dark color system, micro-interactions, and spacing/layout rules so the guide is actionable for both humans and AI agents. + +Q: How should the project emails page cards align with the design system? +A: In `apps/dashboard/src/app/(main)/(protected)/projects/[projectId]/emails/page-client.tsx`, wrap the major sections with `DesignCard` from `@/components/design-components` (for example `gradient="default"`/`"purple"` with `glassmorphic`) instead of maintaining a local page-specific glass card wrapper. + +Q: Where is the default inner spacing for shared design cards controlled? +A: `apps/dashboard/src/components/design-components/card.tsx` sets the default content padding via `bodyPaddingClass` (currently `p-5`), and compact cards use `p-5` header plus `px-5 py-4` body spacing. + +Q: Why can two `DesignCard` surfaces look like they have different padding? +A: Pages can add extra local wrappers inside `DesignCard` (for example `p-5`, `px-5`, `pb-5`) which stack on top of `DesignCard` defaults; in the emails page, removing those local wrappers (`p-0`, `px-0`, `pb-0`) makes spacing match playground behavior. + +Q: How can a split section inside body-only `DesignCard` match header/content card borders? +A: Inside body-only cards (which already apply `p-5`), use a second section with `-mx-5 px-5` and `border-t border-black/[0.12] dark:border-white/[0.06]` so the divider spans full card width while content alignment matches `DesignCard` header/content layout. + +Q: How should cards handle header action buttons when using `title` + `subtitle`? +A: `DesignCard` now supports an `actions` prop when title/icon are provided; use `title`, `subtitle`, `icon`, and `actions` in pages like emails so header spacing and subtitle-bottom spacing exactly match playground/header variant styles without custom section-header workarounds. + +Q: What should we do after changing props in a core dashboard design component? +A: Update the playground implementation (`apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/playground/page-client.tsx`) in the same change so the component controls/examples reflect the new or changed props immediately. + +Q: How is the new `DesignCard` `actions` prop represented in playground? +A: The card playground now includes a `Header Actions` toggle that injects a sample `actions` slot (`DesignButton` with `Sliders` icon and "Configure") into `DesignCard` preview and generated code, only when `title` is present. + +Q: What is the reliable way to lint a single dashboard file in this monorepo? +A: Run lint from `apps/dashboard` directly (for example `pnpm lint -- "src/app/(main)/(protected)/projects/[projectId]/(overview)/line-chart.tsx"`), because running root `pnpm lint -- ` fans out through Turbo packages where that path does not exist. +Q: How should unsubscribe-link e2e tests avoid breakage from email theme/layout changes? +A: In `apps/e2e/tests/backend/endpoints/api/v1/unsubscribe-link.test.ts`, avoid snapshotting the entire rendered HTML for transactional emails; assert stable behavior instead (email content present and `/api/v1/emails/unsubscribe-link` absent) so cosmetic wrapper/style changes do not fail the test. + +Q: How should dashboard pages update project config values? +A: Do not call `project.updateConfig(...)` directly from dashboard pages; lint enforces using `useUpdateConfig()` from `apps/dashboard/src/lib/config-update.tsx` so pushable-config confirmation flows are handled consistently. + +Q: How can the dashboard find resumable onboarding state without SDK type changes? +A: Query `/internal/projects` via `stackAppInternalsSymbol` and read each project's `onboarding_status`; this avoids relying on `AdminOwnedProject` fields that may lag until generated package copies are rebuilt. + +Q: How should the new-project onboarding page avoid React's "Cannot update a component while rendering a different component" router error? +A: In `apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/new-project/page-client.tsx`, never call `router.replace(...)` during render when an onboarding project is already completed; move that redirect into a `useEffect` and render a plain spinner while the redirect is in progress. + +Q: What is the expected lightweight loading state when reopening an in-progress onboarding project? +A: On `apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/new-project/page-client.tsx`, the "loading onboarding" state should be just a centered `Spinner` with no card chrome or explanatory copy. + +Q: How should dashboard project onboarding status responses be handled to avoid silently bypassing onboarding? +A: Import `ProjectOnboardingStatus`/`projectOnboardingStatusValues` from `@stackframe/stack-shared/dist/schema-fields`, validate every `onboarding_status` from `/internal/projects`, and throw on invalid/missing values instead of defaulting to `"completed"`. + +Q: What E2E updates are required after adding `onboarding_status` to project API responses? +A: Update affected inline snapshots in `apps/e2e/tests/backend/endpoints/api/v1/**` to include `"onboarding_status": "completed"` in project payloads (for example projects, permissions, and integration provisioning/current endpoints), otherwise CI setup/restart E2E jobs fail with snapshot mismatches. + +Q: How should `createOrUpdateProjectWithLegacyConfig` handle `onboardingStatus` for forward-compat checks? +A: Only write `onboardingStatus` when the `Project.onboardingStatus` column exists (for example by checking `information_schema.columns` in-transaction) so current code can still run against older schemas where that column is absent. diff --git a/docker/local-emulator/qemu/build-image.sh b/docker/local-emulator/qemu/build-image.sh index 0e8fb70116..8071fb5012 100755 --- a/docker/local-emulator/qemu/build-image.sh +++ b/docker/local-emulator/qemu/build-image.sh @@ -7,12 +7,14 @@ source "$SCRIPT_DIR/common.sh" IMAGE_DIR="$SCRIPT_DIR/images" CLOUD_INIT_ROOT="$SCRIPT_DIR/cloud-init" +REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)" DEBIAN_VERSION="${DEBIAN_VERSION:-13}" DISK_SIZE="${EMULATOR_DISK_SIZE:-12G}" RAM="${EMULATOR_BUILD_RAM:-4096}" CPUS="${EMULATOR_BUILD_CPUS:-$(sysctl -n hw.ncpu 2>/dev/null || nproc 2>/dev/null || echo 4)}" PROVISION_TIMEOUT="${EMULATOR_PROVISION_TIMEOUT:-3200}" +EMULATOR_IMAGE_NAME="${EMULATOR_IMAGE_NAME:-stack-local-emulator}" RED='\033[0;31m' GREEN='\033[0;32m' @@ -34,7 +36,7 @@ case "$TARGET_ARCH" in *) err "Usage: $0 [arm64|amd64|both]"; exit 1 ;; esac -DOCKER_IMAGES=(stack-local-emulator) +DOCKER_IMAGES=("$EMULATOR_IMAGE_NAME") check_deps() { local missing=() @@ -83,6 +85,28 @@ download_cloud_image() { curl -fSL --progress-bar -o "$dest" "$url" } +docker_platform_for_arch() { + case "$1" in + arm64) echo "linux/arm64" ;; + amd64) echo "linux/amd64" ;; + *) err "Unsupported target arch: $1"; exit 1 ;; + esac +} + +build_local_emulator_image() { + local arch="$1" + local platform + platform="$(docker_platform_for_arch "$arch")" + + log "Building Docker emulator image (${arch})..." + docker buildx build \ + --platform "$platform" \ + --tag "$EMULATOR_IMAGE_NAME" \ + --load \ + -f "$REPO_ROOT/docker/local-emulator/Dockerfile" \ + "$REPO_ROOT" +} + qemu_cmd_prefix_for_arch() { local arch="$1" case "$arch" in @@ -258,6 +282,7 @@ build_one() { for arch in "${TARGET_ARCHS[@]}"; do local_base="$IMAGE_DIR/debian-${DEBIAN_VERSION}-base-${arch}.qcow2" download_cloud_image "$arch" "$local_base" + build_local_emulator_image "$arch" prepare_bundle_artifacts "$arch" build_one "$arch" done diff --git a/docker/local-emulator/qemu/cloud-init/emulator/user-data b/docker/local-emulator/qemu/cloud-init/emulator/user-data index 3a45e03e3b..39b8c33cdb 100644 --- a/docker/local-emulator/qemu/cloud-init/emulator/user-data +++ b/docker/local-emulator/qemu/cloud-init/emulator/user-data @@ -74,6 +74,7 @@ write_files: cat </dev/null || true + if ! mount -t 9p -o trans=virtio,version=9p2000.L hostfs /host; then + echo "Failed to mount host filesystem at /host" >&2 + exit 1 + fi fi - path: /usr/local/bin/run-stack-container diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index 24461976bf..8b18fbb1ba 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -313,7 +313,7 @@ cmd_start() { exit 1 fi - log "All services are green. Overlay disk preserves changes across restarts." + log "All services are green. The qcow2 overlay preserves emulator state across restarts, while /host stays a live host share outside the VM disk." } cmd_stop() { diff --git a/packages/stack-cli/src/commands/emulator.ts b/packages/stack-cli/src/commands/emulator.ts index 1bd74b80f4..ffda8b0105 100644 --- a/packages/stack-cli/src/commands/emulator.ts +++ b/packages/stack-cli/src/commands/emulator.ts @@ -8,6 +8,7 @@ const DEFAULT_REPO = "stack-auth/stack-auth"; const DEFAULT_BRANCH = "dev"; type EmulatorArch = "arm64" | "amd64"; +type BuildTargetArch = EmulatorArch | "both"; function detectArch(): EmulatorArch { switch (process.arch) { @@ -40,24 +41,28 @@ function findQemuDir(): string { ); } -function runScript(qemuDir: string, script: string, args: string[], env?: Record): Promise { +function runCommand(cwd: string, command: string, args: string[], env?: Record): Promise { return new Promise((resolve, reject) => { - const child = spawn(join(qemuDir, script), args, { + const child = spawn(command, args, { stdio: "inherit", env: { ...process.env, ...env }, - cwd: qemuDir, + cwd, }); child.on("close", (code) => { if (code === 0) resolve(); - else reject(new CliError(`${script} exited with code ${code}`)); + else reject(new CliError(`${command} exited with code ${code}`)); }); child.on("error", (err) => { - reject(new CliError(`Failed to run ${script}: ${err.message}`)); + reject(new CliError(`Failed to run ${command}: ${err.message}`)); }); }); } +function runScript(qemuDir: string, script: string, args: string[], env?: Record): Promise { + return runCommand(qemuDir, join(qemuDir, script), args, env); +} + function runEmulatorAction(action: string, env?: Record): Promise { return runScript(findQemuDir(), "run-emulator.sh", [action], env); } @@ -66,14 +71,63 @@ function ghRelease(args: string[]): string { try { return execFileSync("gh", args, { encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"] }).trim(); } catch (err: unknown) { - if (err instanceof Error && "stderr" in err) { - const { stderr } = err as Error & { stderr: string }; - throw new CliError(`GitHub CLI error: ${stderr}`); + if (err instanceof Error && "stderr" in err && typeof err.stderr === "string") { + throw new CliError(`GitHub CLI error: ${err.stderr}`); } throw new CliError("GitHub CLI (gh) is required. Install: https://cli.github.com/"); } } +function parseBuildArch(arch: string | undefined): BuildTargetArch { + const resolvedArch = arch ?? detectArch(); + if (resolvedArch === "arm64" || resolvedArch === "amd64" || resolvedArch === "both") { + return resolvedArch; + } + throw new CliError(`Unsupported build architecture: ${resolvedArch}. Use arm64, amd64, or both.`); +} + +async function pullImage(arch: EmulatorArch, opts: { repo?: string; branch?: string; tag?: string } = {}) { + const repo = opts.repo ?? DEFAULT_REPO; + const branch = opts.branch ?? DEFAULT_BRANCH; + const tag = opts.tag ?? `emulator-${branch}-latest`; + const asset = `stack-emulator-${arch}.qcow2`; + + const qemuDir = findQemuDir(); + const imageDir = join(qemuDir, "images"); + mkdirSync(imageDir, { recursive: true }); + + const dest = join(imageDir, asset); + const tmpDest = `${dest}.download`; + + console.log(`Pulling image for ${arch} from release ${tag}...`); + + try { + execFileSync("gh", [ + "release", + "download", + tag, + "--repo", + repo, + "--pattern", + asset, + "--output", + tmpDest, + "--clobber", + ], { stdio: "inherit" }); + } catch (err) { + if (existsSync(tmpDest)) unlinkSync(tmpDest); + const reason = err instanceof Error + ? (err.stack ?? err.message) + : String(err); + throw new CliError( + `Failed to download ${asset} from release ${tag}: ${reason}\nRun 'stack emulator list-releases' to see available releases.` + ); + } + + renameSync(tmpDest, dest); + console.log(`Downloaded: ${dest}`); +} + export function registerEmulatorCommand(program: Command) { const emulator = program .command("emulator") @@ -88,45 +142,11 @@ export function registerEmulatorCommand(program: Command) { .option("--repo ", `GitHub repository (default: ${DEFAULT_REPO})`) .action(async (opts) => { const arch: EmulatorArch = opts.arch || detectArch(); - const repo = opts.repo || DEFAULT_REPO; - const branch = opts.branch || DEFAULT_BRANCH; - const tag = opts.tag || `emulator-${branch}-latest`; - const asset = `stack-emulator-${arch}.qcow2`; - - const qemuDir = findQemuDir(); - const imageDir = join(qemuDir, "images"); - mkdirSync(imageDir, { recursive: true }); - - const dest = join(imageDir, asset); - const tmpDest = `${dest}.download`; - - console.log(`Pulling image for ${arch} from release ${tag}...`); - - try { - execFileSync("gh", [ - "release", - "download", - tag, - "--repo", - repo, - "--pattern", - asset, - "--output", - tmpDest, - "--clobber", - ], { stdio: "inherit" }); - } catch (err) { - if (existsSync(tmpDest)) unlinkSync(tmpDest); - const reason = err instanceof Error - ? (err.stack ?? err.message) - : String(err); - throw new CliError( - `Failed to download ${asset} from release ${tag}: ${reason}\nRun 'stack emulator list-releases' to see available releases.` - ); - } - - renameSync(tmpDest, dest); - console.log(`Downloaded: ${dest}`); + await pullImage(arch, { + repo: opts.repo, + branch: opts.branch, + tag: opts.tag, + }); }); emulator @@ -140,7 +160,7 @@ export function registerEmulatorCommand(program: Command) { if (!existsSync(img)) { console.log("No emulator image found. Pulling latest..."); - await program.parseAsync(["node", "stack", "emulator", "pull", "--arch", arch], { from: "user" }); + await pullImage(arch); } await runScript(qemuDir, "run-emulator.sh", ["start"], { EMULATOR_ARCH: arch }); @@ -166,7 +186,7 @@ export function registerEmulatorCommand(program: Command) { .description("Build the QEMU emulator image locally") .option("--arch ", "Target architecture (arm64, amd64, or both)") .action(async (opts) => { - const arch = opts.arch || detectArch(); + const arch = parseBuildArch(opts.arch); await runScript(findQemuDir(), "build-image.sh", [arch]); }); From d5d2a0916d46f9c5eab2be9fcf3cd415f5a420cb Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 17:52:41 -0700 Subject: [PATCH 20/30] Increase EMULATOR_READY_TIMEOUT in QEMU emulator build workflow - Updated the EMULATOR_READY_TIMEOUT variable from 600 to 3200 seconds in the GitHub Actions workflow for the QEMU emulator build process. - This change aims to provide additional time for the emulator to become ready, enhancing reliability during the build and testing phases. These adjustments improve the overall performance and stability of the QEMU emulator setup. --- .github/workflows/qemu-emulator-build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/qemu-emulator-build.yaml b/.github/workflows/qemu-emulator-build.yaml index 66b5651b56..52b7ba5415 100644 --- a/.github/workflows/qemu-emulator-build.yaml +++ b/.github/workflows/qemu-emulator-build.yaml @@ -60,7 +60,7 @@ jobs: run: | chmod +x docker/local-emulator/qemu/run-emulator.sh EMULATOR_ARCH=${{ matrix.arch }} \ - EMULATOR_READY_TIMEOUT=600 \ + EMULATOR_READY_TIMEOUT=3200 \ docker/local-emulator/qemu/run-emulator.sh start - name: Verify services are healthy From e5099471e48daa9f083cf5f685dcde9b1f4c2c5c Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 18:05:12 -0700 Subject: [PATCH 21/30] Enhance local emulator setup with environment generation and configuration updates - Introduced a new script to generate the local emulator environment file (`.env.development`) based on existing backend and dashboard configurations, ensuring consistency and ease of use. - Updated `docker-compose.yaml` to reference the new environment file, improving configuration management for the local emulator. - Modified emulator startup scripts to include environment generation, streamlining the process for developers. - Added a new environment file for local development, containing necessary credentials and settings for the emulator. These changes improve the local emulator's usability and configuration management, facilitating a smoother development experience. --- .../{base.env => .env.development} | 14 ++ docker/local-emulator/docker-compose.yaml | 2 +- .../generate-env-development.mjs | 203 ++++++++++++++++++ .../qemu/cloud-init/emulator/user-data | 4 +- docker/local-emulator/qemu/run-emulator.sh | 2 +- package.json | 6 +- 6 files changed, 225 insertions(+), 6 deletions(-) rename docker/local-emulator/{base.env => .env.development} (91%) create mode 100644 docker/local-emulator/generate-env-development.mjs diff --git a/docker/local-emulator/base.env b/docker/local-emulator/.env.development similarity index 91% rename from docker/local-emulator/base.env rename to docker/local-emulator/.env.development index 8d356cfc9a..d04860f669 100644 --- a/docker/local-emulator/base.env +++ b/docker/local-emulator/.env.development @@ -1,3 +1,7 @@ +# Generated by docker/local-emulator/generate-env-development.mjs +# Do not edit manually; update apps/backend/.env.development, apps/dashboard/.env.development, or this generator. + +# Public emulator/app credentials NEXT_PUBLIC_STACK_DOCS_BASE_URL=https://docs.stack-auth.com NEXT_PUBLIC_STACK_IS_LOCAL_EMULATOR=true NEXT_PUBLIC_STACK_PROJECT_ID=internal @@ -5,6 +9,8 @@ NEXT_PUBLIC_STACK_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-loca STACK_SECRET_SERVER_KEY=this-secret-server-key-is-for-local-development-only STACK_SERVER_SECRET=23-wuNpik0gIW4mruTz25rbIvhuuvZFrLOLtL7J4tyo STACK_CHANGELOG_URL=https://raw.githubusercontent.com/stack-auth/stack-auth/refs/heads/dev/CHANGELOG.md + +# Seed/project defaults STACK_SEED_ENABLE_DUMMY_PROJECT=true STACK_SEED_INTERNAL_PROJECT_SIGN_UP_ENABLED=true STACK_SEED_INTERNAL_PROJECT_OTP_ENABLED=true @@ -14,6 +20,8 @@ STACK_SEED_INTERNAL_PROJECT_USER_INTERNAL_ACCESS=true STACK_SEED_INTERNAL_PROJECT_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only STACK_SEED_INTERNAL_PROJECT_SECRET_SERVER_KEY=this-secret-server-key-is-for-local-development-only STACK_SEED_INTERNAL_PROJECT_SUPER_SECRET_ADMIN_KEY=this-super-secret-admin-key-is-for-local-development-only + +# Third-party/test integrations STACK_SVIX_API_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NTUxNDA2MzksImV4cCI6MTk3MDUwMDYzOSwibmJmIjoxNjU1MTQwNjM5LCJpc3MiOiJzdml4LXNlcnZlciIsInN1YiI6Im9yZ18yM3JiOFlkR3FNVDBxSXpwZ0d3ZFhmSGlyTXUifQ.En8w77ZJWbd0qrMlHHupHUB-4cx17RfzFykseg95SUk STACK_OPENAI_API_KEY=mock_openai_api_key STACK_OPENROUTER_API_KEY=FORWARD_TO_PRODUCTION @@ -27,6 +35,8 @@ STACK_DNSIMPLE_API_BASE_URL=https://api.dnsimple.com/v2 STACK_FREESTYLE_API_KEY=mock_stack_freestyle_key STACK_VERCEL_SANDBOX_TOKEN=vercel_sandbox_disabled_for_local_development CRON_SECRET=mock_cron_secret + +# Storage, queueing, and analytics STACK_S3_REGION=us-east-1 STACK_S3_ACCESS_KEY_ID=s3mockroot STACK_S3_SECRET_ACCESS_KEY=s3mockroot @@ -41,6 +51,8 @@ STACK_QSTASH_NEXT_SIGNING_KEY=sig_5ZB6DVzB1wjE8S6rZ7eenA8Pdnhs STACK_CLICKHOUSE_ADMIN_USER=stackframe STACK_CLICKHOUSE_ADMIN_PASSWORD=PASSWORD-PLACEHOLDER--9gKyMxJeMx STACK_CLICKHOUSE_EXTERNAL_PASSWORD=PASSWORD-PLACEHOLDER--EZeHscBMzE + +# Email and dashboard integration STACK_EMAIL_PORT=2500 STACK_EMAIL_SECURE=false STACK_EMAIL_USERNAME=does-not-matter @@ -54,6 +66,8 @@ STACK_EMAIL_MONITOR_RESEND_EMAIL_API_KEY=this-is-a-fake-key STACK_EMAIL_MONITOR_SECRET_TOKEN=this-secret-token-is-for-local-development-only STACK_EMAIL_MONITOR_USE_INBUCKET=true STACK_FEATUREBASE_JWT_SECRET=secret-value + +# Mock OAuth defaults STACK_FORWARD_MOCK_OAUTH_SERVER=false STACK_GITHUB_CLIENT_ID=MOCK STACK_GITHUB_CLIENT_SECRET=MOCK diff --git a/docker/local-emulator/docker-compose.yaml b/docker/local-emulator/docker-compose.yaml index 0a434c5271..4a8bca8966 100644 --- a/docker/local-emulator/docker-compose.yaml +++ b/docker/local-emulator/docker-compose.yaml @@ -27,7 +27,7 @@ services: - inbucket-data:/data/inbucket - "${HOME}:${HOME}" - "/tmp:/tmp" - env_file: ./base.env + env_file: ./.env.development environment: # Port-prefixed URLs — need shell interpolation, can't go in env_file NEXT_PUBLIC_STACK_PORT_PREFIX: "${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}" diff --git a/docker/local-emulator/generate-env-development.mjs b/docker/local-emulator/generate-env-development.mjs new file mode 100644 index 0000000000..5c9cfb7536 --- /dev/null +++ b/docker/local-emulator/generate-env-development.mjs @@ -0,0 +1,203 @@ +#!/usr/bin/env node + +import fs from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const scriptDir = path.dirname(fileURLToPath(import.meta.url)); +const rootDir = path.resolve(scriptDir, "..", ".."); + +const outputPath = path.join(scriptDir, ".env.development"); +const backendEnvPath = path.join(rootDir, "apps", "backend", ".env.development"); +const dashboardEnvPath = path.join(rootDir, "apps", "dashboard", ".env.development"); + +const args = process.argv.slice(2); +if (args.length > 1 || (args[0] != null && args[0] !== "--check")) { + throw new Error("Usage: node docker/local-emulator/generate-env-development.mjs [--check]"); +} + +const parseEnvFile = (filePath) => { + const env = new Map(); + + for (const rawLine of fs.readFileSync(filePath, "utf8").split(/\r?\n/)) { + const trimmedLine = rawLine.trim(); + if (trimmedLine === "" || trimmedLine.startsWith("#")) { + continue; + } + + const separatorIndex = rawLine.indexOf("="); + if (separatorIndex < 0) { + throw new Error(`Invalid env line in ${filePath}: ${rawLine}`); + } + + const key = rawLine.slice(0, separatorIndex).trim(); + const value = rawLine.slice(separatorIndex + 1); + env.set(key, value); + } + + return env; +}; + +const backendEnv = parseEnvFile(backendEnvPath); +const dashboardEnv = parseEnvFile(dashboardEnvPath); + +const getRequiredEnvValue = (sourceName, envMap, key) => { + const value = envMap.get(key); + if (value == null) { + throw new Error(`Missing ${key} in ${sourceName}; update the generator or source env file.`); + } + return value; +}; + +const fromSource = (sourceName, envMap, key) => ({ + type: "entry", + key, + value: getRequiredEnvValue(sourceName, envMap, key), +}); + +const literal = (key, value) => ({ + type: "entry", + key, + value, +}); + +const comment = (value) => ({ + type: "comment", + value, +}); + +const blank = () => ({ + type: "blank", +}); + +const entries = [ + comment("# Generated by docker/local-emulator/generate-env-development.mjs"), + comment("# Do not edit manually; update apps/backend/.env.development, apps/dashboard/.env.development, or this generator."), + blank(), + comment("# Public emulator/app credentials"), + literal("NEXT_PUBLIC_STACK_DOCS_BASE_URL", "https://docs.stack-auth.com"), + literal("NEXT_PUBLIC_STACK_IS_LOCAL_EMULATOR", "true"), + fromSource("apps/dashboard/.env.development", dashboardEnv, "NEXT_PUBLIC_STACK_PROJECT_ID"), + fromSource("apps/dashboard/.env.development", dashboardEnv, "NEXT_PUBLIC_STACK_PUBLISHABLE_CLIENT_KEY"), + fromSource("apps/dashboard/.env.development", dashboardEnv, "STACK_SECRET_SERVER_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SERVER_SECRET"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_CHANGELOG_URL"), + blank(), + comment("# Seed/project defaults"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SEED_ENABLE_DUMMY_PROJECT"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SEED_INTERNAL_PROJECT_SIGN_UP_ENABLED"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SEED_INTERNAL_PROJECT_OTP_ENABLED"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SEED_INTERNAL_PROJECT_ALLOW_LOCALHOST"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SEED_INTERNAL_PROJECT_OAUTH_PROVIDERS"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SEED_INTERNAL_PROJECT_USER_INTERNAL_ACCESS"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SEED_INTERNAL_PROJECT_PUBLISHABLE_CLIENT_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SEED_INTERNAL_PROJECT_SECRET_SERVER_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SEED_INTERNAL_PROJECT_SUPER_SECRET_ADMIN_KEY"), + blank(), + comment("# Third-party/test integrations"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SVIX_API_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_OPENAI_API_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_OPENROUTER_API_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_STRIPE_SECRET_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_STRIPE_WEBHOOK_SECRET"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_RESEND_API_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_RESEND_WEBHOOK_SECRET"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_DNSIMPLE_API_TOKEN"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_DNSIMPLE_ACCOUNT_ID"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_DNSIMPLE_API_BASE_URL"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_FREESTYLE_API_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_VERCEL_SANDBOX_TOKEN"), + fromSource("apps/backend/.env.development", backendEnv, "CRON_SECRET"), + blank(), + comment("# Storage, queueing, and analytics"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_S3_REGION"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_S3_ACCESS_KEY_ID"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_S3_SECRET_ACCESS_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_S3_BUCKET"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_S3_PRIVATE_BUCKET"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_AWS_REGION"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_AWS_ACCESS_KEY_ID"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_AWS_SECRET_ACCESS_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_QSTASH_TOKEN"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_QSTASH_CURRENT_SIGNING_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_QSTASH_NEXT_SIGNING_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_CLICKHOUSE_ADMIN_USER"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_CLICKHOUSE_ADMIN_PASSWORD"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_CLICKHOUSE_EXTERNAL_PASSWORD"), + blank(), + comment("# Email and dashboard integration"), + literal("STACK_EMAIL_PORT", "2500"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_EMAIL_SECURE"), + literal("STACK_EMAIL_USERNAME", "does-not-matter"), + literal("STACK_EMAIL_PASSWORD", "does-not-matter"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_EMAIL_SENDER"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_DEFAULT_EMAIL_CAPACITY_PER_HOUR"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_EMAIL_MONITOR_PROJECT_ID"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_EMAIL_MONITOR_PUBLISHABLE_CLIENT_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_EMAIL_MONITOR_RESEND_EMAIL_DOMAIN"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_EMAIL_MONITOR_RESEND_EMAIL_API_KEY"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_EMAIL_MONITOR_SECRET_TOKEN"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_EMAIL_MONITOR_USE_INBUCKET"), + fromSource("apps/dashboard/.env.development", dashboardEnv, "STACK_FEATUREBASE_JWT_SECRET"), + blank(), + comment("# Mock OAuth defaults"), + literal("STACK_FORWARD_MOCK_OAUTH_SERVER", "false"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_GITHUB_CLIENT_ID"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_GITHUB_CLIENT_SECRET"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_GOOGLE_CLIENT_ID"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_GOOGLE_CLIENT_SECRET"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_MICROSOFT_CLIENT_ID"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_MICROSOFT_CLIENT_SECRET"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SPOTIFY_CLIENT_ID"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_SPOTIFY_CLIENT_SECRET"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_ALLOW_SHARED_OAUTH_ACCESS_TOKENS"), + blank(), + comment("# Internal service endpoints (defaults for docker-compose; overridden in QEMU)"), + literal("STACK_DATABASE_CONNECTION_STRING", "postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@127.0.0.1:5432/stackframe"), + fromSource("apps/backend/.env.development", backendEnv, "STACK_EMAIL_HOST"), + literal("STACK_SVIX_SERVER_URL", "http://127.0.0.1:8071"), + literal("STACK_S3_ENDPOINT", "http://127.0.0.1:9090"), + literal("STACK_QSTASH_URL", "http://127.0.0.1:8080"), + literal("STACK_CLICKHOUSE_URL", "http://127.0.0.1:8123"), + literal("STACK_CLICKHOUSE_DATABASE", "analytics"), + literal("STACK_EMAIL_MONITOR_INBUCKET_API_URL", "http://127.0.0.1:9001"), + literal("BACKEND_PORT", "8102"), + literal("DASHBOARD_PORT", "8101"), +]; + +const seenKeys = new Set(); +for (const entry of entries) { + if (entry.type !== "entry") { + continue; + } + + if (seenKeys.has(entry.key)) { + throw new Error(`Duplicate env key in generator: ${entry.key}`); + } + + seenKeys.add(entry.key); +} + +const content = `${entries.map((entry) => { + if (entry.type === "blank") { + return ""; + } + + if (entry.type === "comment") { + return entry.value; + } + + return `${entry.key}=${entry.value}`; +}).join("\n")}\n`; + +if (args[0] === "--check") { + const currentContent = fs.readFileSync(outputPath, "utf8"); + if (currentContent !== content) { + throw new Error(`${path.relative(rootDir, outputPath)} is out of date. Run pnpm run generate-local-emulator-env.`); + } + + console.log(`${path.relative(rootDir, outputPath)} is up to date.`); +} else { + fs.writeFileSync(outputPath, content); + console.log(`Wrote ${path.relative(rootDir, outputPath)}.`); +} diff --git a/docker/local-emulator/qemu/cloud-init/emulator/user-data b/docker/local-emulator/qemu/cloud-init/emulator/user-data index 39b8c33cdb..0daf66d025 100644 --- a/docker/local-emulator/qemu/cloud-init/emulator/user-data +++ b/docker/local-emulator/qemu/cloud-init/emulator/user-data @@ -55,7 +55,7 @@ write_files: set -a source /mnt/stack-runtime/runtime.env - source /mnt/stack-runtime/base.env + source /mnt/stack-runtime/.env.development set +a # Container-local dependencies run on localhost. Host-only development @@ -67,7 +67,7 @@ write_files: { # Static vars from base config and runtime (e.g. API keys, feature flags) - cat /mnt/stack-runtime/base.env + cat /mnt/stack-runtime/.env.development cat /mnt/stack-runtime/runtime.env # Computed vars — depend on port prefix or deps host diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index 8b18fbb1ba..c82cc33c08 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -79,7 +79,7 @@ prepare_runtime_config_iso() { { printf "STACK_EMULATOR_PORT_PREFIX=%s\n" "$PORT_PREFIX" } > "$cfg_dir/runtime.env" - cp "$SCRIPT_DIR/../base.env" "$cfg_dir/base.env" + cp "$SCRIPT_DIR/../.env.development" "$cfg_dir/.env.development" make_iso_from_dir "$cfg_iso" "STACKCFG" "$cfg_dir" } diff --git a/package.json b/package.json index 02575c0f7f..906fd2d705 100644 --- a/package.json +++ b/package.json @@ -24,14 +24,16 @@ "codegen": "pnpm pre && turbo run codegen && pnpm run generate-sdks", "codegen:backend": "pnpm pre && turbo run codegen --filter=@stackframe/backend...", "deps-compose": "docker compose -p stack-dependencies-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81} -f docker/dependencies/docker.compose.yaml", + "generate-local-emulator-env": "node ./docker/local-emulator/generate-env-development.mjs", + "check-local-emulator-env": "node ./docker/local-emulator/generate-env-development.mjs --check", "emulator-compose": "docker compose -p stack-local-emulator-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81} -f docker/local-emulator/docker-compose.yaml", - "start-emulator": "pnpm pre && pnpm run emulator-compose up --detach --build && pnpm run wait-until-emulator-is-ready && echo \"\\nLocal emulator started. Dashboard: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01 Backend: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02. 'pnpm run stop-emulator' to stop.\"", + "start-emulator": "pnpm pre && pnpm run generate-local-emulator-env && pnpm run emulator-compose up --detach --build && pnpm run wait-until-emulator-is-ready && echo \"\\nLocal emulator started. Dashboard: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01 Backend: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02. 'pnpm run stop-emulator' to stop.\"", "stop-emulator": "pnpm run emulator-compose kill && pnpm run emulator-compose down -v", "restart-emulator": "pnpm pre && pnpm run stop-emulator && pnpm run start-emulator", "wait-until-emulator-is-ready": "pnpx wait-on http-get://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02/health?db=1 http-get://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01/handler/sign-in", "wait-until-postgres-is-ready:emulator": "until pg_isready -h localhost -p ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28 2>/dev/null; do sleep 1; done", "emulator-qemu:build": "docker/local-emulator/qemu/build-image.sh", - "emulator-qemu:start": "docker/local-emulator/qemu/run-emulator.sh start", + "emulator-qemu:start": "pnpm run generate-local-emulator-env && docker/local-emulator/qemu/run-emulator.sh start", "emulator-qemu:stop": "docker/local-emulator/qemu/run-emulator.sh stop", "emulator-qemu:reset": "docker/local-emulator/qemu/run-emulator.sh reset", "emulator-qemu:status": "docker/local-emulator/qemu/run-emulator.sh status", From ad2db193b674d9b5d70321fef34bf819d7509acd Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 19:42:15 -0700 Subject: [PATCH 22/30] Refactor local emulator configuration validation and enhance CLI tests - Improved the `readConfigFromFile` function in `local-emulator.ts` to provide clearer error handling for invalid configurations, ensuring robust validation of the exported config object. - Added a new test in `cli.test.ts` to verify that the emulator CLI correctly rejects invalid architecture values, enhancing test coverage for the emulator command. - Simplified the startup time logging in `run-emulator.sh` by removing unnecessary Python calls, improving script efficiency. These changes enhance the reliability and usability of the local emulator and its CLI interface. --- apps/backend/src/lib/local-emulator.ts | 10 +++++---- apps/e2e/tests/general/cli.test.ts | 24 +++++++++++++++++++++ docker/local-emulator/qemu/run-emulator.sh | 20 ++++------------- packages/stack-cli/src/commands/emulator.ts | 19 +++++++++++++--- 4 files changed, 50 insertions(+), 23 deletions(-) diff --git a/apps/backend/src/lib/local-emulator.ts b/apps/backend/src/lib/local-emulator.ts index 2591460a53..3c679901f6 100644 --- a/apps/backend/src/lib/local-emulator.ts +++ b/apps/backend/src/lib/local-emulator.ts @@ -96,11 +96,13 @@ export async function readConfigFromFile(filePath: string): Promise { }); } + function runCliBareFromSource( + args: string[], + ): Promise<{ stdout: string, stderr: string, exitCode: number | null }> { + return new Promise((resolve) => { + execFile("node", ["--import", "tsx", CLI_SRC_BIN, ...args], { + env: { PATH: process.env.PATH ?? "", HOME: process.env.HOME ?? "", CI: "1" }, + timeout: 15_000, + }, (error, stdout, stderr) => { + resolve({ + stdout: stdout.toString(), + stderr: stderr.toString(), + exitCode: error ? (error as any).code ?? 1 : 0, + }); + }); + }); + } + it("emulator help shows subcommands", async ({ expect }) => { const { stdout, exitCode } = await runCliBare(["emulator", "--help"]); expect(exitCode).toBe(0); @@ -510,6 +528,12 @@ describe("Stack CLI — Emulator", () => { expect(stdout).toContain("--arch"); }); + it("emulator pull rejects invalid arch values", async ({ expect }) => { + const { stderr, exitCode } = await runCliBareFromSource(["emulator", "pull", "--arch", "sparc"]); + expect(exitCode).toBe(1); + expect(stderr).toContain("Invalid --arch: sparc; expected one of: arm64, amd64."); + }); + it("emulator list-releases help shows repo option", async ({ expect }) => { const { stdout, exitCode } = await runCliBare(["emulator", "list-releases", "--help"]); expect(exitCode).toBe(0); diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index c82cc33c08..2adc054834 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -366,24 +366,12 @@ cmd_status() { } cmd_bench() { - local start_time end_time + local elapsed cmd_stop >/dev/null 2>&1 || true - start_time="$(python3 - <<'PY' -import time -print(time.time()) -PY -)" + SECONDS=0 cmd_start - end_time="$(python3 - <<'PY' -import time -print(time.time()) -PY -)" - python3 - <", "Specific release tag") .option("--repo ", `GitHub repository (default: ${DEFAULT_REPO})`) .action(async (opts) => { - const arch: EmulatorArch = opts.arch || detectArch(); + const arch = parseEmulatorArch(opts.arch); await pullImage(arch, { repo: opts.repo, branch: opts.branch, @@ -154,7 +167,7 @@ export function registerEmulatorCommand(program: Command) { .description("Start the emulator (auto-pulls if no image exists)") .option("--arch ", "Target architecture") .action(async (opts) => { - const arch: EmulatorArch = opts.arch || detectArch(); + const arch = parseEmulatorArch(opts.arch); const qemuDir = findQemuDir(); const img = join(qemuDir, "images", `stack-emulator-${arch}.qcow2`); From a233e64b8e2ae8ce1809f969b792bbe033ba6aa4 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Thu, 19 Mar 2026 21:57:41 -0700 Subject: [PATCH 23/30] Update local emulator environment handling and documentation - Modified the local emulator scripts to replace references from `.env.development` to `base.env`, ensuring consistency in environment variable sourcing. - Updated the QEMU runtime path in the documentation to clarify the cause of a regression in the local emulator's startup time. - Enhanced the `.dockerignore` file to exclude new emulator-related files, improving build efficiency. These changes streamline the local emulator setup and improve clarity in documentation regarding environment configurations. --- .dockerignore | 3 ++- claude/CLAUDE-KNOWLEDGE.md | 3 +++ docker/local-emulator/qemu/cloud-init/emulator/user-data | 4 ++-- docker/local-emulator/qemu/run-emulator.sh | 2 +- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/.dockerignore b/.dockerignore index 836abeef03..893ad8ac96 100644 --- a/.dockerignore +++ b/.dockerignore @@ -61,6 +61,8 @@ dist .docusaurus .cache-loader **.tsbuildinfo +docker/local-emulator/qemu/images +docker/local-emulator/qemu/run .xata* @@ -149,4 +151,3 @@ packages/stack/* !packages/react/package.json !packages/next/package.json !packages/stack/package.json - diff --git a/claude/CLAUDE-KNOWLEDGE.md b/claude/CLAUDE-KNOWLEDGE.md index 4d5b54ba9e..0e158e682c 100644 --- a/claude/CLAUDE-KNOWLEDGE.md +++ b/claude/CLAUDE-KNOWLEDGE.md @@ -99,3 +99,6 @@ A: Update affected inline snapshots in `apps/e2e/tests/backend/endpoints/api/v1/ Q: How should `createOrUpdateProjectWithLegacyConfig` handle `onboardingStatus` for forward-compat checks? A: Only write `onboardingStatus` when the `Project.onboardingStatus` column exists (for example by checking `information_schema.columns` in-transaction) so current code can still run against older schemas where that column is absent. + +Q: What caused the March 19, 2026 QEMU local emulator deps startup regression? +A: The QEMU runtime path regressed when it switched from mounting `docker/local-emulator/base.env` into the runtime ISO to mounting the generated hidden file `docker/local-emulator/.env.development` instead. In testing, the `.env.development` QEMU path left cold boot stuck with only PostgreSQL healthy, while restoring the runtime ISO back to `base.env` brought deps startup back to about 12-13 seconds. The env payloads were effectively the same, so the likely issue was the QEMU runtime bundle/path handling for `.env.development`, not the actual env values. diff --git a/docker/local-emulator/qemu/cloud-init/emulator/user-data b/docker/local-emulator/qemu/cloud-init/emulator/user-data index 0daf66d025..39b8c33cdb 100644 --- a/docker/local-emulator/qemu/cloud-init/emulator/user-data +++ b/docker/local-emulator/qemu/cloud-init/emulator/user-data @@ -55,7 +55,7 @@ write_files: set -a source /mnt/stack-runtime/runtime.env - source /mnt/stack-runtime/.env.development + source /mnt/stack-runtime/base.env set +a # Container-local dependencies run on localhost. Host-only development @@ -67,7 +67,7 @@ write_files: { # Static vars from base config and runtime (e.g. API keys, feature flags) - cat /mnt/stack-runtime/.env.development + cat /mnt/stack-runtime/base.env cat /mnt/stack-runtime/runtime.env # Computed vars — depend on port prefix or deps host diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index 2adc054834..6e09b7746a 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -79,7 +79,7 @@ prepare_runtime_config_iso() { { printf "STACK_EMULATOR_PORT_PREFIX=%s\n" "$PORT_PREFIX" } > "$cfg_dir/runtime.env" - cp "$SCRIPT_DIR/../.env.development" "$cfg_dir/.env.development" + cp "$SCRIPT_DIR/../.env.development" "$cfg_dir/base.env" make_iso_from_dir "$cfg_iso" "STACKCFG" "$cfg_dir" } From 14f816449ffe10efc6b84dd52dff573da0e9141e Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Mon, 23 Mar 2026 11:02:59 -0700 Subject: [PATCH 24/30] Enhance local emulator project handling and configuration validation - Introduced file existence checks in the local emulator project route to ensure the specified config file is present before proceeding with project creation. - Updated the `setBranchConfigOverride` function to streamline writing configurations directly to the file when the local emulator is enabled, improving reliability. - Refactored the `readConfigFromFile` function to handle empty file scenarios by writing a default config, enhancing user experience. - Adjusted related tests to reflect changes in error handling and file creation logic, ensuring robust validation of local emulator functionality. These updates improve the local emulator's configuration management and enhance the overall developer experience. --- .../internal/local-emulator/project/route.tsx | 25 +++- apps/backend/src/lib/config.tsx | 86 ++------------ apps/backend/src/lib/local-emulator.ts | 112 +++++++----------- .../projects/page-client.tsx | 6 +- .../internal/local-emulator-project.test.ts | 30 +---- examples/demo/src/components/header.tsx | 2 +- 6 files changed, 81 insertions(+), 180 deletions(-) diff --git a/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx b/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx index 49c0efd451..e660c21c75 100644 --- a/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx +++ b/apps/backend/src/app/api/latest/internal/local-emulator/project/route.tsx @@ -5,6 +5,7 @@ import { LOCAL_EMULATOR_OWNER_TEAM_ID, isLocalEmulatorEnabled, readConfigFromFile, + resolveEmulatorPath, writeConfigToFile, } from "@/lib/local-emulator"; import { DEFAULT_BRANCH_ID, getSoleTenancyFromProjectBranch } from "@/lib/tenancies"; @@ -14,6 +15,7 @@ import { clientOrHigherAuthTypeSchema, yupNumber, yupObject, yupString } from "@ import { generateSecureRandomString } from "@stackframe/stack-shared/dist/utils/crypto"; import { StackAssertionError, StatusError } from "@stackframe/stack-shared/dist/utils/errors"; import { generateUuid } from "@stackframe/stack-shared/dist/utils/uuids"; +import fs from "fs/promises"; import * as path from "path"; type LocalEmulatorProjectMappingRow = { @@ -191,6 +193,25 @@ export const POST = createSmartRouteHandler({ } const absoluteFilePath = path.resolve(req.body.absolute_file_path); + const resolvedFilePath = resolveEmulatorPath(absoluteFilePath); + + // Validate file exists before creating a project + let fileExists: boolean; + try { + await fs.access(resolvedFilePath); + fileExists = true; + } catch { + fileExists = false; + } + if (!fileExists) { + throw new StatusError(StatusError.BadRequest, `Config file not found: ${absoluteFilePath}`); + } + + // If the file is empty, write a default config + const fileContent = await fs.readFile(resolvedFilePath, "utf-8"); + if (fileContent.trim() === "") { + await writeConfigToFile(absoluteFilePath, {}); + } await assertLocalEmulatorOwnerTeamReadiness(); @@ -198,10 +219,6 @@ export const POST = createSmartRouteHandler({ const credentials = await getOrCreateCredentials(projectId); const fileConfig = await readConfigFromFile(absoluteFilePath); - if (Object.keys(fileConfig).length === 0) { - await writeConfigToFile(absoluteFilePath, {}); - } - return { statusCode: 200 as const, bodyType: "json" as const, diff --git a/apps/backend/src/lib/config.tsx b/apps/backend/src/lib/config.tsx index 5e58e4bb70..078dd9be83 100644 --- a/apps/backend/src/lib/config.tsx +++ b/apps/backend/src/lib/config.tsx @@ -266,6 +266,14 @@ export async function setBranchConfigOverride(options: { }): Promise { const newConfig = migrateConfigOverride("branch", options.branchConfigOverride); + if (isLocalEmulatorEnabled() && await isLocalEmulatorProject(options.projectId)) { + const filePath = await getLocalEmulatorFilePath(options.projectId); + if (filePath) { + await writeConfigToFile(filePath, newConfig); + return; + } + } + // large configs make our DB slow; let's prevent them early const newConfigString = JSON.stringify(newConfig); if (newConfigString.length > 1_000_000) { @@ -279,17 +287,6 @@ export async function setBranchConfigOverride(options: { if (overrideErrors.status === "error") { captureError("setBranchConfigOverride", new StackAssertionError(`Config override is invalid — at a place where it should have already been validated! ${overrideErrors.error}`, { projectId: options.projectId, branchId: options.branchId })); } - - if (isLocalEmulatorEnabled()) { - const filePath = await getLocalEmulatorFilePath(options.projectId); - if (filePath != null) { - // Local emulator projects read branch config directly from the host config file, - // so the file stays the single source of truth to avoid split DB/file state. - await writeConfigToFile(filePath, newConfig); - return; - } - } - await globalPrismaClient.branchConfigOverride.upsert({ where: { projectId_branchId: { @@ -1107,73 +1104,6 @@ import.meta.vitest?.test('setEnvironmentConfigOverride blocks writes in local em } }); -import.meta.vitest?.test('setBranchConfigOverride writes local emulator config to the file instead of the DB', async ({ expect }) => { - const vi = import.meta.vitest?.vi; - if (!vi) { - throw new StackAssertionError("Vitest context is required for in-source tests."); - } - - const localEmulator = await import("./local-emulator"); - const isLocalEmulatorEnabledSpy = vi.spyOn(localEmulator, "isLocalEmulatorEnabled").mockReturnValue(true); - const getLocalEmulatorFilePathSpy = vi.spyOn(localEmulator, "getLocalEmulatorFilePath").mockResolvedValue("/Users/foo/project/stack.config.ts"); - const writeConfigToFileSpy = vi.spyOn(localEmulator, "writeConfigToFile").mockResolvedValue(undefined); - const upsertSpy = vi.spyOn(globalPrismaClient.branchConfigOverride, "upsert").mockImplementation(() => { - throw new StackAssertionError("DB upsert should not run for local emulator branch config writes."); - }); - - try { - await setBranchConfigOverride({ - projectId: "project-id", - branchId: "branch-id", - branchConfigOverride: { - "teams.allowClientTeamCreation": true, - }, - }); - - expect(writeConfigToFileSpy).toHaveBeenCalledWith("/Users/foo/project/stack.config.ts", { - "teams.allowClientTeamCreation": true, - }); - expect(upsertSpy).not.toHaveBeenCalled(); - } finally { - upsertSpy.mockRestore(); - writeConfigToFileSpy.mockRestore(); - getLocalEmulatorFilePathSpy.mockRestore(); - isLocalEmulatorEnabledSpy.mockRestore(); - } -}); - -import.meta.vitest?.test('setBranchConfigOverride surfaces local emulator file write failures before touching the DB', async ({ expect }) => { - const vi = import.meta.vitest?.vi; - if (!vi) { - throw new StackAssertionError("Vitest context is required for in-source tests."); - } - - const localEmulator = await import("./local-emulator"); - const isLocalEmulatorEnabledSpy = vi.spyOn(localEmulator, "isLocalEmulatorEnabled").mockReturnValue(true); - const getLocalEmulatorFilePathSpy = vi.spyOn(localEmulator, "getLocalEmulatorFilePath").mockResolvedValue("/Users/foo/project/stack.config.ts"); - const writeConfigToFileSpy = vi.spyOn(localEmulator, "writeConfigToFile").mockRejectedValue(new Error("virtio-9p timeout")); - const upsertSpy = vi.spyOn(globalPrismaClient.branchConfigOverride, "upsert").mockImplementation(() => { - throw new StackAssertionError("DB upsert should not run when the local emulator file write fails."); - }); - - try { - await expect(setBranchConfigOverride({ - projectId: "project-id", - branchId: "branch-id", - branchConfigOverride: { - "teams.allowClientTeamCreation": true, - }, - })).rejects.toThrow("virtio-9p timeout"); - - expect(upsertSpy).not.toHaveBeenCalled(); - } finally { - upsertSpy.mockRestore(); - writeConfigToFileSpy.mockRestore(); - getLocalEmulatorFilePathSpy.mockRestore(); - isLocalEmulatorEnabledSpy.mockRestore(); - } -}); - // --------------------------------------------------------------------------------------------------------------------- // Conversions // --------------------------------------------------------------------------------------------------------------------- diff --git a/apps/backend/src/lib/local-emulator.ts b/apps/backend/src/lib/local-emulator.ts index 3c679901f6..8b433d1d03 100644 --- a/apps/backend/src/lib/local-emulator.ts +++ b/apps/backend/src/lib/local-emulator.ts @@ -1,10 +1,10 @@ -import fs from "fs/promises"; -import path from "path"; -import { createJiti } from "jiti"; +import { globalPrismaClient } from "@/prisma-client"; import { isValidConfig } from "@stackframe/stack-shared/dist/config/format"; import { getEnvVariable } from "@stackframe/stack-shared/dist/utils/env"; -import { StatusError, throwErr } from "@stackframe/stack-shared/dist/utils/errors"; -import { globalPrismaClient } from "@/prisma-client"; +import { StatusError } from "@stackframe/stack-shared/dist/utils/errors"; +import fs from "fs/promises"; +import { createJiti } from "jiti"; +import path from "path"; export const LOCAL_EMULATOR_ADMIN_USER_ID = "63abbc96-5329-454a-ba56-e0460173c6c1"; export const LOCAL_EMULATOR_OWNER_TEAM_ID = "5a0c858b-d9e9-49d4-9943-8ce385d86428"; @@ -21,10 +21,6 @@ export function isLocalEmulatorEnabled() { return getEnvVariable("NEXT_PUBLIC_STACK_IS_LOCAL_EMULATOR", "") === "true"; } -function isErrnoException(error: unknown): error is NodeJS.ErrnoException { - return typeof error === "object" && error !== null && "code" in error; -} - export async function isLocalEmulatorProject(projectId: string) { if (!isLocalEmulatorEnabled()) { return false; @@ -41,84 +37,66 @@ export async function isLocalEmulatorProject(projectId: string) { return project !== null; } -async function pathExists(filePath: string): Promise { - try { - await fs.access(filePath); - return true; - } catch (error) { - if (isErrnoException(error) && error.code === "ENOENT") { - return false; - } - throw error; - } +export async function getLocalEmulatorFilePath(projectId: string): Promise { + const result = await globalPrismaClient.localEmulatorProject.findUnique({ + where: { projectId }, + select: { absoluteFilePath: true }, + }); + return result?.absoluteFilePath ?? null; } -/** - * Resolves the file path for config files in the local emulator. - * - * In the QEMU emulator, the host filesystem is mounted separately at /host, outside the - * guest qcow2 overlay. The DB stores absolute host paths (for example - * /Users/foo/project/stack.config.ts), so we map them to /host/ when the host mount - * is configured. We fail loudly if the host mount root is configured but inaccessible, - * because silently writing to a guest-local lookalike path would desync the dashboard from - * the user's real stack.config.ts file. - */ -async function resolveConfigFilePath(filePath: string): Promise { +export function resolveEmulatorPath(filePath: string): string { const hostMountRoot = getEnvVariable(LOCAL_EMULATOR_HOST_MOUNT_ROOT_ENV, ""); - if (hostMountRoot !== "") { - const hostMountedPath = path.join(hostMountRoot, filePath); - if (await pathExists(hostMountedPath) || await pathExists(path.dirname(hostMountedPath))) { - return hostMountedPath; - } - - throw new Error( - `Local emulator host mount root ${hostMountRoot} is configured, but ${hostMountedPath} is not accessible. ` + - "Restart the QEMU emulator so the host share is mounted, or choose a config path under the shared host root." - ); + if (hostMountRoot) { + return path.join(hostMountRoot, filePath); } - return filePath; } export async function readConfigFromFile(filePath: string): Promise> { + // Check for base64-encoded config content override from env var const envContent = getEnvVariable("STACK_LOCAL_EMULATOR_CONFIG_CONTENT", ""); - const resolvedPath = envContent ? filePath : await resolveConfigFilePath(filePath); - const content = envContent - ? Buffer.from(envContent, "base64").toString("utf-8") - : await fs.readFile(resolvedPath, "utf-8").catch((error: NodeJS.ErrnoException) => { - if (error.code === "ENOENT") return null; - throw error; - }); + let content: string; + if (envContent) { + content = Buffer.from(envContent, "base64").toString("utf-8"); + } else { + const resolvedPath = resolveEmulatorPath(filePath); + try { + content = await fs.readFile(resolvedPath, "utf-8"); + } catch (e: any) { + if (e?.code === "ENOENT") { + return {}; + } + throw e; + } + } - if (content === null || content.trim() === "") { + if (content.trim() === "") { return {}; } const jiti = createJiti(import.meta.url, { cache: false }); - const mod = jiti.evalModule(content, { filename: resolvedPath }); - const invalidConfigMessage = `Invalid config in ${filePath}. The file must export a 'config' object.`; - if (typeof mod !== "object" || mod === null || !("config" in mod)) { - throw new StatusError(StatusError.BadRequest, invalidConfigMessage); - } + const mod = jiti.evalModule(content, { filename: filePath }) as Record; const config = mod.config; if (!isValidConfig(config)) { - throw new StatusError(StatusError.BadRequest, invalidConfigMessage); + throw new StatusError(StatusError.BadRequest, `Invalid config in ${filePath}. The file must export a 'config' object.`); } return config; } export async function writeConfigToFile(filePath: string, config: Record): Promise { - const resolvedPath = await resolveConfigFilePath(filePath); - await fs.mkdir(path.dirname(resolvedPath), { recursive: true }); - const configString = JSON.stringify(config, null, 2); - const content = `export const config = ${configString};\n`; + const resolvedPath = resolveEmulatorPath(filePath); + const dir = path.dirname(resolvedPath); + const hostMountRoot = getEnvVariable(LOCAL_EMULATOR_HOST_MOUNT_ROOT_ENV, ""); + if (hostMountRoot) { + try { + await fs.access(dir); + } catch { + throw new Error(`Local emulator host mount root ${hostMountRoot} is configured but the parent directory for ${filePath} is not available at ${dir}. Ensure the host filesystem is mounted correctly.`); + } + } else { + await fs.mkdir(dir, { recursive: true }); + } + const content = `export const config = ${JSON.stringify(config, null, 2)};\n`; await fs.writeFile(resolvedPath, content, "utf-8"); } - -export async function getLocalEmulatorFilePath(projectId: string): Promise { - const project = await globalPrismaClient.localEmulatorProject.findUnique({ - where: { projectId }, - select: { absoluteFilePath: true }, - }); - return project?.absoluteFilePath ?? null; -} diff --git a/apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/projects/page-client.tsx b/apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/projects/page-client.tsx index 7bad0173d5..7398c60865 100644 --- a/apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/projects/page-client.tsx +++ b/apps/dashboard/src/app/(main)/(protected)/(outside-dashboard)/projects/page-client.tsx @@ -8,7 +8,7 @@ import { getPublicEnvVar } from "@/lib/env"; import { stackAppInternalsSymbol } from "@/lib/stack-app-internals"; import { GearIcon } from "@phosphor-icons/react"; import { AdminOwnedProject, Team, useStackApp, useUser } from "@stackframe/stack"; -import { projectOnboardingStatusValues, strictEmailSchema, type ProjectOnboardingStatus, yupObject } from "@stackframe/stack-shared/dist/schema-fields"; +import { projectOnboardingStatusValues, strictEmailSchema, yupObject, type ProjectOnboardingStatus } from "@stackframe/stack-shared/dist/schema-fields"; import { groupBy } from "@stackframe/stack-shared/dist/utils/arrays"; import { runAsynchronously, runAsynchronouslyWithAlert, wait } from "@stackframe/stack-shared/dist/utils/promises"; import { useQueryState } from "@stackframe/stack-shared/dist/utils/react"; @@ -267,7 +267,7 @@ export default function PageClient() {
- Enter the absolute path to your local Stack config file. If it does not exist yet, the local emulator will generate it with a default config, create or reuse the mapped project, and open it in the dashboard. + Enter the absolute path to your local Stack config file. The local emulator will create or reuse the mapped project and open it in the dashboard. diff --git a/apps/e2e/tests/backend/endpoints/api/v1/internal/local-emulator-project.test.ts b/apps/e2e/tests/backend/endpoints/api/v1/internal/local-emulator-project.test.ts index d329eaf128..75162e738f 100644 --- a/apps/e2e/tests/backend/endpoints/api/v1/internal/local-emulator-project.test.ts +++ b/apps/e2e/tests/backend/endpoints/api/v1/internal/local-emulator-project.test.ts @@ -1,6 +1,5 @@ import { randomUUID } from "crypto"; import fs from "fs/promises"; -import os from "os"; import path from "path"; import { describe } from "vitest"; import { it } from "../../../../../helpers"; @@ -54,7 +53,7 @@ describe("local emulator project endpoint", () => { } }); - it.runIf(isLocalEmulator)("creates non-existent config files with a default config", async ({ expect }) => { + it.runIf(isLocalEmulator)("rejects non-existent config files", async ({ expect }) => { const nonExistentPath = `/tmp/${randomUUID()}/stack.config.ts`; const response = await niceBackendFetch(LOCAL_EMULATOR_PROJECT_ENDPOINT, { @@ -65,11 +64,8 @@ describe("local emulator project endpoint", () => { }, }); - expect(response.status).toBe(200); - expect(JSON.parse(response.body.branch_config_override_string)).toEqual({}); - - const fileContent = await fs.readFile(nonExistentPath, "utf-8"); - expect(fileContent).toContain("export const config"); + expect(response.status).toBe(400); + expect(response.body).toContain("Config file not found"); }); it.runIf(isLocalEmulator)("writes default config for empty files", async ({ expect }) => { @@ -92,26 +88,6 @@ describe("local emulator project endpoint", () => { expect(fileContent).toContain("export const config"); }); - it.runIf(isLocalEmulator)("creates non-existent config files under the user home directory", async ({ expect }) => { - const nonExistentPath = path.join(os.homedir(), ".stack-auth-test", randomUUID(), "stack.config.ts"); - - const response = await niceBackendFetch(LOCAL_EMULATOR_PROJECT_ENDPOINT, { - accessType: "admin", - method: "POST", - body: { - absolute_file_path: nonExistentPath, - }, - }); - - expect(response.status).toBe(200); - expect(JSON.parse(response.body.branch_config_override_string)).toEqual({}); - - const fileContent = await fs.readFile(nonExistentPath, "utf-8"); - expect(fileContent).toContain("export const config"); - - await fs.rm(path.dirname(nonExistentPath), { recursive: true, force: true }); - }); - it.runIf(isLocalEmulator)("creates path-based projects, reuses mappings, and returns valid credentials", async ({ expect }) => { const pathA = await createTempConfigFile(); const pathB = await createTempConfigFile(); diff --git a/examples/demo/src/components/header.tsx b/examples/demo/src/components/header.tsx index 8957fbb076..28742b7de9 100644 --- a/examples/demo/src/components/header.tsx +++ b/examples/demo/src/components/header.tsx @@ -33,7 +33,7 @@ export default function Header() { setTheme(theme === 'dark' ? 'light' : 'dark')} />
-
+
{/* Placeholder for fixed header */} ); } From 9d0f7c1acd1ebf175da377e9a2fa59831d43235e Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Mon, 23 Mar 2026 11:43:09 -0700 Subject: [PATCH 25/30] Refactor local emulator scripts and configuration management - Updated local emulator scripts to standardize command names and improve clarity, changing references from `generate-local-emulator-env` to `emulator:generate-env` and similar adjustments for other commands. - Introduced a new configuration file (`stack.config.ts`) to centralize authentication and team settings, enhancing the management of user permissions and onboarding processes. - Removed the outdated `.env.development` file to streamline environment handling, ensuring that the local emulator relies on generated configurations. - Adjusted documentation and example usage to reflect the new command structure and configuration file, improving developer experience. These changes enhance the usability and maintainability of the local emulator setup, providing a clearer structure for configuration and command usage. --- .github/workflows/qemu-emulator-build.yaml | 5 - .gitignore | 1 + docker/local-emulator/.env.development | 92 ------------------- .../generate-env-development.mjs | 2 +- .../demo/src/app/emulator-status/page.tsx | 4 +- package.json | 28 +++--- packages/stack-cli/src/commands/emulator.ts | 30 ++---- stack.config.ts | 64 +++++++++++++ 8 files changed, 88 insertions(+), 138 deletions(-) delete mode 100644 docker/local-emulator/.env.development create mode 100644 stack.config.ts diff --git a/.github/workflows/qemu-emulator-build.yaml b/.github/workflows/qemu-emulator-build.yaml index 52b7ba5415..cc9d8e9ea4 100644 --- a/.github/workflows/qemu-emulator-build.yaml +++ b/.github/workflows/qemu-emulator-build.yaml @@ -5,11 +5,6 @@ on: branches: - main - dev - pull_request: - paths: - - 'docker/local-emulator/**' - - 'packages/stack-cli/src/commands/emulator.ts' - - '.github/workflows/qemu-emulator-build.yaml' workflow_dispatch: inputs: publish: diff --git a/.gitignore b/.gitignore index 592215d698..a8264f9e7d 100644 --- a/.gitignore +++ b/.gitignore @@ -24,6 +24,7 @@ vite.config.ts.timestamp-* .eslintcache .env.local .env.*.local +docker/local-emulator/.env.development scratch/ npm-debug.log* diff --git a/docker/local-emulator/.env.development b/docker/local-emulator/.env.development deleted file mode 100644 index d04860f669..0000000000 --- a/docker/local-emulator/.env.development +++ /dev/null @@ -1,92 +0,0 @@ -# Generated by docker/local-emulator/generate-env-development.mjs -# Do not edit manually; update apps/backend/.env.development, apps/dashboard/.env.development, or this generator. - -# Public emulator/app credentials -NEXT_PUBLIC_STACK_DOCS_BASE_URL=https://docs.stack-auth.com -NEXT_PUBLIC_STACK_IS_LOCAL_EMULATOR=true -NEXT_PUBLIC_STACK_PROJECT_ID=internal -NEXT_PUBLIC_STACK_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only -STACK_SECRET_SERVER_KEY=this-secret-server-key-is-for-local-development-only -STACK_SERVER_SECRET=23-wuNpik0gIW4mruTz25rbIvhuuvZFrLOLtL7J4tyo -STACK_CHANGELOG_URL=https://raw.githubusercontent.com/stack-auth/stack-auth/refs/heads/dev/CHANGELOG.md - -# Seed/project defaults -STACK_SEED_ENABLE_DUMMY_PROJECT=true -STACK_SEED_INTERNAL_PROJECT_SIGN_UP_ENABLED=true -STACK_SEED_INTERNAL_PROJECT_OTP_ENABLED=true -STACK_SEED_INTERNAL_PROJECT_ALLOW_LOCALHOST=true -STACK_SEED_INTERNAL_PROJECT_OAUTH_PROVIDERS=github,spotify,google,microsoft -STACK_SEED_INTERNAL_PROJECT_USER_INTERNAL_ACCESS=true -STACK_SEED_INTERNAL_PROJECT_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only -STACK_SEED_INTERNAL_PROJECT_SECRET_SERVER_KEY=this-secret-server-key-is-for-local-development-only -STACK_SEED_INTERNAL_PROJECT_SUPER_SECRET_ADMIN_KEY=this-super-secret-admin-key-is-for-local-development-only - -# Third-party/test integrations -STACK_SVIX_API_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NTUxNDA2MzksImV4cCI6MTk3MDUwMDYzOSwibmJmIjoxNjU1MTQwNjM5LCJpc3MiOiJzdml4LXNlcnZlciIsInN1YiI6Im9yZ18yM3JiOFlkR3FNVDBxSXpwZ0d3ZFhmSGlyTXUifQ.En8w77ZJWbd0qrMlHHupHUB-4cx17RfzFykseg95SUk -STACK_OPENAI_API_KEY=mock_openai_api_key -STACK_OPENROUTER_API_KEY=FORWARD_TO_PRODUCTION -STACK_STRIPE_SECRET_KEY=sk_test_mockstripekey -STACK_STRIPE_WEBHOOK_SECRET=mock_stripe_webhook_secret -STACK_RESEND_API_KEY=mock_resend_api_key -STACK_RESEND_WEBHOOK_SECRET=mock_resend_webhook_secret -STACK_DNSIMPLE_API_TOKEN=mock_dnsimple_api_token -STACK_DNSIMPLE_ACCOUNT_ID=mock_dnsimple_account_id -STACK_DNSIMPLE_API_BASE_URL=https://api.dnsimple.com/v2 -STACK_FREESTYLE_API_KEY=mock_stack_freestyle_key -STACK_VERCEL_SANDBOX_TOKEN=vercel_sandbox_disabled_for_local_development -CRON_SECRET=mock_cron_secret - -# Storage, queueing, and analytics -STACK_S3_REGION=us-east-1 -STACK_S3_ACCESS_KEY_ID=s3mockroot -STACK_S3_SECRET_ACCESS_KEY=s3mockroot -STACK_S3_BUCKET=stack-storage -STACK_S3_PRIVATE_BUCKET=stack-storage-private -STACK_AWS_REGION=us-east-1 -STACK_AWS_ACCESS_KEY_ID=test -STACK_AWS_SECRET_ACCESS_KEY=test -STACK_QSTASH_TOKEN=eyJVc2VySUQiOiJkZWZhdWx0VXNlciIsIlBhc3N3b3JkIjoiZGVmYXVsdFBhc3N3b3JkIn0= -STACK_QSTASH_CURRENT_SIGNING_KEY=sig_7kYjw48mhY7kAjqNGcy6cr29RJ6r -STACK_QSTASH_NEXT_SIGNING_KEY=sig_5ZB6DVzB1wjE8S6rZ7eenA8Pdnhs -STACK_CLICKHOUSE_ADMIN_USER=stackframe -STACK_CLICKHOUSE_ADMIN_PASSWORD=PASSWORD-PLACEHOLDER--9gKyMxJeMx -STACK_CLICKHOUSE_EXTERNAL_PASSWORD=PASSWORD-PLACEHOLDER--EZeHscBMzE - -# Email and dashboard integration -STACK_EMAIL_PORT=2500 -STACK_EMAIL_SECURE=false -STACK_EMAIL_USERNAME=does-not-matter -STACK_EMAIL_PASSWORD=does-not-matter -STACK_EMAIL_SENDER=noreply@example.com -STACK_DEFAULT_EMAIL_CAPACITY_PER_HOUR=10000 -STACK_EMAIL_MONITOR_PROJECT_ID=internal -STACK_EMAIL_MONITOR_PUBLISHABLE_CLIENT_KEY=this-publishable-client-key-is-for-local-development-only -STACK_EMAIL_MONITOR_RESEND_EMAIL_DOMAIN=stack-generated.example.com -STACK_EMAIL_MONITOR_RESEND_EMAIL_API_KEY=this-is-a-fake-key -STACK_EMAIL_MONITOR_SECRET_TOKEN=this-secret-token-is-for-local-development-only -STACK_EMAIL_MONITOR_USE_INBUCKET=true -STACK_FEATUREBASE_JWT_SECRET=secret-value - -# Mock OAuth defaults -STACK_FORWARD_MOCK_OAUTH_SERVER=false -STACK_GITHUB_CLIENT_ID=MOCK -STACK_GITHUB_CLIENT_SECRET=MOCK -STACK_GOOGLE_CLIENT_ID=MOCK -STACK_GOOGLE_CLIENT_SECRET=MOCK -STACK_MICROSOFT_CLIENT_ID=MOCK -STACK_MICROSOFT_CLIENT_SECRET=MOCK -STACK_SPOTIFY_CLIENT_ID=MOCK -STACK_SPOTIFY_CLIENT_SECRET=MOCK -STACK_ALLOW_SHARED_OAUTH_ACCESS_TOKENS=true - -# Internal service endpoints (defaults for docker-compose; overridden in QEMU) -STACK_DATABASE_CONNECTION_STRING=postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@127.0.0.1:5432/stackframe -STACK_EMAIL_HOST=127.0.0.1 -STACK_SVIX_SERVER_URL=http://127.0.0.1:8071 -STACK_S3_ENDPOINT=http://127.0.0.1:9090 -STACK_QSTASH_URL=http://127.0.0.1:8080 -STACK_CLICKHOUSE_URL=http://127.0.0.1:8123 -STACK_CLICKHOUSE_DATABASE=analytics -STACK_EMAIL_MONITOR_INBUCKET_API_URL=http://127.0.0.1:9001 -BACKEND_PORT=8102 -DASHBOARD_PORT=8101 diff --git a/docker/local-emulator/generate-env-development.mjs b/docker/local-emulator/generate-env-development.mjs index 5c9cfb7536..f0b0b20d23 100644 --- a/docker/local-emulator/generate-env-development.mjs +++ b/docker/local-emulator/generate-env-development.mjs @@ -193,7 +193,7 @@ const content = `${entries.map((entry) => { if (args[0] === "--check") { const currentContent = fs.readFileSync(outputPath, "utf8"); if (currentContent !== content) { - throw new Error(`${path.relative(rootDir, outputPath)} is out of date. Run pnpm run generate-local-emulator-env.`); + throw new Error(`${path.relative(rootDir, outputPath)} is out of date. Run pnpm run emulator:generate-env.`); } console.log(`${path.relative(rootDir, outputPath)} is up to date.`); diff --git a/examples/demo/src/app/emulator-status/page.tsx b/examples/demo/src/app/emulator-status/page.tsx index 32b1a9d9ac..e52b502431 100644 --- a/examples/demo/src/app/emulator-status/page.tsx +++ b/examples/demo/src/app/emulator-status/page.tsx @@ -166,10 +166,10 @@ export default function EmulatorStatusPage() { Start the all-in-one local emulator dependencies: 
               {`# Start (single container with all services)
-pnpm run emulator-compose up --detach --build
+pnpm run emulator:compose up --detach --build
 
 # Stop and remove volumes
-pnpm run emulator-compose down -v`}
+pnpm run emulator:compose down -v`}
This single container replaces the 17+ containers from the full docker-compose setup. diff --git a/package.json b/package.json index 906fd2d705..1a23f25ca4 100644 --- a/package.json +++ b/package.json @@ -24,20 +24,20 @@ "codegen": "pnpm pre && turbo run codegen && pnpm run generate-sdks", "codegen:backend": "pnpm pre && turbo run codegen --filter=@stackframe/backend...", "deps-compose": "docker compose -p stack-dependencies-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81} -f docker/dependencies/docker.compose.yaml", - "generate-local-emulator-env": "node ./docker/local-emulator/generate-env-development.mjs", - "check-local-emulator-env": "node ./docker/local-emulator/generate-env-development.mjs --check", - "emulator-compose": "docker compose -p stack-local-emulator-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81} -f docker/local-emulator/docker-compose.yaml", - "start-emulator": "pnpm pre && pnpm run generate-local-emulator-env && pnpm run emulator-compose up --detach --build && pnpm run wait-until-emulator-is-ready && echo \"\\nLocal emulator started. Dashboard: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01 Backend: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02. 'pnpm run stop-emulator' to stop.\"", - "stop-emulator": "pnpm run emulator-compose kill && pnpm run emulator-compose down -v", - "restart-emulator": "pnpm pre && pnpm run stop-emulator && pnpm run start-emulator", - "wait-until-emulator-is-ready": "pnpx wait-on http-get://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02/health?db=1 http-get://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01/handler/sign-in", - "wait-until-postgres-is-ready:emulator": "until pg_isready -h localhost -p ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28 2>/dev/null; do sleep 1; done", - "emulator-qemu:build": "docker/local-emulator/qemu/build-image.sh", - "emulator-qemu:start": "pnpm run generate-local-emulator-env && docker/local-emulator/qemu/run-emulator.sh start", - "emulator-qemu:stop": "docker/local-emulator/qemu/run-emulator.sh stop", - "emulator-qemu:reset": "docker/local-emulator/qemu/run-emulator.sh reset", - "emulator-qemu:status": "docker/local-emulator/qemu/run-emulator.sh status", - "emulator-qemu:bench": "docker/local-emulator/qemu/run-emulator.sh bench", + "emulator:generate-env": "node ./docker/local-emulator/generate-env-development.mjs", + "emulator:check-env": "node ./docker/local-emulator/generate-env-development.mjs --check", + "emulator:compose": "docker compose -p stack-local-emulator-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81} -f docker/local-emulator/docker-compose.yaml", + "emulator:start": "pnpm pre && pnpm run emulator:generate-env && pnpm run emulator:compose up --detach --build && pnpm run emulator:wait-ready && echo \"\\nLocal emulator started. Dashboard: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01 Backend: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02. 'pnpm run emulator:stop' to stop.\"", + "emulator:stop": "pnpm run emulator:compose kill && pnpm run emulator:compose down -v", + "emulator:restart": "pnpm pre && pnpm run emulator:stop && pnpm run emulator:start", + "emulator:wait-ready": "pnpx wait-on http-get://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02/health?db=1 http-get://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01/handler/sign-in", + "emulator:wait-postgres": "until pg_isready -h localhost -p ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28 2>/dev/null; do sleep 1; done", + "emulator:qemu:build": "docker/local-emulator/qemu/build-image.sh", + "emulator:qemu:start": "pnpm run emulator:generate-env && docker/local-emulator/qemu/run-emulator.sh start", + "emulator:qemu:stop": "docker/local-emulator/qemu/run-emulator.sh stop", + "emulator:qemu:reset": "docker/local-emulator/qemu/run-emulator.sh reset", + "emulator:qemu:status": "docker/local-emulator/qemu/run-emulator.sh status", + "emulator:qemu:bench": "docker/local-emulator/qemu/run-emulator.sh bench", "stop-deps": "POSTGRES_DELAY_MS=0 pnpm run deps-compose kill && POSTGRES_DELAY_MS=0 pnpm run deps-compose down -v", "wait-until-postgres-is-ready:pg_isready": "until pg_isready -h localhost -p ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28 && pg_isready -h localhost -p ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}34; do sleep 1; done", "wait-until-postgres-is-ready": "command -v pg_isready >/dev/null 2>&1 && pnpm run wait-until-postgres-is-ready:pg_isready || sleep 10 # not everyone has pg_isready installed, so we fallback to sleeping", diff --git a/packages/stack-cli/src/commands/emulator.ts b/packages/stack-cli/src/commands/emulator.ts index 27022754a3..e71ed53790 100644 --- a/packages/stack-cli/src/commands/emulator.ts +++ b/packages/stack-cli/src/commands/emulator.ts @@ -9,7 +9,6 @@ const DEFAULT_BRANCH = "dev"; const EMULATOR_ARCHES = ["arm64", "amd64"] as const; type EmulatorArch = typeof EMULATOR_ARCHES[number]; -type BuildTargetArch = EmulatorArch | "both"; function detectArch(): EmulatorArch { switch (process.arch) { @@ -79,14 +78,6 @@ function ghRelease(args: string[]): string { } } -function parseBuildArch(arch: string | undefined): BuildTargetArch { - const resolvedArch = arch ?? detectArch(); - if (resolvedArch === "arm64" || resolvedArch === "amd64" || resolvedArch === "both") { - return resolvedArch; - } - throw new CliError(`Unsupported build architecture: ${resolvedArch}. Use arm64, amd64, or both.`); -} - function isValidEmulatorArch(arch: string): arch is EmulatorArch { return arch === "arm64" || arch === "amd64"; } @@ -149,9 +140,9 @@ export function registerEmulatorCommand(program: Command) { emulator .command("pull") .description("Download the latest emulator image from GitHub Releases") - .option("--arch ", "Target architecture (arm64 or amd64)") + .option("--arch ", `Target architecture (arm64 or amd64, default: current system arch)`) .option("--branch ", `Release branch (default: ${DEFAULT_BRANCH})`) - .option("--tag ", "Specific release tag") + .option("--tag ", "Specific release tag (default: latest)") .option("--repo ", `GitHub repository (default: ${DEFAULT_REPO})`) .action(async (opts) => { const arch = parseEmulatorArch(opts.arch); @@ -163,9 +154,9 @@ export function registerEmulatorCommand(program: Command) { }); emulator - .command("run") - .description("Start the emulator (auto-pulls if no image exists)") - .option("--arch ", "Target architecture") + .command("start") + .description("Start the emulator in the background (auto-pulls the latest image if none exists)") + .option("--arch ", "Target architecture (arm64 or amd64, default: current system arch). Using a non-native architecture will use software emulation and be significantly slower.") .action(async (opts) => { const arch = parseEmulatorArch(opts.arch); const qemuDir = findQemuDir(); @@ -181,7 +172,7 @@ export function registerEmulatorCommand(program: Command) { emulator .command("stop") - .description("Stop the emulator") + .description("Stop the emulator (data is preserved; use 'reset' to clear all state)") .action(() => runEmulatorAction("stop")); emulator @@ -194,15 +185,6 @@ export function registerEmulatorCommand(program: Command) { .description("Show emulator and service health") .action(() => runEmulatorAction("status")); - emulator - .command("build") - .description("Build the QEMU emulator image locally") - .option("--arch ", "Target architecture (arm64, amd64, or both)") - .action(async (opts) => { - const arch = parseBuildArch(opts.arch); - await runScript(findQemuDir(), "build-image.sh", [arch]); - }); - emulator .command("list-releases") .description("List available emulator releases") diff --git a/stack.config.ts b/stack.config.ts new file mode 100644 index 0000000000..faadd7e822 --- /dev/null +++ b/stack.config.ts @@ -0,0 +1,64 @@ +export const config = { + "auth": { + "allowSignUp": true, + "password": { + "allowSignIn": true + }, + "otp": { + "allowSignIn": false + }, + "passkey": { + "allowSignIn": false + }, + "oauth": { + "accountMergeStrategy": "link_method", + "providers": { + "google": { + "type": "google", + "allowSignIn": true, + "allowConnectedAccounts": true + }, + "github": { + "type": "github", + "allowSignIn": true, + "allowConnectedAccounts": true + } + } + }, + "signUpRulesDefaultAction": "allow" + }, + "teams": { + "createPersonalTeamOnSignUp": false, + "allowClientTeamCreation": false + }, + "users": { + "allowClientUserDeletion": false + }, + "onboarding": { + "requireEmailVerification": false + }, + "apiKeys": { + "enabled": { + "team": false, + "user": false + } + }, + "domains": { + "allowLocalhost": true, + "trustedDomains": {} + }, + "rbac": { + "permissions": {}, + "defaultPermissions": { + "teamCreator": {}, + "teamMember": {}, + "signUp": {} + } + }, + "apps": { + "installed": {} + }, + "apps.installed.authentication.enabled": true, + "apps.installed.analytics.enabled": true, + "apps.installed.api-keys.enabled": true +}; From 77d87fd2ddad09cdf281bcf9c36b8948e8f01fb2 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Mon, 23 Mar 2026 15:25:05 -0700 Subject: [PATCH 26/30] Refactor local emulator setup and enhance GitHub Actions workflow - Updated the local emulator scripts to streamline command usage, replacing the previous docker-compose setup with a QEMU-based approach for improved performance and simplicity. - Removed the outdated `docker-compose.yaml` file and adjusted related scripts to utilize QEMU for running the emulator, enhancing clarity and maintainability. - Enhanced the GitHub Actions workflow to include a smoke test for the emulator, ensuring that services are healthy after startup. - Updated environment variable handling in the emulator scripts to reflect new port configurations, improving service accessibility. - Adjusted documentation to reflect the new emulator commands and usage patterns, providing clearer guidance for developers. These changes improve the usability and reliability of the local emulator setup, facilitating a smoother development experience. --- .github/workflows/qemu-emulator-build.yaml | 84 ++++++- docker/local-emulator/docker-compose.yaml | 54 ----- docker/local-emulator/qemu/run-emulator.sh | 61 +++-- .../demo/src/app/api/emulator-status/route.ts | 79 ++----- .../demo/src/app/emulator-status/page.tsx | 18 +- package.json | 18 +- packages/stack-cli/src/commands/emulator.ts | 221 ++++++------------ pnpm-lock.yaml | 34 +-- 8 files changed, 216 insertions(+), 353 deletions(-) delete mode 100644 docker/local-emulator/docker-compose.yaml diff --git a/.github/workflows/qemu-emulator-build.yaml b/.github/workflows/qemu-emulator-build.yaml index cc9d8e9ea4..9b10347e29 100644 --- a/.github/workflows/qemu-emulator-build.yaml +++ b/.github/workflows/qemu-emulator-build.yaml @@ -5,6 +5,7 @@ on: branches: - main - dev + pull_request: workflow_dispatch: inputs: publish: @@ -51,6 +52,9 @@ jobs: EMULATOR_PROVISION_TIMEOUT=6000 \ docker/local-emulator/qemu/build-image.sh ${{ matrix.arch }} + - name: Generate emulator env + run: node docker/local-emulator/generate-env-development.mjs + - name: Start emulator and verify run: | chmod +x docker/local-emulator/qemu/run-emulator.sh @@ -64,7 +68,7 @@ jobs: docker/local-emulator/qemu/run-emulator.sh status - name: Stop emulator - if: ${{ always() }} + if: always() run: | EMULATOR_ARCH=${{ matrix.arch }} \ docker/local-emulator/qemu/run-emulator.sh stop @@ -82,9 +86,71 @@ jobs: retention-days: 30 compression-level: 0 + test: + name: Smoke Test (${{ matrix.arch }}) + needs: build + runs-on: ubicloud-standard-8 + timeout-minutes: 30 + strategy: + fail-fast: false + matrix: + include: + - arch: amd64 + + steps: + - uses: actions/checkout@v6 + + - name: Install QEMU dependencies + run: | + sudo apt-get update + sudo apt-get install -y qemu-system-x86 qemu-utils genisoimage socat + + - name: Download built image + uses: actions/download-artifact@v4 + with: + name: qemu-emulator-${{ matrix.arch }} + path: docker/local-emulator/qemu/images/ + + - name: Generate emulator env + run: node docker/local-emulator/generate-env-development.mjs + + - name: Start emulator from artifact + run: | + chmod +x docker/local-emulator/qemu/run-emulator.sh docker/local-emulator/qemu/common.sh + EMULATOR_ARCH=${{ matrix.arch }} \ + EMULATOR_READY_TIMEOUT=600 \ + docker/local-emulator/qemu/run-emulator.sh start + + - name: Verify services are healthy + run: | + EMULATOR_ARCH=${{ matrix.arch }} \ + docker/local-emulator/qemu/run-emulator.sh status + + - name: Smoke test — backend health + run: curl -sf http://localhost:26701/health?db=1 + + - name: Smoke test — dashboard reachable + run: curl -sf -o /dev/null -w "HTTP %{http_code}\n" http://localhost:26700/handler/sign-in + + - name: Smoke test — MinIO health + run: curl -sf http://localhost:26702/minio/health/live + + - name: Smoke test — Inbucket reachable + run: curl -sf -o /dev/null -w "HTTP %{http_code}\n" http://localhost:26703/ + + - name: Stop emulator + if: always() + run: | + EMULATOR_ARCH=${{ matrix.arch }} \ + docker/local-emulator/qemu/run-emulator.sh stop + + - name: Print serial log on failure + if: failure() + run: tail -100 docker/local-emulator/qemu/run/vm/serial.log 2>/dev/null || true + publish: name: Publish to GitHub Releases - needs: build + needs: [build, test] if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev' || (github.event_name == 'workflow_dispatch' && inputs.publish) runs-on: ubuntu-latest permissions: @@ -112,22 +178,22 @@ jobs: cp "$f" release/ done - cat > release-notes.md <<'EOF' + cat > release-notes.md </dev/null && curl -fsS http://127.0.0.1:8101/handler/sign-in >/dev/null"] - interval: 10s - timeout: 5s - retries: 30 - start_period: 120s - -volumes: - postgres-data: - redis-data: - clickhouse-data: - minio-data: - inbucket-data: diff --git a/docker/local-emulator/qemu/run-emulator.sh b/docker/local-emulator/qemu/run-emulator.sh index 6e09b7746a..f2f3028ca6 100755 --- a/docker/local-emulator/qemu/run-emulator.sh +++ b/docker/local-emulator/qemu/run-emulator.sh @@ -13,6 +13,13 @@ VM_CPUS="${EMULATOR_CPUS:-4}" PORT_PREFIX="${PORT_PREFIX:-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}}" READY_TIMEOUT="${EMULATOR_READY_TIMEOUT:-240}" +# Fixed host-side ports for the QEMU emulator (267xx range). +# Only user-facing services are exposed; internal deps stay inside the VM. +EMULATOR_DASHBOARD_PORT="${EMULATOR_DASHBOARD_PORT:-26700}" +EMULATOR_BACKEND_PORT="${EMULATOR_BACKEND_PORT:-26701}" +EMULATOR_MINIO_PORT="${EMULATOR_MINIO_PORT:-26702}" +EMULATOR_INBUCKET_PORT="${EMULATOR_INBUCKET_PORT:-26703}" + RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[0;33m' @@ -106,18 +113,13 @@ service_is_up() { } deps_ready() { - service_is_up "${PORT_PREFIX}28" tcp && - service_is_up "${PORT_PREFIX}05" http / && - service_is_up "${PORT_PREFIX}29" tcp && - service_is_up "${PORT_PREFIX}13" http /api/v1/health/ && - service_is_up "${PORT_PREFIX}36" http /ping && - service_is_up "${PORT_PREFIX}21" http /minio/health/live && - service_is_up "${PORT_PREFIX}25" http / 401 + service_is_up "$EMULATOR_MINIO_PORT" http /minio/health/live && + service_is_up "$EMULATOR_INBUCKET_PORT" http / } app_ready() { - service_is_up "${PORT_PREFIX}02" http "/health?db=1" && - service_is_up "${PORT_PREFIX}01" http /handler/sign-in + service_is_up "$EMULATOR_BACKEND_PORT" http "/health?db=1" && + service_is_up "$EMULATOR_DASHBOARD_PORT" http /handler/sign-in } all_ready() { @@ -194,19 +196,11 @@ build_qemu_cmd() { esac local netdev="user,id=net0" - # Deps services - netdev+=",hostfwd=tcp::${PORT_PREFIX}28-:5432" - netdev+=",hostfwd=tcp::${PORT_PREFIX}29-:2500" - netdev+=",hostfwd=tcp::${PORT_PREFIX}05-:9001" - netdev+=",hostfwd=tcp::${PORT_PREFIX}30-:1100" - netdev+=",hostfwd=tcp::${PORT_PREFIX}13-:8071" - netdev+=",hostfwd=tcp::${PORT_PREFIX}21-:9090" - netdev+=",hostfwd=tcp::${PORT_PREFIX}25-:8080" - netdev+=",hostfwd=tcp::${PORT_PREFIX}36-:8123" - netdev+=",hostfwd=tcp::${PORT_PREFIX}37-:9009" - # App services - netdev+=",hostfwd=tcp::${PORT_PREFIX}01-:${PORT_PREFIX}01" - netdev+=",hostfwd=tcp::${PORT_PREFIX}02-:${PORT_PREFIX}02" + # Only expose user-facing services; internal deps stay inside the VM. + netdev+=",hostfwd=tcp::${EMULATOR_DASHBOARD_PORT}-:${PORT_PREFIX}01" + netdev+=",hostfwd=tcp::${EMULATOR_BACKEND_PORT}-:${PORT_PREFIX}02" + netdev+=",hostfwd=tcp::${EMULATOR_MINIO_PORT}-:9090" + netdev+=",hostfwd=tcp::${EMULATOR_INBUCKET_PORT}-:9001" QEMU_CMD=( "$qemu_bin" @@ -251,11 +245,11 @@ tail_vm_logs() { } ensure_ports_free() { - local ports=("${PORT_PREFIX}01" "${PORT_PREFIX}02" "${PORT_PREFIX}05" "${PORT_PREFIX}13" "${PORT_PREFIX}21" "${PORT_PREFIX}25" "${PORT_PREFIX}28" "${PORT_PREFIX}29" "${PORT_PREFIX}30" "${PORT_PREFIX}36" "${PORT_PREFIX}37") + local ports=("$EMULATOR_DASHBOARD_PORT" "$EMULATOR_BACKEND_PORT" "$EMULATOR_MINIO_PORT" "$EMULATOR_INBUCKET_PORT") local port for port in "${ports[@]}"; do if lsof -iTCP:"$port" -sTCP:LISTEN >/dev/null 2>&1; then - err "Port $port is already in use. Stop the Docker emulator or other services first." + err "Port $port is already in use. Stop any conflicting services first." exit 1 fi done @@ -297,7 +291,8 @@ cmd_start() { mkdir -p "$RUN_DIR" info "Starting QEMU local emulator" - info "Arch: $ARCH | Accel: $ACCEL | Prefix: $PORT_PREFIX" + info "Arch: $ARCH | Accel: $ACCEL" + info "Ports: Dashboard=$EMULATOR_DASHBOARD_PORT Backend=$EMULATOR_BACKEND_PORT MinIO=$EMULATOR_MINIO_PORT Inbucket=$EMULATOR_INBUCKET_PORT" start_vm @@ -313,7 +308,9 @@ cmd_start() { exit 1 fi - log "All services are green. The qcow2 overlay preserves emulator state across restarts, while /host stays a live host share outside the VM disk." + log "All services are green." + info "Dashboard: http://localhost:${EMULATOR_DASHBOARD_PORT}" + info "Backend: http://localhost:${EMULATOR_BACKEND_PORT}" } cmd_stop() { @@ -354,14 +351,10 @@ cmd_status() { fi echo "" echo "Services:" - print_service_status "Dashboard" "${PORT_PREFIX}01" http /handler/sign-in - print_service_status "Backend" "${PORT_PREFIX}02" http "/health?db=1" - print_service_status "PostgreSQL" "${PORT_PREFIX}28" tcp - print_service_status "Inbucket HTTP" "${PORT_PREFIX}05" http / - print_service_status "Svix" "${PORT_PREFIX}13" http /api/v1/health/ - print_service_status "MinIO" "${PORT_PREFIX}21" http /minio/health/live - print_service_status "QStash" "${PORT_PREFIX}25" http / 401 - print_service_status "ClickHouse" "${PORT_PREFIX}36" http /ping + print_service_status "Dashboard" "$EMULATOR_DASHBOARD_PORT" http /handler/sign-in + print_service_status "Backend" "$EMULATOR_BACKEND_PORT" http "/health?db=1" + print_service_status "MinIO" "$EMULATOR_MINIO_PORT" http /minio/health/live + print_service_status "Inbucket HTTP" "$EMULATOR_INBUCKET_PORT" http / exit "$STATUS_FAILED" } diff --git a/examples/demo/src/app/api/emulator-status/route.ts b/examples/demo/src/app/api/emulator-status/route.ts index 34e08145c4..1cf488711c 100644 --- a/examples/demo/src/app/api/emulator-status/route.ts +++ b/examples/demo/src/app/api/emulator-status/route.ts @@ -1,10 +1,7 @@ import { NextResponse } from 'next/server'; -import net from 'net'; export const dynamic = 'force-dynamic'; -const PORT_PREFIX = process.env.NEXT_PUBLIC_STACK_PORT_PREFIX ?? '81'; - type ServiceCheck = { name: string; description: string; @@ -15,63 +12,30 @@ type ServiceCheck = { const SERVICES: ServiceCheck[] = [ { - name: 'PostgreSQL', - description: 'Primary database', - port: Number(`${PORT_PREFIX}28`), - protocol: 'tcp', - }, - { - name: 'Inbucket (HTTP)', - description: 'Email capture UI', - port: Number(`${PORT_PREFIX}05`), - protocol: 'http', - httpPath: '/', - }, - { - name: 'Inbucket (SMTP)', - description: 'Email SMTP server', - port: Number(`${PORT_PREFIX}29`), - protocol: 'tcp', - }, - { - name: 'Svix', - description: 'Webhook delivery', - port: Number(`${PORT_PREFIX}13`), + name: 'Stack Dashboard', + description: 'Dashboard UI', + port: 26700, protocol: 'http', - httpPath: '/api/v1/health/', + httpPath: '/handler/sign-in', }, { - name: 'ClickHouse', - description: 'Analytics database', - port: Number(`${PORT_PREFIX}36`), + name: 'Stack Backend', + description: 'API server', + port: 26701, protocol: 'http', - httpPath: '/ping', + httpPath: '/health?db=1', }, { name: 'MinIO (S3)', description: 'Object storage', - port: Number(`${PORT_PREFIX}21`), + port: 26702, protocol: 'http', httpPath: '/minio/health/live', }, { - name: 'QStash', - description: 'Job queue', - port: Number(`${PORT_PREFIX}25`), - protocol: 'http', - httpPath: '/', - }, - { - name: 'Stack Backend', - description: 'API server', - port: Number(`${PORT_PREFIX}02`), - protocol: 'http', - httpPath: '/', - }, - { - name: 'Stack Dashboard', - description: 'Dashboard UI', - port: Number(`${PORT_PREFIX}01`), + name: 'Inbucket (HTTP)', + description: 'Email capture UI', + port: 26703, protocol: 'http', httpPath: '/', }, @@ -90,27 +54,10 @@ async function checkHttp(port: number, path: string, timeoutMs = 3000): Promise< } } -async function checkTcp(port: number, timeoutMs = 3000): Promise<{ up: boolean; latencyMs: number }> { - const start = performance.now(); - return await new Promise((resolve) => { - const socket = net.createConnection({ host: '127.0.0.1', port }, () => { - socket.destroy(); - resolve({ up: true, latencyMs: Math.round(performance.now() - start) }); - }); - socket.on('error', () => resolve({ up: false, latencyMs: Math.round(performance.now() - start) })); - socket.setTimeout(timeoutMs, () => { - socket.destroy(); - resolve({ up: false, latencyMs: Math.round(performance.now() - start) }); - }); - }); -} - export async function GET() { const results = await Promise.all( SERVICES.map(async (svc) => { - const check = svc.protocol === 'http' - ? await checkHttp(svc.port, svc.httpPath ?? '/') - : await checkTcp(svc.port); + const check = await checkHttp(svc.port, svc.httpPath ?? '/'); return { name: svc.name, description: svc.description, diff --git a/examples/demo/src/app/emulator-status/page.tsx b/examples/demo/src/app/emulator-status/page.tsx index e52b502431..61c57e22bb 100644 --- a/examples/demo/src/app/emulator-status/page.tsx +++ b/examples/demo/src/app/emulator-status/page.tsx @@ -163,16 +163,22 @@ export default function EmulatorStatusPage() { Quick Start - Start the all-in-one local emulator dependencies: + Start the QEMU local emulator: 
-              {`# Start (single container with all services)
-pnpm run emulator:compose up --detach --build
+              {`# Pull the latest image and start the emulator
+pnpm run emulator:start
 
-# Stop and remove volumes
-pnpm run emulator:compose down -v`}
+# Check service health
+pnpm run emulator:status
+
+# Stop (data is preserved)
+pnpm run emulator:stop
+
+# Reset for a fresh boot
+pnpm run emulator:reset`}
- This single container replaces the 17+ containers from the full docker-compose setup. + Dashboard: localhost:26700 | Backend: localhost:26701 diff --git a/package.json b/package.json index 1a23f25ca4..0aca2a60cc 100644 --- a/package.json +++ b/package.json @@ -26,18 +26,12 @@ "deps-compose": "docker compose -p stack-dependencies-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81} -f docker/dependencies/docker.compose.yaml", "emulator:generate-env": "node ./docker/local-emulator/generate-env-development.mjs", "emulator:check-env": "node ./docker/local-emulator/generate-env-development.mjs --check", - "emulator:compose": "docker compose -p stack-local-emulator-${NEXT_PUBLIC_STACK_PORT_PREFIX:-81} -f docker/local-emulator/docker-compose.yaml", - "emulator:start": "pnpm pre && pnpm run emulator:generate-env && pnpm run emulator:compose up --detach --build && pnpm run emulator:wait-ready && echo \"\\nLocal emulator started. Dashboard: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01 Backend: http://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02. 'pnpm run emulator:stop' to stop.\"", - "emulator:stop": "pnpm run emulator:compose kill && pnpm run emulator:compose down -v", - "emulator:restart": "pnpm pre && pnpm run emulator:stop && pnpm run emulator:start", - "emulator:wait-ready": "pnpx wait-on http-get://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}02/health?db=1 http-get://localhost:${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}01/handler/sign-in", - "emulator:wait-postgres": "until pg_isready -h localhost -p ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28 2>/dev/null; do sleep 1; done", - "emulator:qemu:build": "docker/local-emulator/qemu/build-image.sh", - "emulator:qemu:start": "pnpm run emulator:generate-env && docker/local-emulator/qemu/run-emulator.sh start", - "emulator:qemu:stop": "docker/local-emulator/qemu/run-emulator.sh stop", - "emulator:qemu:reset": "docker/local-emulator/qemu/run-emulator.sh reset", - "emulator:qemu:status": "docker/local-emulator/qemu/run-emulator.sh status", - "emulator:qemu:bench": "docker/local-emulator/qemu/run-emulator.sh bench", + "emulator:start": "pnpm run emulator:generate-env && docker/local-emulator/qemu/run-emulator.sh start", + "emulator:stop": "docker/local-emulator/qemu/run-emulator.sh stop", + "emulator:reset": "docker/local-emulator/qemu/run-emulator.sh reset", + "emulator:status": "docker/local-emulator/qemu/run-emulator.sh status", + "emulator:build": "docker/local-emulator/qemu/build-image.sh", + "emulator:bench": "docker/local-emulator/qemu/run-emulator.sh bench", "stop-deps": "POSTGRES_DELAY_MS=0 pnpm run deps-compose kill && POSTGRES_DELAY_MS=0 pnpm run deps-compose down -v", "wait-until-postgres-is-ready:pg_isready": "until pg_isready -h localhost -p ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28 && pg_isready -h localhost -p ${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}34; do sleep 1; done", "wait-until-postgres-is-ready": "command -v pg_isready >/dev/null 2>&1 && pnpm run wait-until-postgres-is-ready:pg_isready || sleep 10 # not everyone has pg_isready installed, so we fallback to sleeping", diff --git a/packages/stack-cli/src/commands/emulator.ts b/packages/stack-cli/src/commands/emulator.ts index e71ed53790..a4878b2371 100644 --- a/packages/stack-cli/src/commands/emulator.ts +++ b/packages/stack-cli/src/commands/emulator.ts @@ -4,202 +4,135 @@ import { existsSync, mkdirSync, renameSync, unlinkSync } from "fs"; import { join, resolve } from "path"; import { CliError } from "../lib/errors.js"; -const DEFAULT_REPO = "stack-auth/stack-auth"; -const DEFAULT_BRANCH = "dev"; -const EMULATOR_ARCHES = ["arm64", "amd64"] as const; - -type EmulatorArch = typeof EMULATOR_ARCHES[number]; - -function detectArch(): EmulatorArch { - switch (process.arch) { - case "arm64": { - return "arm64"; - } - case "x64": { - return "amd64"; - } - default: { - throw new CliError(`Unsupported architecture: ${process.arch}`); +function gh(args: string[]): string { + try { + return execFileSync("gh", args, { encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"] }).trim(); + } catch (err: unknown) { + if (err instanceof Error && "stderr" in err && typeof err.stderr === "string") { + throw new CliError(`GitHub CLI error: ${err.stderr}`); } + throw new CliError("GitHub CLI (gh) is required. Install: https://cli.github.com/"); } } function findQemuDir(): string { - const candidates = [ - resolve(process.cwd(), "docker/local-emulator/qemu"), - resolve(process.cwd(), "../docker/local-emulator/qemu"), - ]; - - for (const candidate of candidates) { - if (existsSync(join(candidate, "run-emulator.sh"))) { - return candidate; - } + for (const rel of ["docker/local-emulator/qemu", "../docker/local-emulator/qemu"]) { + const dir = resolve(process.cwd(), rel); + if (existsSync(join(dir, "run-emulator.sh"))) return dir; } - - throw new CliError( - "Could not find QEMU emulator directory. Run this from the stack-auth repo root." - ); + throw new CliError("Could not find QEMU emulator directory. Run this from the stack-auth repo root."); } -function runCommand(cwd: string, command: string, args: string[], env?: Record): Promise { +function runEmulator(action: string, env?: Record): Promise { + const qemuDir = findQemuDir(); return new Promise((resolve, reject) => { - const child = spawn(command, args, { + const child = spawn(join(qemuDir, "run-emulator.sh"), [action], { stdio: "inherit", env: { ...process.env, ...env }, - cwd, - }); - - child.on("close", (code) => { - if (code === 0) resolve(); - else reject(new CliError(`${command} exited with code ${code}`)); - }); - child.on("error", (err) => { - reject(new CliError(`Failed to run ${command}: ${err.message}`)); + cwd: qemuDir, }); + child.on("close", (code) => code === 0 ? resolve() : reject(new CliError(`run-emulator.sh ${action} exited with code ${code}`))); + child.on("error", (err) => reject(new CliError(`Failed to run run-emulator.sh: ${err.message}`))); }); } -function runScript(qemuDir: string, script: string, args: string[], env?: Record): Promise { - return runCommand(qemuDir, join(qemuDir, script), args, env); +function resolveArch(raw?: string): "arm64" | "amd64" { + const arch = raw ?? (process.arch === "arm64" ? "arm64" : process.arch === "x64" ? "amd64" : null); + if (arch === "arm64" || arch === "amd64") return arch; + throw new CliError(`Invalid architecture: ${raw ?? process.arch}. Expected arm64 or amd64.`); } -function runEmulatorAction(action: string, env?: Record): Promise { - return runScript(findQemuDir(), "run-emulator.sh", [action], env); -} - -function ghRelease(args: string[]): string { - try { - return execFileSync("gh", args, { encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"] }).trim(); - } catch (err: unknown) { - if (err instanceof Error && "stderr" in err && typeof err.stderr === "string") { - throw new CliError(`GitHub CLI error: ${err.stderr}`); - } - throw new CliError("GitHub CLI (gh) is required. Install: https://cli.github.com/"); - } -} - -function isValidEmulatorArch(arch: string): arch is EmulatorArch { - return arch === "arm64" || arch === "amd64"; -} - -function parseEmulatorArch(arch: string | undefined): EmulatorArch { - const resolvedArch = arch ?? detectArch(); - if (isValidEmulatorArch(resolvedArch)) { - return resolvedArch; - } - throw new CliError(`Invalid --arch: ${resolvedArch}; expected one of: ${EMULATOR_ARCHES.join(", ")}.`); -} - -async function pullImage(arch: EmulatorArch, opts: { repo?: string; branch?: string; tag?: string } = {}) { - const repo = opts.repo ?? DEFAULT_REPO; - const branch = opts.branch ?? DEFAULT_BRANCH; +function pullRelease(arch: "arm64" | "amd64", opts: { repo?: string; branch?: string; tag?: string } = {}) { + const repo = opts.repo ?? "stack-auth/stack-auth"; + const branch = opts.branch ?? "dev"; const tag = opts.tag ?? `emulator-${branch}-latest`; const asset = `stack-emulator-${arch}.qcow2`; - - const qemuDir = findQemuDir(); - const imageDir = join(qemuDir, "images"); + const imageDir = join(findQemuDir(), "images"); mkdirSync(imageDir, { recursive: true }); - const dest = join(imageDir, asset); const tmpDest = `${dest}.download`; - console.log(`Pulling image for ${arch} from release ${tag}...`); - + console.log(`Pulling ${asset} from release ${tag}...`); try { - execFileSync("gh", [ - "release", - "download", - tag, - "--repo", - repo, - "--pattern", - asset, - "--output", - tmpDest, - "--clobber", - ], { stdio: "inherit" }); + execFileSync("gh", ["release", "download", tag, "--repo", repo, "--pattern", asset, "--output", tmpDest, "--clobber"], { stdio: "inherit" }); } catch (err) { if (existsSync(tmpDest)) unlinkSync(tmpDest); - const reason = err instanceof Error - ? (err.stack ?? err.message) - : String(err); - throw new CliError( - `Failed to download ${asset} from release ${tag}: ${reason}\nRun 'stack emulator list-releases' to see available releases.` - ); + throw new CliError(`Failed to download ${asset} from release ${tag}: ${err instanceof Error ? err.message : err}\nRun 'stack emulator list-releases' to see available releases.`); } - renameSync(tmpDest, dest); console.log(`Downloaded: ${dest}`); } export function registerEmulatorCommand(program: Command) { - const emulator = program - .command("emulator") - .description("Manage the QEMU local emulator"); + const emulator = program.command("emulator").description("Manage the QEMU local emulator"); emulator .command("pull") - .description("Download the latest emulator image from GitHub Releases") - .option("--arch ", `Target architecture (arm64 or amd64, default: current system arch)`) - .option("--branch ", `Release branch (default: ${DEFAULT_BRANCH})`) + .description("Download an emulator image from GitHub Releases or a PR build") + .option("--arch ", "Target architecture (default: current system arch)") + .option("--branch ", "Release branch (default: dev)") .option("--tag ", "Specific release tag (default: latest)") - .option("--repo ", `GitHub repository (default: ${DEFAULT_REPO})`) + .option("--repo ", "GitHub repository (default: stack-auth/stack-auth)") + .option("--pr ", "Pull from a PR's CI artifacts") + .option("--run ", "Pull from a specific workflow run's artifacts") .action(async (opts) => { - const arch = parseEmulatorArch(opts.arch); - await pullImage(arch, { - repo: opts.repo, - branch: opts.branch, - tag: opts.tag, - }); + const arch = resolveArch(opts.arch); + const repo = opts.repo ?? "stack-auth/stack-auth"; + + if (opts.run || opts.pr) { + let runId = opts.run as string | undefined; + if (!runId) { + console.log(`Finding latest successful build for PR #${opts.pr}...`); + const { headRefName } = JSON.parse(gh(["pr", "view", opts.pr, "--repo", repo, "--json", "headRefName"])); + const runs = JSON.parse(gh(["run", "list", "--repo", repo, "--workflow", "qemu-emulator-build.yaml", "--branch", headRefName, "--status", "success", "--limit", "1", "--json", "databaseId"])); + if (runs.length === 0) throw new CliError(`No successful build found for PR #${opts.pr} (branch: ${headRefName}).`); + runId = String(runs[0].databaseId); + } + + const imageDir = join(findQemuDir(), "images"); + mkdirSync(imageDir, { recursive: true }); + const dest = join(imageDir, `stack-emulator-${arch}.qcow2`); + if (existsSync(dest)) unlinkSync(dest); + console.log(`Downloading qemu-emulator-${arch} from workflow run ${runId}...`); + try { + execFileSync("gh", ["run", "download", runId, "--repo", repo, "--name", `qemu-emulator-${arch}`, "--dir", imageDir], { stdio: "inherit" }); + } catch (err) { + throw new CliError(`Failed to download artifact from run ${runId}: ${err instanceof Error ? err.message : err}`); + } + if (!existsSync(dest)) throw new CliError(`Expected image not found at ${dest} after download.`); + console.log(`Downloaded: ${dest}`); + } else { + pullRelease(arch, { repo, branch: opts.branch, tag: opts.tag }); + } }); emulator .command("start") .description("Start the emulator in the background (auto-pulls the latest image if none exists)") - .option("--arch ", "Target architecture (arm64 or amd64, default: current system arch). Using a non-native architecture will use software emulation and be significantly slower.") + .option("--arch ", "Target architecture (default: current system arch). Non-native uses software emulation and is significantly slower.") .action(async (opts) => { - const arch = parseEmulatorArch(opts.arch); - const qemuDir = findQemuDir(); - const img = join(qemuDir, "images", `stack-emulator-${arch}.qcow2`); - + const arch = resolveArch(opts.arch); + const img = join(findQemuDir(), "images", `stack-emulator-${arch}.qcow2`); if (!existsSync(img)) { console.log("No emulator image found. Pulling latest..."); - await pullImage(arch); + pullRelease(arch); } - - await runScript(qemuDir, "run-emulator.sh", ["start"], { EMULATOR_ARCH: arch }); + await runEmulator("start", { EMULATOR_ARCH: arch }); }); - emulator - .command("stop") - .description("Stop the emulator (data is preserved; use 'reset' to clear all state)") - .action(() => runEmulatorAction("stop")); - - emulator - .command("reset") - .description("Reset emulator state for a fresh boot") - .action(() => runEmulatorAction("reset")); - - emulator - .command("status") - .description("Show emulator and service health") - .action(() => runEmulatorAction("status")); + emulator.command("stop").description("Stop the emulator (data preserved; use 'reset' to clear)").action(() => runEmulator("stop")); + emulator.command("reset").description("Reset emulator state for a fresh boot").action(() => runEmulator("reset")); + emulator.command("status").description("Show emulator and service health").action(() => runEmulator("status")); emulator .command("list-releases") .description("List available emulator releases") - .option("--repo ", `GitHub repository (default: ${DEFAULT_REPO})`) - .action(async (opts) => { - const repo = opts.repo || DEFAULT_REPO; + .option("--repo ", "GitHub repository (default: stack-auth/stack-auth)") + .action((opts) => { + const repo = opts.repo ?? "stack-auth/stack-auth"; console.log(`Available emulator releases from ${repo}:\n`); - const output = ghRelease(["release", "list", "--repo", repo, "--limit", "20"]); - const lines = output.split("\n").filter((l) => l.toLowerCase().includes("emulator")); - if (lines.length === 0) { - console.log("No emulator releases found."); - } else { - for (const line of lines) { - console.log(line); - } - } + const lines = gh(["release", "list", "--repo", repo, "--limit", "20"]).split("\n").filter((l) => l.toLowerCase().includes("emulator")); + if (lines.length === 0) console.log("No emulator releases found."); + else for (const line of lines) console.log(line); }); } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index ed8650826c..77dfbbbb79 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -4976,14 +4976,6 @@ packages: '@types/node': optional: true - '@isaacs/balanced-match@4.0.1': - resolution: {integrity: sha512-yzMTt9lEb8Gv7zRioUilSglI0c0smZ9k5D65677DLWLtWJaXIS3CqcGyUFByYKlnUj6TkjLVs54fBl6+TiGQDQ==} - engines: {node: 20 || >=22} - - '@isaacs/brace-expansion@5.0.0': - resolution: {integrity: sha512-ZT55BDLV0yv0RBm2czMiZ+SqCGO7AvmOM3G/w2xhVPH+te0aKgFjmBvGlL1dH+ql2tgGO3MVrbb3jCKyvpgnxA==} - engines: {node: 20 || >=22} - '@isaacs/cliui@8.0.2': resolution: {integrity: sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==} engines: {node: '>=12'} @@ -13581,10 +13573,6 @@ packages: resolution: {integrity: sha512-ethXTt3SGGR+95gudmqJ1eNhRO7eGEGIgYA9vnPatK4/etz2MEVDno5GMCibdMTuBMyElzIlgxMna3K94XDIDQ==} engines: {node: 20 || >=22} - minimatch@10.1.1: - resolution: {integrity: sha512-enIvLvRAFZYXJzkCYG5RKmPfrFArdLv+R+lbQ53BmIMLIry74bjKzX6iHAm8WYamJkhSSEabrWN5D97XnKObjQ==} - engines: {node: 20 || >=22} - minimatch@10.2.4: resolution: {integrity: sha512-oRjTw/97aTBN0RHbYCdtF1MQfvusSIBQM0IZEgzl6426+8jSC0nF1a/GmnVLpfB9yyr6g6FTqWqiZVbxrtaCIg==} engines: {node: 18 || 20 || >=22} @@ -19873,12 +19861,6 @@ snapshots: optionalDependencies: '@types/node': 20.17.6 - '@isaacs/balanced-match@4.0.1': {} - - '@isaacs/brace-expansion@5.0.0': - dependencies: - '@isaacs/balanced-match': 4.0.1 - '@isaacs/cliui@8.0.2': dependencies: string-width: 5.1.2 @@ -25529,7 +25511,7 @@ snapshots: '@ts-morph/common@0.27.0': dependencies: fast-glob: 3.3.3 - minimatch: 10.1.1 + minimatch: 10.2.4 path-browserify: 1.0.1 '@turf/boolean-point-in-polygon@7.1.0': @@ -28542,7 +28524,7 @@ snapshots: eslint: 8.57.1 eslint-import-resolver-node: 0.3.9 eslint-import-resolver-typescript: 3.6.3(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-plugin-import@2.31.0)(eslint@8.57.1) - eslint-plugin-import: 2.31.0(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint@8.57.1) + eslint-plugin-import: 2.31.0(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-typescript@3.6.3)(eslint@8.57.1) eslint-plugin-jsx-a11y: 6.10.2(eslint@8.57.1) eslint-plugin-react: 7.37.2(eslint@8.57.1) eslint-plugin-react-hooks: 5.1.0(eslint@8.57.1) @@ -28566,7 +28548,7 @@ snapshots: debug: 4.4.3 enhanced-resolve: 5.17.0 eslint: 8.57.1 - eslint-module-utils: 2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1)(eslint@8.57.1) + eslint-module-utils: 2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint@8.57.1))(eslint@8.57.1))(eslint@8.57.1) eslint-plugin-import: 2.31.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-typescript@3.6.1)(eslint@8.57.1) fast-glob: 3.3.2 get-tsconfig: 4.8.1 @@ -28616,7 +28598,7 @@ snapshots: - eslint-import-resolver-webpack - supports-color - eslint-module-utils@2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1)(eslint@8.57.1): + eslint-module-utils@2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint@8.57.1))(eslint@8.57.1))(eslint@8.57.1): dependencies: debug: 3.2.7 optionalDependencies: @@ -28676,7 +28658,7 @@ snapshots: doctrine: 2.1.0 eslint: 8.57.1 eslint-import-resolver-node: 0.3.9 - eslint-module-utils: 2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1)(eslint@8.57.1) + eslint-module-utils: 2.12.0(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.6.1(@typescript-eslint/parser@6.21.0(eslint@8.57.1)(typescript@5.9.3))(eslint-import-resolver-node@0.3.9)(eslint-plugin-import@2.31.0(@typescript-eslint/parser@8.56.1(eslint@8.57.1)(typescript@5.9.3))(eslint@8.57.1))(eslint@8.57.1))(eslint@8.57.1) hasown: 2.0.2 is-core-module: 2.15.1 is-glob: 4.0.3 @@ -29788,7 +29770,7 @@ snapshots: glob@13.0.0: dependencies: - minimatch: 10.1.1 + minimatch: 10.2.4 minipass: 7.1.2 path-scurry: 2.0.0 @@ -31481,10 +31463,6 @@ snapshots: dependencies: brace-expansion: 2.0.1 - minimatch@10.1.1: - dependencies: - '@isaacs/brace-expansion': 5.0.0 - minimatch@10.2.4: dependencies: brace-expansion: 5.0.4 From bef4aab9559c1fed4b546d95dd9fc7ebd4eb5255 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Mon, 23 Mar 2026 17:20:48 -0700 Subject: [PATCH 27/30] Enhance QEMU emulator build workflow and add documentation - Updated the GitHub Actions workflow to include specific paths for the QEMU emulator build, ensuring that changes in the local emulator directory trigger the workflow. - Added a new README.md file for the QEMU local emulator, detailing its architecture, usage, and scripts for building and running the emulator, improving clarity for developers. - Documented the build process and optimizations taken, providing comprehensive guidance for users on how to utilize the local emulator effectively. These changes improve the usability and maintainability of the QEMU emulator setup, facilitating a smoother development experience. --- .github/workflows/qemu-emulator-build.yaml | 3 + docker/local-emulator/qemu/README.md | 121 +++++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100644 docker/local-emulator/qemu/README.md diff --git a/.github/workflows/qemu-emulator-build.yaml b/.github/workflows/qemu-emulator-build.yaml index 9b10347e29..db45d6e1a2 100644 --- a/.github/workflows/qemu-emulator-build.yaml +++ b/.github/workflows/qemu-emulator-build.yaml @@ -6,6 +6,9 @@ on: - main - dev pull_request: + paths: + - 'docker/local-emulator/**' + - '.github/workflows/qemu-emulator-build.yaml' workflow_dispatch: inputs: publish: diff --git a/docker/local-emulator/qemu/README.md b/docker/local-emulator/qemu/README.md new file mode 100644 index 0000000000..57ed713149 --- /dev/null +++ b/docker/local-emulator/qemu/README.md @@ -0,0 +1,121 @@ +# QEMU Local Emulator + +The local emulator packages the entire Stack Auth backend (PostgreSQL, Redis, ClickHouse, MinIO, Inbucket, Svix, QStash, Dashboard, and Backend) into a single QEMU virtual machine image. Users run it via the `stack emulator` CLI commands. + +## Architecture + +``` +Host machine + └─ QEMU VM (Debian 13 cloud image) + └─ Docker container (all-in-one image from ../Dockerfile) + ├─ PostgreSQL 16 + ├─ Redis 7 + ├─ ClickHouse + ├─ MinIO + ├─ Inbucket + ├─ Svix + ├─ QStash + ├─ Stack Dashboard (→ host:26700) + └─ Stack Backend (→ host:26701) +``` + +Only four services are exposed to the host via port forwarding: + +| Service | Host Port | Description | +|-----------|-----------|--------------------------| +| Dashboard | 26700 | Stack Auth dashboard UI | +| Backend | 26701 | Stack Auth API server | +| MinIO | 26702 | S3-compatible storage | +| Inbucket | 26703 | Email testing interface | + +All other services (PostgreSQL, Redis, ClickHouse, Svix, QStash) remain internal to the VM. + +## Scripts + +| Script | Purpose | +|--------------------|----------------------------------------------------------------| +| `build-image.sh` | Builds a QEMU disk image for a target architecture | +| `run-emulator.sh` | Manages the VM lifecycle: `start`, `stop`, `reset`, `status`, `bench` | +| `common.sh` | Shared helpers: host detection, QEMU binary selection, firmware lookup, ISO creation | + +## Building an Image + +```bash +# Build for current architecture +./docker/local-emulator/qemu/build-image.sh + +# Build for a specific architecture (arm64 or amd64) +./docker/local-emulator/qemu/build-image.sh arm64 + +# Build both +./docker/local-emulator/qemu/build-image.sh both +``` + +The build process: +1. Builds the all-in-one Docker image from `../Dockerfile` and exports it as a tarball +2. Downloads a Debian 13 cloud base image +3. Boots a QEMU VM with cloud-init provisioning (`cloud-init/emulator/user-data`) +4. Cloud-init loads the Docker image and runs a full startup cycle to warm caches +5. Shuts down and compresses the disk image to `images/stack-emulator-.qcow2` + +Default resources: 4 CPUs, 4096 MB RAM. Override with `EMULATOR_CPUS` / `EMULATOR_RAM_MB`. + +### Why a single Docker image? + +The `../Dockerfile` bundles all services into one image rather than using separate containers. This keeps the QEMU disk image size small — separate images would each carry their own base layers, significantly inflating the final qcow2. + +## Running the Emulator + +```bash +# Via CLI (recommended) +stack emulator start +stack emulator stop +stack emulator reset # wipe data +stack emulator status + +# Via script directly +EMULATOR_ARCH=arm64 ./docker/local-emulator/qemu/run-emulator.sh start +``` + +The VM uses an overlay disk (`run/vm/disk.qcow2`) on top of the base image, so data persists across stop/start cycles. Use `reset` to wipe the overlay and start fresh. + +### Hardware acceleration + +- **macOS**: Uses HVF (Hypervisor.framework) for native-arch VMs +- **Linux**: Uses KVM when available +- **Cross-arch**: Falls back to TCG (software emulation) — significantly slower + +## Optimizations Taken + +- **Single bundled Docker image** to minimize qcow2 size +- **Cloud-init provisioning** pre-warms all services during build so first boot is fast +- **Overlay disks** avoid copying the multi-GB base image on each start +- **Compressed qcow2** images (`-c` flag) reduce download size +- **Only 4 ports forwarded** to minimize host-side surface area + +## Possible Future Optimizations + +- External server for reads and writes to relative dir instead of full host access allowing snapshots + - Or copying the config file on start with --config-file enforced and writing the config file to host directory on stop + +## Updating the Image + +1. Make changes to the `../Dockerfile`, `../entrypoint.sh`, or cloud-init config +2. Rebuild: `./docker/local-emulator/qemu/build-image.sh ` +3. The CI workflow (`.github/workflows/qemu-emulator-build.yaml`) builds and publishes images on push to `main`/`dev` +4. Users pull the latest via `stack emulator pull` + +## Directory Layout + +``` +qemu/ +├── build-image.sh # Image builder +├── run-emulator.sh # VM lifecycle manager +├── common.sh # Shared utilities +├── cloud-init/ +│ └── emulator/ +│ ├── meta-data # VM instance metadata +│ └── user-data # Provisioning script +├── images/ # Built qcow2 images (gitignored) +└── run/ # Runtime state: overlay disk, PID, logs (gitignored) +``` From 771ed156fe7c760741924ab7c61422009e0fbda4 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Mon, 23 Mar 2026 18:06:20 -0700 Subject: [PATCH 28/30] Update CLI tests for emulator commands - Replaced the 'run' command expectation with 'start' in the emulator help output. - Removed the test for 'emulator build' help as it is no longer relevant. - Updated error message for invalid architecture in the 'emulator pull' test to improve clarity. These changes enhance the accuracy and relevance of the CLI tests for the emulator functionality. --- apps/e2e/tests/general/cli.test.ts | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/apps/e2e/tests/general/cli.test.ts b/apps/e2e/tests/general/cli.test.ts index 62dbe6ac5a..10430cb8f0 100644 --- a/apps/e2e/tests/general/cli.test.ts +++ b/apps/e2e/tests/general/cli.test.ts @@ -505,11 +505,10 @@ describe("Stack CLI — Emulator", () => { const { stdout, exitCode } = await runCliBare(["emulator", "--help"]); expect(exitCode).toBe(0); expect(stdout).toContain("pull"); - expect(stdout).toContain("run"); + expect(stdout).toContain("start"); expect(stdout).toContain("stop"); expect(stdout).toContain("reset"); expect(stdout).toContain("status"); - expect(stdout).toContain("build"); expect(stdout).toContain("list-releases"); }); @@ -522,16 +521,10 @@ describe("Stack CLI — Emulator", () => { expect(stdout).toContain("--repo"); }); - it("emulator build help shows arch option", async ({ expect }) => { - const { stdout, exitCode } = await runCliBare(["emulator", "build", "--help"]); - expect(exitCode).toBe(0); - expect(stdout).toContain("--arch"); - }); - it("emulator pull rejects invalid arch values", async ({ expect }) => { const { stderr, exitCode } = await runCliBareFromSource(["emulator", "pull", "--arch", "sparc"]); expect(exitCode).toBe(1); - expect(stderr).toContain("Invalid --arch: sparc; expected one of: arm64, amd64."); + expect(stderr).toContain("Invalid architecture: sparc. Expected arm64 or amd64."); }); it("emulator list-releases help shows repo option", async ({ expect }) => { From 950d74a6a52d5290ebe26cb221bc270a2b123390 Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Wed, 1 Apr 2026 15:17:10 -0700 Subject: [PATCH 29/30] remove stack.config.ts --- stack.config.ts | 64 ------------------------------------------------- 1 file changed, 64 deletions(-) delete mode 100644 stack.config.ts diff --git a/stack.config.ts b/stack.config.ts deleted file mode 100644 index faadd7e822..0000000000 --- a/stack.config.ts +++ /dev/null @@ -1,64 +0,0 @@ -export const config = { - "auth": { - "allowSignUp": true, - "password": { - "allowSignIn": true - }, - "otp": { - "allowSignIn": false - }, - "passkey": { - "allowSignIn": false - }, - "oauth": { - "accountMergeStrategy": "link_method", - "providers": { - "google": { - "type": "google", - "allowSignIn": true, - "allowConnectedAccounts": true - }, - "github": { - "type": "github", - "allowSignIn": true, - "allowConnectedAccounts": true - } - } - }, - "signUpRulesDefaultAction": "allow" - }, - "teams": { - "createPersonalTeamOnSignUp": false, - "allowClientTeamCreation": false - }, - "users": { - "allowClientUserDeletion": false - }, - "onboarding": { - "requireEmailVerification": false - }, - "apiKeys": { - "enabled": { - "team": false, - "user": false - } - }, - "domains": { - "allowLocalhost": true, - "trustedDomains": {} - }, - "rbac": { - "permissions": {}, - "defaultPermissions": { - "teamCreator": {}, - "teamMember": {}, - "signUp": {} - } - }, - "apps": { - "installed": {} - }, - "apps.installed.authentication.enabled": true, - "apps.installed.analytics.enabled": true, - "apps.installed.api-keys.enabled": true -}; From 7daea68cc7a62f71322470e7f774e5252d40dacd Mon Sep 17 00:00:00 2001 From: mantrakp04 Date: Fri, 3 Apr 2026 15:54:15 -0700 Subject: [PATCH 30/30] Increase QEMU smoke test timeout to 60 minutes --- .github/workflows/qemu-emulator-build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/qemu-emulator-build.yaml b/.github/workflows/qemu-emulator-build.yaml index db45d6e1a2..e4a42207ca 100644 --- a/.github/workflows/qemu-emulator-build.yaml +++ b/.github/workflows/qemu-emulator-build.yaml @@ -93,7 +93,7 @@ jobs: name: Smoke Test (${{ matrix.arch }}) needs: build runs-on: ubicloud-standard-8 - timeout-minutes: 30 + timeout-minutes: 60 strategy: fail-fast: false matrix: