Skip to content

Dev productivity improvements#1289

Merged
vklimontovich merged 8 commits into
newjitsufrom
dev-productivity-improvements
May 8, 2026
Merged

Dev productivity improvements#1289
vklimontovich merged 8 commits into
newjitsufrom
dev-productivity-improvements

Conversation

@vklimontovich
Copy link
Copy Markdown
Contributor

@vklimontovich vklimontovich commented May 7, 2026

Summary

  • Portless dev hosts. pnpm console:dev / pnpm ee-api:dev now serve at https://console.jitsu.localhost / https://ee.jitsu.localhost via portless. New pnpm ui:dev runs both in parallel through turbo.
  • Auto-detected branch suffix. When the current git branch is not the repo's default branch (resolved via git rev-parse origin/HEAD), the host becomes https://console-<branch>.jitsu.localhost / https://ee-<branch>.jitsu.localhost. Branch is sanitized for DNS (lowercased, [^a-z0-9-]-, capped at 30 chars). Pass --no-branch to opt out.
  • dev-scripts workspace package with a pnpm dev <subcommand> dispatcher. Bare pnpm dev runs turbo run dev as before.
  • pnpm dev copy-db --src URL --dst URL — full-database clone via pg_dump | psql. $ENV_VAR placeholders in URLs are resolved against process.env (populated by the layered .env.local loading — see below). Drops and recreates the dst database (prompts first; pass --clean-dst to skip the prompt). Skips all CREATE EXTENSION statements so dst doesn't need plv8/vector/etc. installed. Strips top-level SET statements from the dump so newer pg_dump versions can feed older dst servers. By default a few large log/audit tables (betteruptime.events, newjitsu.StatusChange, newjitsu.AuditLog, newjitsu.source_task) are structure-only — pass --all-tables to include their rows.
  • Layered .env.local loading, no wrapper. Single mechanism replaces five scattered loaders (dotenv-cli, tsx --env-file-if-exists, process.loadEnvFile, hand-rolled parser, plus duplicates). node --inspect and your debugger attach to the script's own process — no dotenv-cli parent in the tree. See "Env resolution" below.
  • AUTH_COOKIE_DOMAIN env var for NextAuth. Set to .jitsu.localhost in dev to share the session across console.jitsu.localhost / ee.jitsu.localhost and any branch hosts. Unset in production — when unset, behavior is unchanged (cookie stays host-only, no Domain attribute).
  • Next.js 16.2.3 → 16.2.6 (console + ee-api + bundle-analyzer + eslint-config-next). Patch bump.
  • allowedDevOrigins: ["*.jitsu.localhost"] in both Next configs so next dev doesn't block cross-origin _next/* requests from the portless hosts.
  • turbopack.root pinned to the repo root in both Next configs. Without it, Next walks up looking for a lockfile and on a worktree picks up the main checkout's pnpm-workspace.yaml, which makes Turbopack's module graph panic.
  • Drop unused root scriptsrelease / release:canary / release:build-scripts (the cli/build-scripts package they referenced doesn't exist) and the monorel + semantic-release* devDeps that backed them. Also drops the obsolete sync-bulker.sh.

Why dash style for branch subdomains

*.jitsu.localhost (single-level wildcard) covers every host in dash style, so a single Firebase Authorized-Domains entry is all that's ever needed. Dot style (console.<branch>.jitsu.localhost) would require either per-branch console clicks or a multi-level wildcard.

Implementation note: dev-scripts/src/bin/run-app.ts runs portless from a non-git scratch dir to sidestep portless's hardcoded dot-style worktree prefix, and wraps the actual command as bash -c "cd <ws> && <cmd>" so it executes at the workspace cwd.

Env resolution

Two .env.local layers, auto-loaded for every node process spawned by a pnpm script (later wins; existing process.env always wins; missing files skipped silently):

  1. ~/.jitsu/.env.local — shared across all worktrees of all branches (Firebase, Stripe, OIDC, GitHub OAuth — anything that doesn't change per branch).
  2. <repo>/.env.local — per-worktree (DATABASE_URL, NEXTAUTH_URL, AUTH_COOKIE_DOMAIN, anything branch-specific).

How: root .npmrc sets node-options=--require=env-preload. pnpm exports that as NODE_OPTIONS for every script-spawned node process. The preload (env-preload/preload-env.cjs, ~50 LOC of plain CommonJS) loads the shared file, then walks up from process.cwd() to find pnpm-workspace.yaml and load the repo .env.local. (Why a preload, not --env-file-if-exists: Node disallows --env-file* in NODE_OPTIONS for security; --require is allowed.)

What this drops: dotenv-cli from root + console + rotor; dotenv direct dep from ee-api + dev-scripts; the loadRepoEnv block in run-app.ts; the hand-rolled parser in dev-scripts/src/utils/env.ts; the dotenv-cli spawn in dev-scripts/src/index.ts; --env-file-if-exists flags in rotor + console scripts.

See CONTRIBUTING.md → Environment variables for the full story.

Setup

pnpm install   # portless is a workspace devDep; first run prompts once for sudo to bind 443 + trust the local CA

pnpm console:dev                      # → https://console.jitsu.localhost (on default branch)
pnpm console:dev --no-branch          # → https://console.jitsu.localhost (force, on any branch)
pnpm ee-api:dev                       # → https://ee.jitsu.localhost
pnpm ui:dev                           # both, in parallel

On a feature branch, the suffix is auto-applied: e.g. on dev-productivity-improvements you get https://console-dev-productivity-improv.jitsu.localhost.

In Firebase Console → Authentication → Settings → Authorized domains, add *.jitsu.localhost once.

For shared secrets across worktrees:

mkdir -p ~/.jitsu && chmod 700 ~/.jitsu
touch ~/.jitsu/.env.local && chmod 600 ~/.jitsu/.env.local
# put Firebase / Stripe / OIDC / GitHub OAuth keys here

In <repo>/.env.local for shared session across console/ee/branch hosts:

AUTH_COOKIE_DOMAIN=.jitsu.localhost

Test plan

  • pnpm install clean
  • pnpm dev help shows subcommands
  • On default branch, pnpm console:dev opens at https://console.jitsu.localhost
  • On a feature branch, pnpm console:dev opens at https://console-<branch>.jitsu.localhost
  • pnpm console:dev --no-branch opens at https://console.jitsu.localhost regardless of branch
  • Firebase login works on both with *.jitsu.localhost authorized
  • With AUTH_COOKIE_DOMAIN=.jitsu.localhost: log in once on console, ee-api recognizes the session
  • pnpm dev copy-db --src '$DATABASE_URL' --dst 'postgres://...local-dev' prompts to drop dst, then clones
  • pnpm dev copy-db --src ... --dst ... --clean-dst skips the prompt
  • pnpm dev copy-db --src ... --dst ... --all-tables includes data for the otherwise structure-only tables
  • Env resolution: pnpm exec node -e 'console.log(process.env.DATABASE_URL ? "SET" : "UNSET")' prints SET if <repo>/.env.local defines it; same from inside any workspace dir
  • Debugger sanity: pnpm console:dev boots; attach Node debugger; confirm no dotenv-cli parent in the process tree and process.env shows the layered values

Dev productivity rollup:

- Portless dev hosts: pnpm console:dev / ee-api:dev now serve at
  https://console.jitsu.localhost / https://ee.jitsu.localhost. New
  pnpm ui:dev runs both via turbo. Auto-detects the current branch and
  appends -<branch> for non-default branches; --no-branch opts out.
- dev-scripts workspace package with `pnpm dev <subcommand>` dispatcher.
  Bare `pnpm dev` runs `turbo run dev` as before.
- `pnpm dev copy-db --src URL --dst URL`: full-DB pg_dump | psql clone.
  Drops+recreates dst (prompts; --clean-dst skips). Skips CREATE
  EXTENSION so dst doesn't need plv8/vector/etc. Strips top-level SET
  statements so newer pg_dump can feed older dst. Structure-only by
  default for large log/audit tables; --all-tables overrides.
- AUTH_COOKIE_DOMAIN env var for NextAuth: opt-in. Unset = unchanged
  (host-only cookie). Set to .jitsu.localhost in dev to share session
  across console/ee/branch hosts.
- Next.js 16.2.3 → 16.2.6. allowedDevOrigins=*.jitsu.localhost so
  next dev doesn't block portless hosts. turbopack.root pinned to
  repo root to avoid worktree picking up the main checkout's lockfile.
- Drop unused release / release:canary / release:build-scripts root
  scripts and the monorel + semantic-release* devDeps. Drop obsolete
  sync-bulker.sh.
@vklimontovich vklimontovich force-pushed the dev-productivity-improvements branch from 777b21f to b6aff45 Compare May 7, 2026 21:24
vklimontovich added a commit that referenced this pull request May 8, 2026
Mirrors the fix from #1289 so the OpenAPI dev work isn't blocked when
running console:dev from a git worktree. Turbopack panicked because Next
was walking up past the worktree and picking the main checkout's
pnpm-workspace.yaml as the workspace root.
Single mechanism replaces five scattered .env loaders (dotenv-cli wrappers,
tsx --env-file-if-exists flags, process.loadEnvFile, hand-rolled parser).
Layered hierarchy (later wins; existing process.env always wins; missing
files skipped):

  1. ~/.jitsu/.env.local     (shared across worktrees)
  2. <repo>/.env.local       (per-worktree)

Implementation: root .npmrc sets node-options=--require=...preload-env.cjs,
so pnpm exports NODE_OPTIONS for every script-spawned node process. The
preload walks up from cwd to find pnpm-workspace.yaml and load the repo
.env.local. (Node disallows --env-file* in NODE_OPTIONS for security;
--require is allowed.)

No wrapper means `node --inspect` and the debugger attach to the script's
own process directly — no dotenv-cli parent in the tree.

Drops dotenv-cli + dotenv direct deps from root, console, ee-api, rotor,
dev-scripts. CONTRIBUTING.md gains an "Environment variables" section.
Decouples the .env.local preload from dev-scripts (which is for tools like
copy-db). New top-level workspace package @jitsu-internal/env-preload owns
the preload module exclusively. .npmrc references it by package name, so
resolution from any sub-workspace still walks node_modules up to the root.

No behavior change.
@vklimontovich vklimontovich marked this pull request as ready for review May 8, 2026 15:22
Copy link
Copy Markdown

@jitsu-code-review jitsu-code-review Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found two correctness/safety issues in the new copy-db tool that should be fixed before merge.

Comment thread dev-scripts/src/commands/copy-db.ts Outdated
Comment thread dev-scripts/src/commands/copy-db.ts
Two fixes from PR review:

1. makeSetStripper used to drop every top-level SET, including
   row_security and check_function_bodies that real restores need.
   Narrow to a denylist (just transaction_timeout for now — the
   PG 17→<17 incompatibility we hit). Other SETs pass through.

2. runCopyDb now refuses to start when --src and --dst resolve to the
   same (host, port, database). Without this, a typo plus --clean-dst
   would drop the only copy before pg_dump runs.
Copy link
Copy Markdown

@jitsu-code-review jitsu-code-review Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one correctness issue that can cause destructive data loss in copy-db when src/dst use different host aliases for the same DB endpoint.

Comment thread dev-scripts/src/commands/copy-db.ts
Copy link
Copy Markdown

@jitsu-code-review jitsu-code-review Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one correctness issue in the new env layering logic that can cause the wrong environment values to be used in dev.

Comment thread env-preload/preload-env.cjs Outdated
Comment thread env-preload/preload-env.cjs Outdated
Copy link
Copy Markdown

@jitsu-code-review jitsu-code-review Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one correctness regression in the new env preload path.

  • currently prevents repo-level from overriding because it checks the live while loading each layer. That reverses the intended precedence documented in the PR (shared first, then repo override).

Please adjust the loader so only pre-existing environment variables are protected, while later file layers can override earlier file layers.

Copy link
Copy Markdown

@jitsu-code-review jitsu-code-review Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one correctness regression in the new env preload path.

  • env-preload/preload-env.cjs currently prevents repo-level .env.local from overriding ~/.jitsu/.env.local because it checks the live process.env while loading each layer. That reverses the intended precedence documented in the PR (shared first, then repo override).

Please adjust the loader so only pre-existing environment variables are protected, while later file layers can override earlier file layers.

The old guard `if (process.env[key] !== undefined) continue` skipped any
key that already existed in process.env at the time of the check. After
~/.jitsu/.env.local loaded, every shared key was in process.env, so the
repo-level <repo>/.env.local could never override it — backwards from
the documented "later wins" precedence.

Snapshot the shell's original env keys at startup; only those are
untouchable. Keys set by a previous file load are overwritable, so the
order is now:

  shell  >  <repo>/.env.local  >  ~/.jitsu/.env.local

Verified end-to-end: repo overrides home for shared keys; home-only
keys still load; shell still wins over both.
Copy link
Copy Markdown

@jitsu-code-review jitsu-code-review Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one correctness/safety issue that can still cause destructive data loss in copy-db.

Comment thread dev-scripts/src/commands/copy-db.ts
@vklimontovich vklimontovich dismissed stale reviews from jitsu-code-review[bot], jitsu-code-review[bot], jitsu-code-review[bot], jitsu-code-review[bot], jitsu-code-review[bot], and jitsu-code-review[bot] May 8, 2026 16:12

Dismissing stale bot review: duplicate of the previously-resolved host-aliasing concern (#discussion_r3209668650). Author chose to keep the simple host/port/database string compare for this dev-only tool — see thread reply.

@vklimontovich vklimontovich merged commit 3e6292b into newjitsu May 8, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant