feat(cli, migrate): add stash encrypt commands + @cipherstash/migrate#357
Draft
feat(cli, migrate): add stash encrypt commands + @cipherstash/migrate#357
stash encrypt commands + @cipherstash/migrate#357Conversation
🦋 Changeset detectedLatest commit: 52da3ae The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
coderdan
force-pushed
the
encryption-migrations
branch
from
700009a to
90519c2
Compare
coderdan
added a commit
that referenced
this pull request
May 4, 2026
Two doc updates in support of #357 now that the rulebook package is gone: - `docs/plans/encryption-migrations.md`: drop "rulebook" references (5 of them) and the stale `packages/cli/src/commands/wizard/lib` paths. Re-point the agent-handoff bits at the post-#395 architecture: Claude / Codex / AGENTS.md handoffs from `init/steps/handoff-*.ts`, with the integration skill installed by init providing the per-stack guidance. Repoint `introspectDatabase` to its current home in `init/lib/introspect.ts`. - `/skills/stash-cli/SKILL.md`: add an `encrypt` section documenting every subcommand (`status`, `plan`, `advance`, `backfill`, `cutover`, `drop`) with flags, examples, and a one-line note on runner-prefix substitution so the docs are not pinned to npm. - `/skills/stash-encryption/SKILL.md`: add a "Column Migration Lifecycle" section covering the six-phase model (schema-added → dual-writing → backfilling → backfilled → cut-over → dropped), the three-source state model (`migrations.json` / `eql_v2_configuration` / `cs_migrations`), the CLI sequence, and the library `runBackfill` shape. Agents reading this skill now have the migration vocabulary they need. No CLI behaviour changes. Buckets 3+ from the audit (advance handoff integration, runner-aware help in encrypt commands, setup-prompt recommending `stash encrypt`, AGENTS-doctrine pointing at the CLI path) deferred until the encrypt-step UX has been reviewed.
Adds first-class support for migrating existing plaintext columns to
`eql_v2_encrypted` in production databases — the flow that currently has
no good answer in either Stack or Proxy land.
Per-column lifecycle:
schema-added → dual-writing → backfilling → backfilled → cut-over → dropped
State lives in three layers so Proxy interop stays clean:
- `.cipherstash/migrations.json` — repo-side intent (indexes, target phase)
- `eql_v2_configuration` — EQL intent, unchanged; Proxy reads as before
- `cipherstash.cs_migrations` — NEW append-only event log for per-column
runtime state (phase, backfill cursor, rows processed). Installed by
`stash db install`. Designed to upstream into EQL as `eql_v2_migrations`
in a later release so Stack and Proxy own it jointly.
New CLI commands under `stash encrypt`:
- status per-column table: phase, EQL state, indexes, progress, drift
- plan diff intent vs observed
- advance record a phase transition (dual-writing is user-declared)
- backfill chunked, resumable, idempotent; txn-per-chunk with checkpoint;
SIGINT-safe; uses user's encryption client via jiti dynamic
import; auto-detects single-column PK
- cutover `eql_v2.rename_encrypted_columns()` in a txn; optional Proxy
refresh via CIPHERSTASH_PROXY_URL
- drop generates a DROP COLUMN <col>_plaintext migration file
New package `@cipherstash/migrate` exposes the same primitives as a library
(`runBackfill`, `appendEvent`, `progress`, `renameEncryptedColumns`, …) so
users can embed backfill in their own workers/cron without the CLI process.
Design doc: docs/plans/encryption-migrations.md
Manual e2e script: packages/cli/scripts/e2e-encrypt.sh
Phase 1 scope: Protect/Stack client-side backfill. Proxy-mode backfill
(SQL-through-Proxy using the same cs_migrations state) is Phase 2.
Expand TypeDoc across the @cipherstash/migrate public API and the stash
encrypt command option interfaces. No behaviour change — docs only.
Highlights:
- BackfillOptions: each field now explains the three separate name
spaces (physical table/column vs. schema column key) and common
defaults (chunkSize = 1000, encryptedColumn = <col>_encrypted).
- BackfillCommandOptions: CLI flag semantics with an example of when
schemaColumnKey needs to differ from column.
- MigrationEvent / MigrationPhase: describes the event-vs-phase
mapping and the backfill_started/backfill_checkpoint distinction.
- EQL wrappers: explain that renameEncryptedColumns is the cut-over
primitive, and that reloadConfig must run through Proxy.
- installMigrationsSchema: documents why cs_migrations is kept
separate from eql_v2_configuration (CHECK constraint, global
state enum, write-frequency mismatch).
- Manifest: field-level documentation of cast_as values, index kinds,
and how targetPhase interacts with advance/plan/drop.
- Module-level @packageDocumentation in src/index.ts for TypeDoc's
package overview.
…stgres
Adds packages/migrate/src/__tests__/backfill.integration.test.ts —
gated on PG_TEST_URL so it skips in CI without a Postgres available.
Covers the full backfill state machine against a real transactional
Postgres using a stub encryption client (no CipherStash credentials
required):
- happy-path completion + correct terminal state event
- idempotency on re-run (row-level hash unchanged; zero new writes)
- resume from checkpoint after mid-run AbortSignal
- error event recorded + exception rethrown on encrypt failure
- pre-encrypted rows preserved (the `encrypted IS NULL` guard)
- empty-table fast path
- event log ordering (backfill_started → checkpoint* → backfilled)
- latestByColumn / progress readbacks
Run locally:
cd local && docker compose up -d
PG_TEST_URL=postgres://cipherstash:password@localhost:5432/cipherstash \\
pnpm -F @cipherstash/migrate test backfill.integration
…ation `stash db install --drizzle` now appends the cipherstash.cs_migrations schema DDL to the generated EQL migration file, so `drizzle-kit migrate` rolls the tracking table out to every environment alongside EQL itself. Before this change the drizzle path only wrote EQL SQL; the cs_migrations schema was installed directly against the connected DB (in the non-drizzle branch) and never appeared in migration history. That meant prod deploys running from drizzle migrations alone got EQL but no cs_migrations, and `stash encrypt ...` would fail with "schema cipherstash does not exist" until someone ran an out-of-band install. Also exports MIGRATIONS_SCHEMA_SQL from @cipherstash/migrate so other consumers can embed the DDL in their own migration pipelines.
…orts
loadEncryptionContext used to require the user's encryption client file
to export an EncryptedTable-shaped object (tableName + build()). Users
following the drizzle pattern typically only export the pgTable and the
initialised client, leaving the extractEncryptionSchema(...) result as
a non-exported const — which the loader couldn't see. Backfill would
then fail with "Table X was not found in the encryption client exports.
Available: (none)".
Now the loader does a second pass over module exports, detects drizzle
pgTables via Symbol.for('drizzle:Name'), dynamic-imports
@cipherstash/stack/drizzle, and calls extractEncryptionSchema() on each
to derive the EncryptedTable on the fly. Silently no-ops if the drizzle
subpath isn't installed (Supabase / generic projects are unaffected).
Manually-exported EncryptedTables still win over auto-derived ones
(the set-if-absent check preserves the explicit export).
Two correctness bugs in the backfill path, diagnosed from a real run
that wrote plaintext values through to the encrypted column:
1) The CLI defaulted `schemaColumnKey` to the plaintext column name
(`--column`). But under the drizzle convention the EncryptedTable's
column keys are the *encrypted* column names — because that's what
the user declared via `encryptedType('foo_encrypted', ...)`. With
the wrong key, `bulkEncryptModels` saw a model key that didn't
match any configured encrypted column and returned the models
unchanged. The runner then wrote the plaintext into the encrypted
column, which Postgres rendered as `(82.60)`-shaped composite values
because `eql_v2_encrypted` is a composite type. Default now uses
the encrypted column name.
2) Added a leak guard inside runBackfill: after bulkEncryptModels
returns, inspect `data[0][schemaColumnKey]`. Real ciphertext is
always an object (the EQL envelope with c/k/v fields); if we see
a primitive, throw with an actionable message that names the key
the schema should use. Prevents any future schema/key mismatch
from silently corrupting data — it fails loudly on the first chunk
before any write commits.
Updated the TypeDoc on BackfillOptions to make the two conventions
(drizzle-extracted vs handwritten encryptedTable) explicit.
… leak guard
Replace the hand-rolled object-shape check in runBackfill with the
canonical isEncryptedPayload helper already exported by @cipherstash/stack.
The helper checks for the actual EQL envelope shape (v, i, and either
c or sv) rather than just `typeof === 'object'`, so it also catches
non-null objects that happen to lack ciphertext fields.
Also validates every row in the returned chunk (not just the first)
and reports the offending primary key in the error message so a user
hitting a partial failure knows which row to look at.
Integration test stubs updated to return valid-shaped payloads
({v, i, c}) so they still exercise the write path under the new guard.
…ryption
pg's node driver returns `numeric` as a JS string (to preserve
precision), but an EncryptedTable schema declaring `dataType('number')`
expects a JS number — so bulkEncryptModels errored out with "Cannot
convert String to Float. String values can only be used with Utf8Str".
Fix is split across both packages:
- @cipherstash/migrate: new optional `transformPlaintext` callback on
BackfillOptions. Invoked on each row's plaintext before it goes into
the model passed to bulkEncryptModels. Library stays generic; does
not know anything about schemas.
- @cipherstash/cli: new `buildPlaintextCoercer` inspects
`tableSchema.build().columns[schemaColumnKey].cast_as` and returns
an appropriate coercer:
number / double / real / int / decimal → Number(string)
bigint / big_int → BigInt(string)
date / timestamp → new Date(string)
boolean → "true"/"false" → boolean
string / text / json / jsonb / unknown → identity
Null and undefined are always passed through unchanged.
The backfill "Backfilling x.y → y_enc" log line now also prints the
schema's cast_as value so a user diagnosing a type-coercion issue can
see immediately whether the coercer is reading the right dataType from
the EncryptedTable (vs. falling through to identity).
Refactored buildPlaintextCoercer to return { transform, castAs } so
the caller can log the detected value; behaviour unchanged.
… by protect-ffi Investigation into "Cannot convert String to Date" for a column with cast_as: 'date' turned up a genuine protect-ffi 0.21.2 limitation: its JsPlaintext wire enum has only String/Number/Boolean/JsonB variants — no JS Date representation. napi-rs serialises JS Date to ISO string via Date.toJSON, and the Rust side then refuses it because string values are only valid for Utf8Str columns. The Rust-internal NaiveDate / Timestamp types exist but have no JS-visible wire format. Not a tool bug; not fixable here. But running a backfill that will inevitably fail on the first chunk is a poor UX. Add a pre-flight check: if the schema declares cast_as 'date' or 'timestamp', print a warning explaining the FFI limitation and the mitigation (change to dataType: 'string' / ISO strings) and prompt before continuing. Accepts --yes-style confirmation via the standard clack confirm UI.
Two doc updates in support of #357 now that the rulebook package is gone: - `docs/plans/encryption-migrations.md`: drop "rulebook" references (5 of them) and the stale `packages/cli/src/commands/wizard/lib` paths. Re-point the agent-handoff bits at the post-#395 architecture: Claude / Codex / AGENTS.md handoffs from `init/steps/handoff-*.ts`, with the integration skill installed by init providing the per-stack guidance. Repoint `introspectDatabase` to its current home in `init/lib/introspect.ts`. - `/skills/stash-cli/SKILL.md`: add an `encrypt` section documenting every subcommand (`status`, `plan`, `advance`, `backfill`, `cutover`, `drop`) with flags, examples, and a one-line note on runner-prefix substitution so the docs are not pinned to npm. - `/skills/stash-encryption/SKILL.md`: add a "Column Migration Lifecycle" section covering the six-phase model (schema-added → dual-writing → backfilling → backfilled → cut-over → dropped), the three-source state model (`migrations.json` / `eql_v2_configuration` / `cs_migrations`), the CLI sequence, and the library `runBackfill` shape. Agents reading this skill now have the migration vocabulary they need. No CLI behaviour changes. Buckets 3+ from the audit (advance handoff integration, runner-aware help in encrypt commands, setup-prompt recommending `stash encrypt`, AGENTS-doctrine pointing at the CLI path) deferred until the encrypt-step UX has been reviewed.
…-force
The `stash encrypt advance --to <phase>` command was over-modelled: it
only ever did meaningful work for one transition (`dual-writing`), and
the user had to remember to invoke it as a prerequisite to backfill —
easy to miss, and a missed invocation just produced a confusing failure
mode where backfill couldn't tell whether dual-writes were live.
Fold the dual-write confirmation into `backfill` itself. The first run
against a column either prompts the user (interactive) or accepts
`--confirm-dual-writes-deployed` (non-interactive, with a loud warning),
appends the `dual_writing` event to `cs_migrations`, and proceeds. Re-runs
/ resumes are no-ops for the prompt — the bookmark is persisted.
Add `--force` for the recovery path. Drops the `<col>_encrypted IS NULL`
guard from both the SELECT and UPDATE so every plaintext row is
re-encrypted, including ones that already have a (potentially stale)
ciphertext. This handles the "I confirmed dual-writes but they weren't
actually live" failure mode where rows landed in plaintext only mid-
backfill, or where the application updated plaintext without dual-
writing the encrypted twin. Not destructive — re-encrypting a correctly-
encrypted value just rewrites the same payload — but expensive enough
that backfill prompts for explicit confirmation when --force is set.
The recovery run is recorded with `details.force = true` in
cs_migrations so audit-log queries can spot it.
Library changes (`@cipherstash/migrate`):
- `KeysetPageOptions.force` and `countUnencrypted(..., force)` drop the
encrypted-IS-NULL clause from the WHERE.
- `BackfillOptions.force` plumbs the flag through `runBackfill` and the
chunk writer; UPDATE WHERE drops `t.<enc> IS NULL` when force.
- The `backfill_started` event includes `force: true` in details.
CLI changes:
- Delete `commands/encrypt/advance.ts`.
- `commands/encrypt/backfill.ts` gains `confirmDualWritesDeployed`,
`force`, and an `ensureDualWritesDeployed` guard that handles all
three paths (already advanced, interactive prompt, non-interactive
flag) and appends the `dual_writing` event on first acceptance.
- `bin/stash.ts` drops the `advance` route, parses
`--confirm-dual-writes-deployed` and `--force` for `backfill`,
removes `encrypt advance` from the help banner.
- `commands/encrypt/{status,plan}.ts` prose hints updated to point at
`backfill` instead of `advance`.
Docs / skills:
- `docs/plans/encryption-migrations.md` — drop the standalone advance
section; add a "Dual-write confirmation, folded into backfill"
section explaining the rationale; update the verification flow.
- `skills/stash-cli/SKILL.md` — drop the `encrypt advance` subsection;
rewrite the `encrypt backfill` subsection with the new flags and the
dual-write precondition explanation.
- `skills/stash-encryption/SKILL.md` — update the lifecycle CLI
sequence to fold phases 2 + 3 into one `backfill` command; add the
`--force` recovery path.
- `packages/cli/scripts/e2e-encrypt.sh` — drop the `advance` step;
use `--confirm-dual-writes-deployed` in the non-interactive backfill.
- `.changeset/encryption-migrations.md` — describe the new shape;
also fix the package name in the frontmatter (`@cipherstash/cli` →
`stash` post-rename).
Out of scope: `stash encrypt update` for re-encrypting an already
cut-over column when the EQL configuration changes — handled in the
next change.
coderdan
force-pushed
the
encryption-migrations
branch
from
4989081 to
671c2e0
Compare
… drizzle lifecycle worked example
Three coupled changes that fix the failure mode reported on the spike
project where the agent stopped at the "stop and ask" rule because the
setup prompt didn't route it toward `stash encrypt` for live-data
columns.
setup-prompt.ts is rewritten from "imperative TODO list" to
"orient and ask". The new prompt:
- Tells the agent its FIRST response is a routing question, not an
edit. The agent must orient the user with the two paths and ask
which they want before touching anything.
- Names every installed skill with a one-line purpose so the user
can see what's available.
- Describes path 1 (new encrypted column from scratch) and path 3
(migrate an existing populated column via `stash encrypt
backfill/cutover/drop`) explicitly, with the right CLI commands
inline (runner-aware, per package manager).
- Names path 2 (convert in place) as not supported and explains
why, so the agent routes to path 3 if the user asks for it.
- Preserves the stop-and-ask invariants but ties them to the
unsupported-path 2 case.
build-schema.ts no longer prompts the user to pick which columns
to encrypt during init. Deciding which columns to encrypt is the
user's choice in conversation with their agent — not a question to
answer at init time, because path 1 and path 3 need different
treatment and init can't tell which the user wants. Init now
always writes a placeholder encryption client; introspection-based
codegen is removed.
utils.ts:generatePlaceholderClient is rewritten. Used to synthesise
a fully-formed `pgTable('users', { email, name })` mirror of the
DB. That left users with two parallel definitions (real schema
file + synthesised stub) that the agent had to reconcile blind.
The new placeholder is a heavily-commented file showing the
encryption-client patterns inline (path 1 and path 3 examples
for both Drizzle and generic), exporting `Encryption({ schemas:
[] })` so the encrypt commands surface a clear error pointing
back at this file. The agent's job is to declare encrypted
columns directly in the user's real schema files and update this
file to reference them.
write-context.ts:buildContextFile no longer throws on empty
schemas. Init's `state.schemas` is now `[]` after the refactor.
skills/stash-drizzle/SKILL.md gets a new "Migrating an Existing
Column to Encrypted" section with a phase-by-phase Drizzle worked
example: schema-add (encryptedType twin column, nullable, generate
+ apply migration), dual-write (insert/update code change),
backfill (`stash encrypt backfill`), cutover (rename swap, switch
schema and read paths), drop (generated migration removes
plaintext). Mirrors the lifecycle vocabulary in stash-encryption.
setup-prompt tests rewritten to match the new orient-and-route
shape (12 tests). 165 unit tests pass; biome clean.
Out of scope (follow-ups tracked in conversation):
- stash-supabase skill needs the same worked example.
- Public docs repo needs the migration tool + lifecycle covered.
- Wizard's gateway prompt template needs the orient-and-route
vocabulary update.
- `stash encrypt update` for re-encrypting after EQL config
changes.
- `loadStashConfig` re-export from @cipherstash/migrate.
- AGENTS.md handoff validation in Cursor / Windsurf.
- Richer Codex skill structure (`scripts/`, `references/`).
…invariant Two fixes from a smoke-test run on the supatest spike project. Fix 1: backfill / drop never wrote `.cipherstash/migrations.json` The manifest was modelled as the *intent* leg of the three-source state model (intent in repo, EQL config in DB, runtime state in cs_migrations) but no CLI command actually wrote the file — `writeManifest` was exported from @cipherstash/migrate but never called from the CLI. Plan and status emitted "no manifest" forever and the drift-detection features were dead code. Wired: - New `upsertManifestColumn(table, column, cwd?)` in @cipherstash/migrate. Reads the existing manifest (or starts fresh), replaces the matching column entry under the named table, writes back. Preserves entries for other columns / other tables. - New `setManifestTargetPhase(table, columnName, phase, cwd?)` — no-op when the column isn't tracked yet, used by `drop` to bump intent forward. - `backfill.ts` calls `upsertManifestColumn` after the dual-write confirmation. The entry is derived from the encryption client's EncryptedTable schema (cast_as → manifest.castAs, configured index kinds → manifest.indexes); pkColumn flows through when the user passed `--pk-column`. targetPhase defaults to `cut-over`. Idempotent — re-runs replace the same entry. - `drop.ts` calls `setManifestTargetPhase(... 'dropped')` after the migration file is written, so the manifest reflects the user's commitment to fully removing the plaintext column. Cutover doesn't touch the manifest (current state lives in cs_migrations; the manifest is only intent). 10 new tests in @cipherstash/migrate covering upsert idempotence, target-phase update, and the no-op-when-untracked path. Fix 2: bundler-exclusion invariant promoted The skill mentioned that `@cipherstash/stack` must be excluded from bundling (it wraps a native FFI module) but in a single line buried in Installation. Claude missed it on the smoke test, then hit the runtime crash. - AGENTS-doctrine.md gains it as invariant #7 — the seventh "never break this" rule, alongside never-log-plaintext and jsonb-null-on-creation. Concrete config snippets for Next.js, webpack, esbuild, and Vite SSR included so the agent doesn't have to guess the field names. - stash-encryption skill's Installation section gets a more prominent callout (`> [!IMPORTANT]`) plus the same per-bundler snippets. - setup-prompt.ts adds it to path 1 step 1 ("if this is the first encrypted column in the project, configure the bundler exclusion first") and to path 3 schema-add as the same precondition. The exclusion now appears at every layer the agent reads: doctrine, skill, and project-specific action prompt. Test asserts `serverExternalPackages` and `@cipherstash/protect-ffi` appear in the rendered prompt.
`stash encrypt cutover` failed with "No pending configuration exists to encrypt" because `stash db push` wrote configs straight to `active`, skipping `pending` entirely. EQL's `rename_encrypted_columns()` requires a pending row to compute rename targets, so the documented six-phase lifecycle was unrunnable end-to-end. Reported in detail by Dan after a spike on the supatest project. Aligned the SDK with the EQL extension's native pending → encrypting → active state machine (the same flow Proxy uses for hot-reloads): - `db push` now writes the new config as `pending` when an `active` config already exists. First push (no active config) still writes directly to `active` since there's nothing to rename. Prints a clear "next step" note routing the user to the appropriate finalisation command. - `cutover` now runs the full lifecycle in one transaction: rename_encrypted_columns → migrate_config → activate_config. Pending is promoted to active alongside the physical rename. Verifies pending exists upfront with a clear error if not. - New `stash db activate` command for non-rename activations (path 1: brand-new encrypted column added to a project that already has an active config). Chains migrate_config + activate_config without any rename. Use after `db push` when no `<col>_encrypted` twin needs swapping. - `@cipherstash/migrate` exports new `migrateConfig`, `activateConfig`, and `discardPendingConfig` wrappers around the corresponding EQL functions. The `renameEncryptedColumns` docstring was wrong — it claimed idempotency when no renames are pending, but the underlying SQL throws if there's no pending row at all. Fixed. - `setup-prompt.ts` updated: path 1 now includes `db push → db activate`; path 3 walks through the schema flip + re-push between backfill and cutover so the agent knows to update the pending row. - Skill updates (stash-cli, stash-drizzle, stash-encryption) document the new pending/active flow explicitly. The stash-cli `db push` section gained a decision table for "what to run next" based on whether the change is additive or includes a rename. After this, the user's acceptance criterion holds: clean `init → schema edit → db push → encrypt backfill → encrypt cutover` flow ends with the rename applied, prior config marked inactive, new config active, and a `cut_over` event in `cs_migrations`.
Three issues from the spike's lifecycle smoke-test, all in the encrypt CLI. Bundled into one commit because they touch adjacent files. Issue 1: `encrypt drop` wrote a self-named timestamped migration (`20260504112456_drop_*.sql`) that drizzle-kit migrate refused to pick up — no journal entry, wrong prefix. Same shape `db install --drizzle` already gets right by shelling out to `drizzle-kit generate --custom`. Fix: detect drizzle and route through a new `packages/cli/src/commands/encrypt/drizzle-helper.ts` that wraps `drizzle-kit generate --custom --name=...`, locates the generated file, and writes the drop SQL into it. The migration now lands with a journal entry; `drizzle-kit migrate` applies it like any other migration. Non-drizzle projects keep the timestamped-file fallback (Prisma / raw-SQL paths planned). Issue 2: `encrypt cutover` ran `eql_v2.rename_encrypted_columns()` live and never told drizzle. Drizzle's `meta/_journal.json` and snapshot stayed pinned to the pre-rename shape, so the next `drizzle-kit generate` against the source produced a confused diff trying to recreate the old layout. Fix: after cutover succeeds (transaction committed), scaffold a follow-up custom drizzle migration containing idempotent `ALTER TABLE … RENAME COLUMN` statements wrapped in a `DO` block that checks whether `<col>_encrypted` still exists. On the source DB the rename already ran, so the block is a no-op and Drizzle's journal still records the migration; on a fresh restore the block performs the rename. Same file, both behaviours, reproducible. Non-drizzle projects skip the resync step (logged-only warning if scaffolding fails). Issue 3: `encrypt status` rendered `rowsProcessed/rowsTotal (pct%)` uniformly across every phase. The same fraction means different things at different points in the lifecycle, and `0/0 (100%)` for a `backfilled` column that needed no encrypting reads as nonsense. Fix: phase-aware framing for the PROGRESS column. `schema-added` shows `—`. `dual-writing` shows `(awaiting backfill)`. `backfilling` keeps the fraction. `backfilled` / `cut-over` / `dropped` show a plain completion marker instead of a degenerate ratio. Same data, phase-appropriate label. Wire `--migrations-dir <path>` through to cutover for projects with non-default drizzle out dirs. 166 tests pass; biome clean. Coverage check during dual-writing (the bug report's bonus suggestion — show "rows-with-both-columns / total" rather than just "awaiting backfill") needs a live SELECT against the user's table, not just the cs_migrations data we already have. Tracked as a follow-up; today's status surfaces phase awareness without new queries.
The post-install panel still recommended `stash wizard` as the headline path and showed a hand-rolled `client.encryptModel(record, table).run()` snippet — both stale post-#395 and post-#357. Replace with brief guidance that bridges install → agent handoff: two canonical "ask your agent X" phrasings (one per real path, migrate-existing vs add-new), a short note that the agent will do the schema edits and run the lifecycle commands, and a pointer at the skills + public docs. Same panel runs from any `db install` invocation — including the one init triggers in install-eql — so the new copy makes sense both during init's handoff and when `db install` is run standalone (where "your agent" can be any agent the user has open, or someone reading the lifecycle commands directly).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds first-class support for migrating existing plaintext columns to
eql_v2_encrypted— a production-shaped flow that today has no good answer in either Stack or Proxy land. Ships as a new CLI command group + library, usable by both Stack (Protect.js) and Proxy users.Lifecycle
Each column walks through:
State model (three layers, kept separate on purpose)
.cipherstash/migrations.json: desired columns, index set, target phase. Code-reviewable intent.eql_v2_configuration: unchanged. Proxy continues to read this as its source of truth.cipherstash.cs_migrations: append-only event log — per-column phase, backfill cursor, rows processed. Installed bystash db install. Designed to be upstreamed into EQL aseql_v2_migrationsin a later release so Stack and Proxy own it jointly.Why a new table instead of reusing
eql_v2_configuration: its CHECK constraint rejects custom metadata, its state enum is global (only one{active, pending, encrypting}at a time) so it can't represent multiple columns in different phases, and backfill-cadence writes would collide with Proxy's 60s config refresh. Full reasoning in the design doc.New CLI commands (under
stash encrypt)statusplan.cipherstash/migrations.json) vs observed stateadvance --to <phase>backfillcutovereql_v2.rename_encrypted_columns()in a txn; optional Proxy refresh viaCIPHERSTASH_PROXY_URLdropDROP COLUMN <col>_plaintextmigration fileNew package
@cipherstash/migrateExposes the same primitives (
runBackfill,appendEvent,progress,renameEncryptedColumns, …) so users can embed backfill in their own workers/cron without the CLI. Example inpackages/migrate/README.md.Phase 1 scope / Phase 2 follow-ups
cs_migrationsstate),stash db introspect --json/stash env setCLI subcommands, upstreamcs_migrations→eql_v2_migrationsin EQL.Test plan
pnpm --filter @cipherstash/migrate test— 14 unit tests pass (state DAO, manifest round-trip, SQL identifier quoting)pnpm --filter @cipherstash/cli test— all 126 existing tests still passpnpm -w build— full workspace builds cleanpnpm exec biome check <changed files>— clean./dist/bin/stash.js --helpshows the six newencryptsubcommandsbash packages/cli/scripts/e2e-encrypt.sh— seeds 5000-rowuserstable, runs install → advance → backfill (with SIGINT + resume) → status → cutover → drop. Requires CipherStash credentials in env.SELECT email FROM usersvia Proxy returns plaintext, direct Postgres returns ciphertext JSON.Design doc
docs/plans/encryption-migrations.md— full architecture including state-layer rationale, index-on-backfill implications, Proxy compatibility gotchas, and phased rollout.