feat(db): add model experiment schema for preview A/B tests#3299
Merged
Conversation
Adds tables for opt-in preview-model A/B experiments (see .plans/experimental-models-1.md): - model_experiment (with partial unique index per public_model_id where status IN (active, paused) and an 'active cannot be archived' check) - model_experiment_variant (positive-weight, unique label per experiment) - model_experiment_variant_version (immutable; upstream JSONB + sibling encrypted_api_key typed EncryptedData; (variant_id, effective_at desc) idx) - model_experiment_request (1:1 with microdollar_usage via usage_id PK FK cascade; prompt sha256 hash columns with sentinel-or-hex CHECK) Schema-only change. No gateway routing, admin UI, or tRPC yet.
RSO
approved these changes
May 18, 2026
markijbema
added a commit
that referenced
this pull request
May 18, 2026
Phase 5 of .plans/experimental-models-1.md. Schema-only PR #3299 provides the tables; this PR adds the editing surface. - ExperimentUpstreamSchema (strict subset of CustomLlmDefinitionSchema; no api_key, no extra_headers) - Redis key helpers: EXPERIMENTED_PUBLIC_IDS_REDIS_KEY and modelExperimentRedisKey(publicId), plus redisDel() helper - adminModelExperimentsRouter with CRUD + state machine (activate / pause / complete / setArchived / delete-on-draft) and variant ops (addVariant / removeVariant / updateVariantLabel / swapVariantVersion / rotateApiKey). encrypted_api_key is never returned by list/get/swap/rotate; admin selects enumerate non-key columns explicitly. BYOK_ENCRYPTION_KEY missing rejects key-touching ops. - React Query hooks + ModelExperimentsContent (list + inline detail + Monaco hot-swap dialog + rotate-key dialog). - Mounted as a 4th tab inside /admin/gateway with a redirect at /admin/model-experiments mirroring /admin/custom-llms. Cache invalidation runs on every routing-affecting mutation (per-public-id cache + recomputed membership set). Phase 3 gateway routing reads these caches; until that lands, invalidation is a no-op for routing but writes the membership set so it's ready.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the database schema for the preview-model A/B experiment system described in
.plans/experimental-models-1.md. Schema-only — no gateway routing, admin tRPC, or UI yet. Splitting this out first so subsequent phases don't fight rebase conflicts on migration journal/snapshot files.New tables (
packages/db/src/schema.ts+0134_black_union_jack.sql):model_experiment— partial unique index onpublic_model_idwherestatus IN ('active', 'paused')so only one routing-relevant experiment per public id can exist.statusCHECK constrains todraft | active | paused | completed. CHECK forbiddingactive + is_archived = true.model_experiment_variant—(experiment_id, label)unique,weight > 0CHECK. The slot identity for a bucketed user.model_experiment_variant_version— immutable per-variant version row. Holdsupstream jsonb(validated byExperimentUpstreamSchemain app code in a later phase) and a siblingencrypted_api_key jsonbtypedEncryptedData(same shape asbyok_api_keys.encrypted_api_key) so secrets stay column-isolatable.(variant_id, effective_at desc)index for "current version" lookups.model_experiment_request— 1:1 withmicrodollar_usageviausage_idPK FK withon delete cascade. Stores prompt sha256 hashes (or reserved sentinel values__absent__/__failed__/__deleted__) — content goes to R2 in a later phase.(variant_version_id, created_at)index for per-RC reports, partial index onclient_request_idfor feedback joins.The plan doc has been updated with an "Implementation Status" table at the top so a follow-up agent can quickly see what's already landed.
Verification
pnpm drizzle generateproduced the migration with no manual edits.pnpm test:db(postgres +drizzle-kit migrate) applies the migration cleanly on a fresh database.pnpm --filter @kilocode/db typecheckpasses.pnpm formatclean.Visual Changes
N/A — backend schema only.
Reviewer Notes
upstreamis plainjsonbfor now. Strict zod validation (ExperimentUpstreamSchema) lives in app code added in a later phase. Storing as plainjsonbhere keeps the migration simple and lets the schema evolve without column shape changes.encrypted_api_keyis intentionally a sibling column rather than embedded inupstreamso reporting views and admin selects can simply omit the column — see plan §"API Keys".softDeleteUserupdate required (per plan §"GDPR and consent", experiment data is governed by a dedicated retention policy, not the default soft-delete; that test lands with the GDPR-test phase).