feat(dsh-storage): complete session mirror (Prisma 7, 4 DBs) + CI suite - #2
Merged
Conversation
…abase providers
Mirror the session event stream into ai_messages / ai_chat_histories:
- projector verified against the dsh persistence catalog: envelope
{type,seq,time,data}, usage from assistant/message data.usage,
tool/result callId from message.source.callId with {name,code} errors,
surface events only; session/title folds into the session row
- effect-based lifecycle for the cordis fork (no ready/dispose events):
init at plugin load, drain in-flight writes at session/flush and on
fiber unload so one-shot runs never lose tail rows
- deterministic-PK upserts (no JSON-path filtering): portable across
connectors and race-free under event bursts
- four schema variants (mysql/postgresql/sqlite/sqlserver) with clients
pre-generated at build time (Prisma 7 driver adapters: mariadb, pg,
libsql, mssql) — consumers never run prisma generate; SQL Server uses
text columns with provider-derived JSON serialization
- 41 vitest cases: projector, plugin wiring on a real cordis Context,
backend against mocked prisma/adapters
…unners - integration.yml: detect-matrix discovers packages with .github/scripts/integration/<pkg>.mjs; base leg boots the real dsh headless profile, runs one deepseek-v4-flash query that reads a seeded marker file, and asserts the full tool round-trip in SQLite; <pkg>.providers adds backend-only legs (mysql/postgresql/sqlserver service containers, db push + real backend writes/reads, no secrets) - labeler.yml: rule labels (area/type/size/platform) on every PR, P0-P3 proposed via the gateway for same-repo PRs - dsh-review.yml: two-axis (Standards/Spec) review via dsh headless, trusted base checkout only, PR content as untrusted data, all model tools disabled, P0/P1 fail the job - ci.yml: moved to ubuntu-latest with lint/typecheck/build/test; fork PRs get all secrets-free stages - docs: README + AGENTS.md (unit-test convention, Prisma 7 pre-generated clients, E2E/.providers conventions, workflow runner policy)
…rkflow at eval time The runner context is not available in job-level env: GitHub rejected the expression and the whole workflow failed with zero job logs, so the E2E never triggered. The scenario script already defaults DSH_HOME under RUNNER_TEMP. Also: pin actionlint into the lint job so workflow-file errors fail fast, and document both rules in AGENTS.md.
- Pack step: pnpm pack may print an absolute path (not a bare filename) — always basename it; the doubled prefix made dsh plugin add ENOENT - Install step: pnpm/action-setup's PNPM_HOME shadows npm-global, so the GITHUB_PATH prepend left pnpm 9 in charge; override PATH via GITHUB_ENV instead. Also pass npm's --allow-scripts list for dsh's subprocess helper
The model may complete the tool round-trip with no wrap-up prose (observed flake: exit 0, empty stdout). Retry the query once, then let the database be the acceptance gate: assistant rows may have empty text, and repeating the marker in prose is a warning, not a failure.
CI showed deepseek-v4-flash stochastically emits malformed tool calls (empty tool name) through the gateway, looping until the turn ends empty. Retry up to 3 attempts with the marker-bearing tool row in the database as the success gate, wiping and re-pushing the sqlite file per attempt so the single-session assertions still hold.
The model itself reported the read tool as unavailable: with reasoning
enabled, the gateway stochastically garbles tool calls into an empty name
("unknown tool""""), looping until the turn ends empty (3 of 4 runs).
thinking: disabled is the adapter's deployment lock and keeps the wire
format simple; the storage E2E does not need thinking.
Pin DSH_HOME under runner temp (deterministic, artifact-able), dump the zstd-compressed session event log after every run (request/header shows what the model actually saw — essential when tool calls come back empty), and upload the sqlite db + session logs as artifacts on every outcome.
The adapter's !== void 0 guards let explicit empty-string id/name in later gateway deltas clobber the accumulated tool-call identity (proven via the session event log: correct id/name in the first delta, empty in block-end, then a 400 for missing tool_call_id). sed the installed package to truthy guards with a grep that fails loudly when upstream code drifts. Remove once dsh ships the fix.
… secrets - per-session processing chain: event handling for one session is serialized through a promise chain, so rollup snapshots complete in event order (latest-wins under bursts) and the first event of an unknown session seeds its accumulator from the stored row via the new StorageBackend.readSession — resume no longer rewrites history from zero - pin @prisma/client + adapters to ^7.9.1: pre-generated clients only work with the runtime they were generated from (7.0.0 breaks) - engines >=20.19 (Prisma 7 floor) - integration.yml: secrets only enter the Stage B step env (job env carries just the availability flag), keeping PR-modified install/build scripts away from them - docs: README upgrade note (cuid → hash PK), preview.png scaffold gap, usageOf comment on failed-request usage
…ded usage bookkeeping - integration.yml split into scenario (base LLM leg, gated by the integration environment) and scenario-backends (secrets-free provider legs, ungated incl. fork PRs) — reusable callers can't carry environment, so the jobs are inline - correct the secrets-blanking idiom: GitHub treats '' as falsy, so x && '' || y leaks; !x && y || '' is the safe form - legacy history rows are absorbed in place: readSession returns the row's PK and upsertSession honors row.pk — no duplicate session rows - usage bookkeeping tracks only the latest per-step sample (samples of a step are adjacent in rc.7), persisted as the namespaced dsh-storage:lastUsage metadata key with a shape guard; existing metadata fields are merged, never overwritten - dispose cleanup is enqueued into the session chain (a synchronous delete raced the queued tasks into a no-op) with the settle-guarded chain delete preserved - dsh-session peer pinned exactly to the verified 0.1.0-rc.7 - 58 vitest cases
The per-session serialization chain (added for burst ordering) removed the original reason session rows needed hash PKs at all — writes for one session are now strictly ordered, so findFirst(sessionId) + update-or- create is race-free again. Benefits: cuid primary keys stay aligned with the source project and the early scaffold, legacy rows are continued instead of absorbed through a pk-passthrough, and the historyPk/fallback machinery is deleted outright. Message rows keep their deterministic hash PKs (no natural unique column, and JSON-path lookup is not portable to SQLite).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Complete dsh-storage implementation
{type, seq, time, data}; per-step replacement usage accounting (usage chunks andassistant/message, cache buckets included, failed steps covered, persisted across resume);tool/resultcallId frommessage.source.callIdwith errors projected to exactly{name, code}; only the three surface events produce rows;session/titlefolds into the session row.ready/disposeevents): backend init at plugin load; in-flight mirror writes drained at the awaitedsession/flushcheckpoint and on fiber unload, so one-shot headless runs never lose tail rows.sha256(sessionId + messageId)): no JSON-path filtering (PostgreSQL/MySQL-only feature), portable across connectors and race-free under event bursts.adapter-mariadb,adapter-pg,adapter-libsql,adapter-mssql). Clients are pre-generated at package build time and shipped — consumers never runprisma generate. SQL Server has no PrismaJsontype, so its variant maps JSON columns to text and the backend serializes on write (derived fromprovider, no config knob).archivedAtwas dropped from the schema by owner decision.Context(mocked backend), and the backend against mocked prisma/adapters — covering the no-throw seam, write draining, per-session serialization, resume seeding, usage replacement, URL decoding, and credential redaction.CI suite (all GitHub-hosted runners; free for this public repo)
.github/scripts/integration/<pkg>.mjs. The base leg boots the real dsh headless profile, asksdeepseek-v4-flashto read a seeded marker file, and asserts the fulluser → assistant(tool-call) → tool/result → assistantchain in SQLite (marker content, callId linkage,message_count >= 4). A sibling<pkg>.providersfile adds backend-only legs (mysql/postgresql/sqlserver service containers:db push+ real backend writes/reads) that need no secrets and therefore also run on fork PRs.area/*,type/*,size:*,platform/*) on every PR; P0–P3 proposed via the integration gateway for same-repo PRs only. (Starts firing once this workflow lands onmain—pull_request_targetrestriction.)pull_request_targetactivation note.)ubuntu-latest; lint (biome + pinned actionlint on workflow files), typecheck, build, and the vitest suites. Fork PRs get every secrets-free stage.First verified end-to-end run
SCENARIO_OK (messages=5, toolCalls=1, title="读取文件并复述内容")— the base leg made a realdeepseek-v4-flashround-trip, read the marker file through a tool call, and every database assertion passed. All five integration legs are green.Verified locally vs. only provable in CI
DatabaseBackend; dry-run of the full headless flow up to the LLM call.