Skip to content

feat(dsh-storage): complete session mirror (Prisma 7, 4 DBs) + CI suite - #2

Merged
Weaxs merged 13 commits into
mainfrom
feat/dsh-storage-complete
Aug 25, 2026
Merged

feat(dsh-storage): complete session mirror (Prisma 7, 4 DBs) + CI suite#2
Weaxs merged 13 commits into
mainfrom
feat/dsh-storage-complete

Conversation

@Weaxs

@Weaxs Weaxs commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Complete dsh-storage implementation

  • Projector verified against the dsh persistence catalog: envelope {type, seq, time, data}; per-step replacement usage accounting (usage chunks and assistant/message, cache buckets included, failed steps covered, persisted across resume); tool/result callId from message.source.callId with errors projected to exactly {name, code}; only the three surface events produce rows; session/title folds into the session row.
  • Effect-based lifecycle (this cordis fork has no ready/dispose events): backend init at plugin load; in-flight mirror writes drained at the awaited session/flush checkpoint and on fiber unload, so one-shot headless runs never lose tail rows.
  • Deterministic-PK upserts (sha256(sessionId + messageId)): no JSON-path filtering (PostgreSQL/MySQL-only feature), portable across connectors and race-free under event bursts.
  • Four database providers (MySQL / PostgreSQL / SQLite / SQL Server): one Prisma schema variant each, driven through Prisma 7 driver adapters (adapter-mariadb, adapter-pg, adapter-libsql, adapter-mssql). Clients are pre-generated at package build time and shipped — consumers never run prisma generate. SQL Server has no Prisma Json type, so its variant maps JSON columns to text and the backend serializes on write (derived from provider, no config knob). archivedAt was dropped from the schema by owner decision.
  • 56 vitest cases: pure projector logic, plugin wiring on a real cordis Context (mocked backend), and the backend against mocked prisma/adapters — covering the no-throw seam, write draining, per-session serialization, resume seeding, usage replacement, URL decoding, and credential redaction.

CI suite (all GitHub-hosted runners; free for this public repo)

  • integration.yml — the matrix is auto-discovered from .github/scripts/integration/<pkg>.mjs. The base leg boots the real dsh headless profile, asks deepseek-v4-flash to read a seeded marker file, and asserts the full user → assistant(tool-call) → tool/result → assistant chain in SQLite (marker content, callId linkage, message_count >= 4). A sibling <pkg>.providers file adds backend-only legs (mysql/postgresql/sqlserver service containers: db push + real backend writes/reads) that need no secrets and therefore also run on fork PRs.
  • labeler.yml — rule labels (area/*, type/*, size:*, platform/*) on every PR; P0–P3 proposed via the integration gateway for same-repo PRs only. (Starts firing once this workflow lands on mainpull_request_target restriction.)
  • dsh-review.yml — two-axis (Standards/Spec) DSH review: trusted base checkout only, PR content wrapped as untrusted data, all model tools disabled, P0/P1 findings fail the job. (Same pull_request_target activation note.)
  • ci.yml — moved to ubuntu-latest; lint (biome + pinned actionlint on workflow files), typecheck, build, and the vitest suites. Fork PRs get every secrets-free stage.

First verified end-to-end run

SCENARIO_OK (messages=5, toolCalls=1, title="读取文件并复述内容") — the base leg made a real deepseek-v4-flash round-trip, read the marker file through a tool call, and every database assertion passed. All five integration legs are green.

Verified locally vs. only provable in CI

  • Verified locally: four real databases (sqlite + docker mysql/postgresql/sqlserver) through DatabaseBackend; dry-run of the full headless flow up to the LLM call.
  • Only provable in CI (now green): the real LLM round-trip and workflow behavior on real PRs.

Weaxs added 12 commits August 20, 2026 16:08
…abase providers

Mirror the session event stream into ai_messages / ai_chat_histories:

- projector verified against the dsh persistence catalog: envelope
  {type,seq,time,data}, usage from assistant/message data.usage,
  tool/result callId from message.source.callId with {name,code} errors,
  surface events only; session/title folds into the session row
- effect-based lifecycle for the cordis fork (no ready/dispose events):
  init at plugin load, drain in-flight writes at session/flush and on
  fiber unload so one-shot runs never lose tail rows
- deterministic-PK upserts (no JSON-path filtering): portable across
  connectors and race-free under event bursts
- four schema variants (mysql/postgresql/sqlite/sqlserver) with clients
  pre-generated at build time (Prisma 7 driver adapters: mariadb, pg,
  libsql, mssql) — consumers never run prisma generate; SQL Server uses
  text columns with provider-derived JSON serialization
- 41 vitest cases: projector, plugin wiring on a real cordis Context,
  backend against mocked prisma/adapters
…unners

- integration.yml: detect-matrix discovers packages with
  .github/scripts/integration/<pkg>.mjs; base leg boots the real dsh
  headless profile, runs one deepseek-v4-flash query that reads a seeded
  marker file, and asserts the full tool round-trip in SQLite;
  <pkg>.providers adds backend-only legs (mysql/postgresql/sqlserver
  service containers, db push + real backend writes/reads, no secrets)
- labeler.yml: rule labels (area/type/size/platform) on every PR,
  P0-P3 proposed via the gateway for same-repo PRs
- dsh-review.yml: two-axis (Standards/Spec) review via dsh headless,
  trusted base checkout only, PR content as untrusted data, all model
  tools disabled, P0/P1 fail the job
- ci.yml: moved to ubuntu-latest with lint/typecheck/build/test; fork
  PRs get all secrets-free stages
- docs: README + AGENTS.md (unit-test convention, Prisma 7 pre-generated
  clients, E2E/.providers conventions, workflow runner policy)
…rkflow at eval time

The runner context is not available in job-level env: GitHub rejected the
expression and the whole workflow failed with zero job logs, so the E2E
never triggered. The scenario script already defaults DSH_HOME under
RUNNER_TEMP. Also: pin actionlint into the lint job so workflow-file
errors fail fast, and document both rules in AGENTS.md.
- Pack step: pnpm pack may print an absolute path (not a bare filename) —
  always basename it; the doubled prefix made dsh plugin add ENOENT
- Install step: pnpm/action-setup's PNPM_HOME shadows npm-global, so the
  GITHUB_PATH prepend left pnpm 9 in charge; override PATH via GITHUB_ENV
  instead. Also pass npm's --allow-scripts list for dsh's subprocess helper
The model may complete the tool round-trip with no wrap-up prose (observed
flake: exit 0, empty stdout). Retry the query once, then let the database
be the acceptance gate: assistant rows may have empty text, and repeating
the marker in prose is a warning, not a failure.
CI showed deepseek-v4-flash stochastically emits malformed tool calls
(empty tool name) through the gateway, looping until the turn ends empty.
Retry up to 3 attempts with the marker-bearing tool row in the database
as the success gate, wiping and re-pushing the sqlite file per attempt so
the single-session assertions still hold.
The model itself reported the read tool as unavailable: with reasoning
enabled, the gateway stochastically garbles tool calls into an empty name
("unknown tool""""), looping until the turn ends empty (3 of 4 runs).
thinking: disabled is the adapter's deployment lock and keeps the wire
format simple; the storage E2E does not need thinking.
Pin DSH_HOME under runner temp (deterministic, artifact-able), dump the
zstd-compressed session event log after every run (request/header shows
what the model actually saw — essential when tool calls come back empty),
and upload the sqlite db + session logs as artifacts on every outcome.
The adapter's !== void 0 guards let explicit empty-string id/name in later
gateway deltas clobber the accumulated tool-call identity (proven via the
session event log: correct id/name in the first delta, empty in block-end,
then a 400 for missing tool_call_id). sed the installed package to truthy
guards with a grep that fails loudly when upstream code drifts. Remove once
dsh ships the fix.
… secrets

- per-session processing chain: event handling for one session is
  serialized through a promise chain, so rollup snapshots complete in
  event order (latest-wins under bursts) and the first event of an
  unknown session seeds its accumulator from the stored row via the new
  StorageBackend.readSession — resume no longer rewrites history from zero
- pin @prisma/client + adapters to ^7.9.1: pre-generated clients only
  work with the runtime they were generated from (7.0.0 breaks)
- engines >=20.19 (Prisma 7 floor)
- integration.yml: secrets only enter the Stage B step env (job env
  carries just the availability flag), keeping PR-modified install/build
  scripts away from them
- docs: README upgrade note (cuid → hash PK), preview.png scaffold gap,
  usageOf comment on failed-request usage
…ded usage bookkeeping

- integration.yml split into scenario (base LLM leg, gated by the
  integration environment) and scenario-backends (secrets-free provider
  legs, ungated incl. fork PRs) — reusable callers can't carry
  environment, so the jobs are inline
- correct the secrets-blanking idiom: GitHub treats '' as falsy, so
  x && '' || y leaks; !x && y || '' is the safe form
- legacy history rows are absorbed in place: readSession returns the
  row's PK and upsertSession honors row.pk — no duplicate session rows
- usage bookkeeping tracks only the latest per-step sample (samples of a
  step are adjacent in rc.7), persisted as the namespaced
  dsh-storage:lastUsage metadata key with a shape guard; existing
  metadata fields are merged, never overwritten
- dispose cleanup is enqueued into the session chain (a synchronous
  delete raced the queued tasks into a no-op) with the settle-guarded
  chain delete preserved
- dsh-session peer pinned exactly to the verified 0.1.0-rc.7
- 58 vitest cases
@Weaxs
Weaxs deployed to integration August 22, 2026 02:55 — with GitHub Actions Active
The per-session serialization chain (added for burst ordering) removed
the original reason session rows needed hash PKs at all — writes for one
session are now strictly ordered, so findFirst(sessionId) + update-or-
create is race-free again. Benefits: cuid primary keys stay aligned with
the source project and the early scaffold, legacy rows are continued
instead of absorbed through a pk-passthrough, and the historyPk/fallback
machinery is deleted outright. Message rows keep their deterministic
hash PKs (no natural unique column, and JSON-path lookup is not portable
to SQLite).
@Weaxs
Weaxs deployed to integration August 22, 2026 07:08 — with GitHub Actions Active
@Weaxs
Weaxs merged commit c29b4b1 into main Aug 25, 2026
7 checks passed
@Weaxs
Weaxs deleted the feat/dsh-storage-complete branch August 29, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant