Skip to content

Epic: extract the Fetch-native ClickHouse client into a reusable package #630

Description

@BorisTyshkevich

Phases

  • 1 — freeze native Fetch, Response, and cancellation semantics
  • 2 — create @altinity/clickhouse-http and move low-level request/URL mechanics
  • 3 — extract progress-stream and late-exception protocol primitives
  • 4 — add consuming query APIs, ClickHouse errors, and stateless KILL QUERY
  • 5 — extract ClickHouse SQL quoting and type-expression grammar
  • 6 — simplify SQL Browser authentication integration around the package
  • 7 — migrate query execution and export, then delete generic client mechanics
  • 8 — stabilize the standalone package boundary and remove migration scaffolding

/ship owns this checklist and should tick a phase only after that phase's PR is merged and verified on origin/main.

Goal

Replace the rejected @clickhouse/client-web adoption path from #585 with a first-party, Fetch-native ClickHouse client package extracted from generic protocol code SQL Browser already maintains.

The package must preserve the behavior that made the official-driver adaptation unsuitable:

  • low-level request returns the exact native Response from fetch();
  • HTTP non-2xx responses resolve normally at the low-level boundary;
  • response bodies are never consumed unless the caller explicitly chooses a consuming API;
  • caller SQL is transmitted unchanged;
  • caller-provided Authorization is opaque and emitted unchanged;
  • the caller's own AbortSignal controls the real request for the entire response-body lifetime, including after headers arrive;
  • raw/binary bytes are never forced through text decoding;
  • ClickHouse progress streams and late-exception framing are reusable protocol primitives.

SQL Browser keeps application policy: OAuth/Basic credential acquisition, refresh, credential epochs, connection lifecycle, operation ownership, retry policy, per-tab session policy, result/view policy, product SQL, export UX, and workbench/dashboard state.

This is an architectural refactor. User-visible behavior and persisted formats must remain compatible.

/ship execution

This issue is intentionally phased in the format consumed by skills/ship.

Run:

/ship 630

or:

/ship 630 --planner chatgpt

Each phase is one /ship unit: approved plan, branch, PR, code-review certification, merge, checklist/ship-log reconciliation, then automatic continuation from freshly fetched origin/main.

The phases form one ordered dependency spine. There is no authored decision gate between them.

Why Phase 8 exists separately from #639

Phase 8 and #639 deliberately operate on opposite sides of a repository boundary.

Phase 8 is still entirely inside altinity-sql-browser

It proves that the extracted package is no longer architecturally dependent on SQL Browser before any cross-repository move occurs. It owns:

  • package-local build/typecheck/test scripts;
  • built ESM and TypeScript declaration surface;
  • npm pack and isolated tarball install/import/typecheck proof;
  • removal of temporary compatibility adapters/re-exports left by Phases 2–7;
  • removal of the rejected @clickhouse/client-web dependency and obsolete executable spike wiring;
  • architecture guards preventing package→SQL Browser imports and SQL Browser deep imports into package internals;
  • final in-repo ownership cleanup and architecture/ADR/wiki/changelog documentation;
  • a tested handoff document describing how to extract the package repository.

At the end of Phase 8, the package can be moved without changing its architecture or API, but it still physically lives under packages/clickhouse-http in this repository.

#639 is the actual cross-repository move and release

#639 starts only after Phase 8 is shipped. It owns:

  • creating Altinity/clickhouse-http;
  • moving package source/history into that repository;
  • adding standalone repository CI/release plumbing;
  • publishing/releasing a version;
  • changing SQL Browser from workspace dependency to released semver dependency;
  • deleting the workspace copy only after the external package passes SQL Browser integration/e2e tests.

The separation is intentional because the current /ship skill is scoped to altinity-sql-browser. If Phase 8 were removed, #639 would combine unresolved architecture cleanup, package isolation, repository creation, release, and consumer cutover in one cross-repository operation. That would make rollback and review substantially harder and would move architecture decisions outside the repo-scoped /ship proof loop.

If the future cross-repository shipping workflow can provide the same plan/review/test/rollback guarantees across both repositories, Phase 8 could theoretically be folded into #639. With the current tooling, keeping the in-repo stabilization phase is the safer boundary.

Architecture decision

If behavior answers “how does ClickHouse HTTP / Fetch work?”, it belongs in the package. If it answers “what should SQL Browser do?”, it stays in SQL Browser.

SQL Browser application policy
            |
            v
 @altinity/clickhouse-http
            |
            v
          Fetch
            |
            v
        ClickHouse

The package must never depend on SQL Browser application, UI, state, auth, workspace, dashboard, or editor modules.

Final ownership boundary

@altinity/clickhouse-http owns

  • endpoint/URL construction;
  • default_format, ClickHouse settings and caller-supplied query/protocol parameters;
  • direct fetch() invocation;
  • opaque Authorization emission;
  • caller AbortSignal passthrough;
  • native Response low-level API;
  • non-consuming HTTP success/error classification;
  • explicit JSON/text/progress response consumers;
  • ClickHouse HTTP exception parsing;
  • progress-bearing JSON-lines decoding;
  • X-ClickHouse-Exception-Tag / __exception__ late-exception framing;
  • stateless KILL QUERY wire operation;
  • ClickHouse SQL string/identifier quoting;
  • generic ClickHouse type-expression AST/parsing/canonicalization.

SQL Browser owns

  • OAuth discovery/PKCE/login/token storage;
  • Basic-auth probing and auth-mode selection;
  • token refresh and refresh single-flight;
  • credential epochs and stale-work fences;
  • login rejection versus query-level 401/403 classification;
  • connection lifecycle state;
  • owner-scoped AbortController lifecycle;
  • authenticated execution scopes and frozen cancellation leases;
  • retry decisions (SESSION_IS_LOCKED, safe read retry, non-idempotent uncertainty);
  • per-tab logical-session policy;
  • logical result modes (Table, KPI, etc.) and format/settings mapping;
  • row caps and UI result accumulation;
  • schema/catalog/documentation/lineage/capability SQL;
  • workbench/dashboard behavior;
  • export file picker/progress/.partial UX.

Hard invariants

  1. Native response identity. Low-level request returns response === fetchResponse, including non-2xx.
  2. No hidden body consumption. Low-level request leaves bodyUsed === false.
  3. One Fetch invocation. No transport retry, credential lookup, or hidden request.
  4. Exact SQL. No trim, trailing-semicolon removal, format rewriting, comment rewriting, or body normalization.
  5. Opaque Authorization. Complete string emitted unchanged and never cached between requests.
  6. Live endpoint/fetch dependencies. Next request observes current accessors.
  7. Native cancellation lifetime. Caller signal controls actual Fetch and post-header body lifetime; no derived-signal bridge may sever it.
  8. Cancellation isolation. Aborting A cannot affect B.
  9. Raw-byte safety. Raw/export/binary paths never require UTF-8 decoding.
  10. Single protocol owner. At completion, generic URL/stream/error/request mechanics have one implementation: the package.
  11. Application policy stays out. Package cannot refresh credentials, publish lifecycle state, own operation registries, decide retries, or understand SQL Browser views.
  12. Single artifact remains. SQL Browser still builds one self-contained dist/sql.html.
  13. Auth authority remains fenced. Credential awaits and async auth-body inspection remain bound to the captured credential epoch.
  14. Raw success can stay raw. Export can classify HTTP success/error without consuming a successful response body.

Delivery phases

Phase 1 — freeze native Fetch, Response, and cancellation semantics

Characterization/test-infrastructure phase only. Do not refactor the transport.

Strengthen the existing transport contract and real-browser fault harness to prove:

  • exactly one injected fetch call, including pre-aborted input;
  • strict native Response identity;
  • non-2xx resolves with untouched body;
  • exact SQL including whitespace/comments/trailing semicolon/authored FORMAT;
  • opaque Bearer/Basic/custom Authorization;
  • raw invalid-UTF-8 bytes remain byte-identical;
  • current URL serialization/encoding including zero/empty values where supported;
  • live origin() and fetch() accessors;
  • pre-aborted signal invokes the injected fetch but does not reach the real server;
  • abort while awaiting headers => AbortError;
  • after headers, request stays resolved but aborting the original signal makes an in-progress body read reject AbortError;
  • no later stream callbacks after observable cancellation;
  • concurrent request remains unaffected;
  • abort after full body completion has no effect.

The post-header proof must execute end-to-end in Chromium and WebKit using the real browser Fetch stack; an already-errored synthetic ReadableStream is insufficient.

Claims: A1, A3.

Phase 2 — create @altinity/clickhouse-http and move low-level request/URL mechanics

Create packages/clickhouse-http as a private npm workspace package with zero runtime dependencies and public package-name imports.

Expose a low-level client equivalent to:

interface ClickHouseHttpRequest {
  sql: string;
  defaultFormat: string;
  settings?: Record<string, string | number>;
  params?: Record<string, string | number>;
  authorization: string;
  signal?: AbortSignal;
}

interface ClickHouseHttpClient {
  request(request: ClickHouseHttpRequest): Promise<Response>;
}

Move current chUrl() behavior into one authoritative package serializer. Preserve default_format, enable_http_compression=1, settings and existing caller-supplied protocol/query parameters exactly.

Keep the old transport only as a temporary compatibility adapter delegating send() to package request(); stream reading remains local until Phase 3.

Add architecture checks forbidding package imports from root src/** and SQL Browser deep imports into packages/clickhouse-http/src/**.

Claims: A2, A4, A5.

Phase 3 — extract progress-stream and late-exception protocol primitives

Move generic protocol mechanics, not SQL Browser result state.

Package owns:

  • progress line shape (meta, row, progress, exception);
  • incremental UTF-8 decoder/newline buffering/malformed-line behavior;
  • per-network-chunk callback semantics;
  • parseExceptionText();
  • tagged late-exception framing;
  • legacy final-tail exception fallback;
  • byte-accurate clean-data boundary.

Expose byte-oriented exception framing such as:

findExceptionFrame(tailBytes: Uint8Array, tag?: string | null)
  => { message: string; cleanBytes: number } | null

Keep StreamResult, row caps, percentages, raw/result presentation state, editor caret extraction and auth-expiry UI policy in SQL Browser.

No duplicate stream/error implementation remains behind compatibility wrappers.

Claims: A6, A7.

Phase 4 — add consuming query APIs, ClickHouse errors, and stateless KILL QUERY

Preserve low-level request() unchanged.

Add explicit higher layers:

ensureClickHouseSuccess(response): Promise<Response>
consumeJsonResponse<T>(response): Promise<T>
consumeTextResponse(response): Promise<string>
consumeProgressResponse(response, callbacks?): Promise<Response>

ensureClickHouseSuccess returns the same successful response without consuming it; only non-2xx consumes error text and throws minimal ClickHouseError.

Add convenience queryJson, queryText, queryProgress composed from exactly one request plus one consumer. They must not know SQL Browser modes such as Table/KPI.

Add stateless killQuery({queryId, authorization, ...}): safely quoted, ASYNC, one request, no registry/retry/credential lookup.

Abort/network/body-reader errors must remain native rather than being wrapped as ClickHouse errors.

Claims: A8, A9.

Phase 5 — extract ClickHouse SQL quoting and type-expression grammar

Move reusable pure ClickHouse language helpers into the package:

  • string-literal quoting;
  • identifier quoting and qualification;
  • generic ClickHouse type-expression AST/parser/canonicalization/wrapper analysis/enum inspection.

Preserve current grammar/test behavior for nested types, literals, Tuple, Enum8/Enum16, Nullable, LowCardinality, arrays, malformed input, wrapper ordering and canonicalization.

killQuery must use the same public string-literal helper.

Keep SQL Browser display formatting, FORMAT detection/preparation, parameter-control policy and Dashboard/UI decisions outside the package.

No duplicate scanner/parser/quoter may remain.

Claims: A10, A11.

Phase 6 — simplify SQL Browser authentication integration around the package

High-risk trust-boundary phase.

Create one SQL Browser-owned authenticated request module that composes credential/epoch/lifecycle policy over package request() and package response consumers.

Preserve:

  • epoch captured before first await;
  • settings/params snapshot before credential await;
  • URL/preparation failure timing before token lookup;
  • epoch fence after every credential-related await and cloned error-body read;
  • final epoch fence immediately before actual request side effect;
  • complete Authorization computed per attempt;
  • at most one auth refresh retry;
  • post-confirmation 401/403 remain query outcomes, not sign-out;
  • only current 2xx reports connected;
  • only current non-abort network rejection reports offline;
  • stale work never mutates replacement lifecycle state;
  • original caller AbortSignal reaches actual Fetch with no derived controller/listener bridge;
  • frozen cancellation lease path does not read mutable tokens/refresh/lifecycle.

ChCtx and cancellation leases remain SQL Browser-owned.

Real-browser cancellation tests must run through the authenticated path, not only the raw package client.

Claims: A12, A13.

Phase 7 — migrate query execution and export, then delete generic client mechanics

Move application consumers off legacy runQuery / exportQuery / ordinary transport helpers.

QueryExecutionService keeps SQL Browser policy:

  • Table/KPI/TSV/raw format mapping;
  • row-limit settings;
  • fresh query ID per attempt;
  • SESSION_IS_LOCKED retry;
  • safe read network retry;
  • no automatic retry for uncertain non-idempotent work;
  • script stop-on-first-failure;
  • owner-scoped cancellation.

ExportService keeps file picker/progress/.partial UX but uses authenticated native Response, package success classification, direct byte streaming, and package findExceptionFrame on retained tail bytes.

Successful raw export must never use response.text().

After all callers migrate, delete superseded generic runQuery, exportQuery, ordinary mutable-context killQuery, old transport stream plumbing and transport seam files when no architectural purpose remains. Keep only SQL Browser product operations/thin authenticated helpers and frozen-lease cancellation policy.

Run full e2e plus Chromium/WebKit cancellation/export streaming proof.

Claims: A14, A15, A16.

Phase 8 — stabilize the standalone package boundary and remove migration scaffolding

This phase does not create another repository and does not publish a release. Its purpose is to leave altinity-sql-browser in a clean state where repository extraction is mechanical.

Make packages/clickhouse-http independently buildable and packable inside the current repository:

  • built browser-first ESM;
  • TypeScript declarations;
  • package-local build, test, check:types scripts;
  • no root-source imports or workspace-only type resolution;
  • publication-shaped exports, files, license and version metadata;
  • no runtime dependency unless explicitly introduced earlier.

Add deterministic isolated-package proof:

  1. build package;
  2. npm pack;
  3. inspect tarball contents;
  4. install tarball into a temporary fixture outside the workspace package tree;
  5. import it as ESM;
  6. compile a TypeScript consumer against its declarations;
  7. prove resolution does not fall back into SQL Browser src/**.

Then remove migration scaffolding:

  • compatibility-only package/root aliases and transport wrappers no longer needed after Phase 7;
  • @clickhouse/client-web dependency and executable vendor-spike wiring;
  • stale import maps/scripts referring to rejected adoption.

Retain ADR/evidence history for #585 and reuse/rename generic browser fault scenarios as first-party package regressions.

Strengthen architecture guards for:

  • package→SQL Browser imports;
  • SQL Browser package deep imports;
  • regrowth of a second generic request/URL implementation;
  • regrowth of duplicate progress/late-exception parsers;
  • accidental reintroduction of @clickhouse/client-web without a new decision.

Update package README, docs/ARCHITECTURE.md, CLAUDE.md, relevant .wiki, ADR-0005 addendum and CHANGELOG.md. Add docs/clickhouse-http-repository-extraction.md as the tested handoff for #639.

At Phase 8 completion the package still lives in this repository, but its source/API/build/test boundary is ready to move unchanged.

Claims: A17, A18.


Tests

Every phase must leave the full repository gate green:

npm run check:types
npm run check:arch
npm run check:schemas
npm run check:examples
npm test
npm run build

Phase 1

Run the targeted Chromium/WebKit fault/cancellation suite. Cover pre-abort, awaiting headers, post-header body abort, concurrent isolation, exact Response identity, raw bytes, SQL/Auth/URL fidelity.

Phase 2

Run package/request contract tests plus relevant e2e because workspace/dependency wiring changes.

Phase 3

Test split/multi-line chunks, UTF-8 split boundaries, malformed lines, trailing remainder, reader error identity, tagged/legacy exception framing, false-positive resistance and invalid-UTF-8 clean bytes.

Phase 4

Test successful non-consuming classification, JSON/text/progress consumption, ClickHouseError on non-2xx, native abort/body errors, one-request convenience APIs, and KILL QUERY quoting/state isolation. Re-run real-browser post-header cancellation.

Phase 5

Run the complete existing type-parser corpus plus quoting edge cases and affected parameter/KPI/dashboard-variable consumers.

Phase 6

Run all auth/epoch/refresh/lifecycle race tests and Chromium/WebKit post-header cancellation through the authenticated path.

Phase 7

Run QueryExecutionService retry/result mapping tests, ExportService raw-byte/late-error/partial-file/cancellation tests, full e2e, and real-browser export cancellation after headers.

Phase 8

Run package-local build/type/test, npm pack, isolated ESM import, isolated TypeScript compile, architecture sabotage tests, root full gate, full e2e, and renamed first-party Chromium/WebKit package regression suite.

Global acceptance criteria

  • A1 Native Fetch/Response/cancellation contract is characterized in unit and Chromium/WebKit tests.
  • A2 @altinity/clickhouse-http exists as an in-repo package with no SQL Browser source imports.
  • A3 Low-level request preserves native Response identity, exact SQL/Auth, raw bytes, one Fetch call and caller-signal lifetime.
  • A4 SQL Browser consumes the package through public package exports.
  • A5 URL serialization has one package implementation.
  • A6 Progress-stream decoding has one package implementation while SQL Browser retains result/view state.
  • A7 ClickHouse HTTP and late exception parsing/framing are package-owned and byte-safe.
  • A8 Explicit non-consuming and consuming response APIs exist with a minimal ClickHouse error model.
  • A9 Stateless package KILL QUERY exists without credential lookup/retry/registry ownership.
  • A10 ClickHouse SQL quoting has one package implementation.
  • A11 Generic ClickHouse type grammar/parser/canonicalization has one package implementation.
  • A12 SQL Browser authentication is composed through one authenticated request layer over package request().
  • A13 Epoch/refresh/lifecycle/cancellation/frozen-lease invariants remain regression-tested and unchanged.
  • A14 QueryExecutionService owns logical format/cap/retry policy and no longer owns generic HTTP/stream mechanics.
  • A15 ExportService streams native bytes and uses package late-exception framing while retaining export UX/policy.
  • A16 Superseded generic transport/client mechanics are deleted rather than retained as a second implementation.
  • A17 Package can build, pack, install, import and typecheck in isolation with no root-source fallback.
  • A18 Final architecture guards/docs are reconciled, obsolete @clickhouse/client-web executable wiring is removed, and follow-up: move @altinity/clickhouse-http into a dedicated Altinity repository #639 has a tested mechanical extraction handoff.

Non-goals

  • Reimplementing the full @clickhouse/client-web feature surface.
  • ORM/query-builder APIs.
  • Automatic retry in the package.
  • Package-owned OAuth, Basic credential acquisition, refresh, lifecycle state or query registry.
  • Moving SQL Browser schema/catalog/lineage/documentation product queries into the package.
  • Redesigning workbench/dashboard UX or changing persisted formats.
  • Adding another third-party runtime dependency.
  • Creating Altinity/clickhouse-http or publishing npm from this issue; that is follow-up: move @altinity/clickhouse-http into a dedicated Altinity repository #639 after Phase 8 ships.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    refactorRestructuring without user-facing behavior changetech-debt

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions