feat(kernel): consolidated DAIS gap-closure + PuPr feature set (supersedes #395/#396/#397)#399
Draft
mani-mathur-arch wants to merge 10 commits into
Draft
feat(kernel): consolidated DAIS gap-closure + PuPr feature set (supersedes #395/#396/#397)#399mani-mathur-arch wants to merge 10 commits into
mani-mathur-arch wants to merge 10 commits into
Conversation
This was referenced Jul 14, 2026
mani-mathur-arch
force-pushed
the
mani/sea-kernel-consolidated
branch
from
543c7a2 to
ba466d3
Compare
mani-mathur-arch
force-pushed
the
mani/sea-kernel-consolidated
branch
from
ba466d3 to
f35b387
Compare
…initial namespace Close the four DAIS-scope rows the SEA-via-kernel backend previously rejected at connect time. All are Go-side only (no new kernel dependency): the OAuth setters are on merged kernel #162, metric-view is an existing session conf, and the namespace uses plain SQL. - Metric view: config.EffectiveSessionParams() derives the server conf (spark.sql.thriftserver.metadata.metricview.enabled) once, backend-neutrally, so both backends send the identical conf. The Thrift OpenSession special-case is removed (behaviour-preserving); the kernel forwards it via SessionConf. Reject dropped; reclassified forwarded. - Initial namespace: applied post-connect via USE CATALOG / USE SCHEMA (the OSS ODBC workaround) since the kernel C ABI has no catalog/schema setter. quoteIdent (untagged) backtick-quotes identifiers; a USE failure fails connect and closes the session. Reject dropped; reclassified forwarded. - OAuth M2M/U2M: the kernel drives its own OAuth flow from raw credentials (mirroring pyo3/napi and the Node/Python kernel bindings), read off cfg.Authenticator — the single source of truth (last-writer-wins, matching Thrift). The m2m/u2m authenticators expose auth.M2MCredentialsProvider / auth.U2MCredentialsProvider; resolveKernelAuth type-switches them and returns a *kernelAuth descriptor. KernelBackend.setAuth branches to set_auth_pat / set_auth_m2m / set_auth_u2m; U2M uses Go's cloud-inferred client id (kernel defaults for scopes/port). No new config fields. Verified: default CGO_ENABLED=0 suite + golangci-lint v2.12.2 clean; tagged databricks_kernel unit tests (auth-mode -> setter mapping, quoteIdent); live staging e2e for initial namespace (current_catalog/current_schema) and metric-view (session opens + queries; the conf is not SET-introspectable on either backend). M2M/U2M covered by unit tests (no staging service principal; U2M is interactive). Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
… single-source auth, coverage + docs Remediation of the code-review pass on the DAIS gap-closure work. Verified against source; full default + tagged suites, golangci-lint v2.12.2, and live staging e2e all green. - Move the OAuth credential-provider interfaces (M2MCredentialsProvider / U2MCredentialsProvider) out of the public auth package into internal/backend/kernel, so the secret-reading capability is not part of the driver's public API. The unexported m2m/u2m authenticators satisfy them structurally. - Collapse the duplicate auth descriptor: validateKernelConfig/resolveKernelAuth now return kernel.Auth directly (its type is in an untagged file, so the default build builds it cgo-free); dropped dbsql.kernelAuth, kernelAuthMode, and toKernelAuth (which also removed a stale build-tag comment). - Route the initial-namespace failure-path session close through call() so a failed close is logged (via lastError's Warn), mirroring CloseSession. - Add an env-guarded live M2M e2e (TestKernelE2EM2M, skips without DATABRICKS_CLIENT_ID/_SECRET) and a last-writer-wins auth regression test; the resolveKernelAuth -> kernel.Auth path is table-tested for M2M/U2M. - Document: U2M is interactive (browser on cache-miss, blocks up to the kernel's ~120s callback timeout, connect ctx deadline not honored during that window, no C-ABI override — use PAT/M2M for headless); the U2M Scopes/RedirectPort fields are dormant-but-wired (no Go option feeds them yet); the metric-view e2e is a deliberate connect-smoke (routing asserted in TestEffectiveSessionParams). Custom M2M scopes remain unforwardable over the C ABI (no scopes arg on set_auth_m2m) — a kernel gap shared with ODBC, no authz impact (all-apis always requested); tracked in the kernel-gaps notes rather than worked around. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
The kernel backend returns INTERVAL columns as native arrow duration (day-time) and month-interval (year-month) values, whereas the Thrift path receives them pre-formatted from the server (its native-interval config is off in prod, so it never scans a duration/month-interval array). Format them Go-side in the shared untagged arrowscan package to the same strings the Thrift path returns — "D HH:MM:SS.nnnnnnnnn" and "years-months", negatives signed — so a query's result is identical across backends. Replaces the fail-loud "intervals are not yet handled" default arm with the two type arms; golden-string unit tests (day/day-to-sec/seconds-unit/negative, year/year-month/months/negative) run in the default CGO_ENABLED=0 build. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
Over Arrow the kernel delivers TIMESTAMP with a tz ("UTC") and TIMESTAMP_NTZ
with an empty tz, but — like the Thrift path — the driver ignores that field
and renders both via ToTime + .In(loc). The LTZ-vs-NTZ difference is carried
entirely by the instant the server sends, not by the client inspecting the
tz, so no arrowscan change is needed: the existing code already matches Thrift.
Verified live on both backends (America/New_York + Asia/Kolkata, including a
DST spring-forward literal, and nested/null shapes): kernel == Thrift
byte-for-byte for both types. Add an untagged parity case (TimeZone "UTC" vs
"") so a future "don't shift NTZ" change — which looks correct in isolation
but would diverge from Thrift, which shifts NTZ too — fails default CI, plus a
live e2e pinning the round-trip.
Co-authored-by: Isaac
Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
VARIANT and GEOMETRY need no special rendering on the kernel path: verified
live on both backends, both arrive over Arrow as plain STRING columns — a
top-level VARIANT is its JSON text ({"a":1,"b":[2,3]}), a scalar VARIANT is
"42", and GEOMETRY is its WKT "POINT(1 2)". Nested inside a container the
variant/geometry element is a string leaf, rendered as a quoted, JSON-escaped
string (the variant's own JSON is escaped as text, NOT re-parsed) — identical
on both backends.
Add untagged parity cases: a top-level string equivalence (variant object /
scalar / geometry WKT) and nested string-leaf cases in an array, so the string
arm's handling of these types can't silently drift between backends. GEOGRAPHY
is intentionally excluded — not enabled on the benchmark warehouse and no
consumer has asked for it.
Co-authored-by: Isaac
Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
A failed kernel query returned the raw *KernelError, so consumers doing errors.As(err, &DBExecutionError) — the way they inspect Thrift failures — didn't get SqlState()/QueryId()/IsRetryable() through the standard interface. kernelOp.ExecutionError now digs the sqlstate out of the underlying *KernelError and wraps the cause via NewExecutionErrorWithState, so kernel query failures surface with the same DBExecutionError shape as Thrift. Adds the neutral NewExecutionErrorWithState to the untagged internal/errors package (Thrift's NewExecutionError needs a TGetOperationStatusResp the kernel backend can't produce), unit-tested in the default CGO_ENABLED=0 build. Parity is type + SQLSTATE, not byte-identical text — kernel messages are richer (they carry the SQL error class + suggestions). Verified live: unknown table → 42P01, unknown column → 42703, byte-identical sqlstate to Thrift. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
…al caveat Record what the kernel backend inherits unchanged above the backend seam — the database/sql connection pool (each conn wraps one kernel session), per-connection CREATE_SESSION / DELETE_SESSION telemetry (recorded unconditionally in connector.go, backend-agnostic), and the telemetry exporter's circuit breaker — and that result types render byte-for-byte with Thrift (scalars, exact DECIMAL, TIMESTAMP / TIMESTAMP_NTZ, INTERVAL, nested + VARIANT as JSON, GEOMETRY as WKT). Remove the now-stale "INTERVAL types are not yet handled by the kernel scanner" caveat (intervals render now), and narrow the telemetry caveat to what is actually missing: only EXECUTE_STATEMENT telemetry (gated on a per-statement query id the kernel C ABI doesn't yet surface) — CREATE_SESSION / DELETE_SESSION are unaffected. Add a live-verified connection-pool e2e (40 concurrent queries over pool cap 8) backing the inherited-pool claim. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
The kernel backend rejected bound parameters at execute time. Now it binds them:
the driver's backend.Param{Name, Type, Value} maps 1:1 onto the kernel's
kernel_statement_bind_parameter (K1) — value already stringified, Type the
Databricks SQL type name, empty Name → positional, nil Value → SQL NULL ("VOID").
bindParams runs after set_sql (which clears any prior binds), using the existing
newCStr/newCStrOrNull helpers and the call() FFI-safety wrapper; a bind failure
closes the statement and surfaces via toStatementError.
Removes the fail-loud reject in Execute (and its now-unused errors import). The
old TestExecuteRejectsParams is repurposed as TestExecuteHandleLessOpContract
(the non-nil handle-less Operation contract, now driven by a nil-session failure
since params no longer reject). Live parity: 10 cases (positional/named, each
scalar type, NULL, multi-param, predicate) produce byte-identical output on the
kernel and Thrift backends.
Requires a kernel build carrying kernel_statement_bind_parameter; the KERNEL_REV
pin is bumped to the K1 merge SHA when it lands.
Co-authored-by: Isaac
Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
kernelOp.StatementID() returned "", so the kernel backend emitted no EXECUTE_STATEMENT telemetry and QueryIdCallback fired with an empty id (connection.go gates both on a non-empty statement id). Wire StatementID() to the server query id via kernel_executed_statement_query_id (K1), captured at execute time into a cached field — the same lifetime discipline as affectedRows, since the C accessor returns a pointer borrowed from the exec handle and the op is closed (nulling exec) before StatementID() is read on some paths. C.GoString deep-copies out of the borrowed string. Live e2e: a registered QueryIdCallback fires with a non-empty server id after a kernel query. Updates doc.go — bound parameters (c6) and EXECUTE_STATEMENT telemetry are now supported; the remaining kernel-backend limitation is batch-boundary (not mid-fetch) read cancellation. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
mani-mathur-arch
force-pushed
the
mani/sea-kernel-consolidated
branch
from
f35b387 to
d0c999e
Compare
Bumps the kernel pin from the #163 canceller rev to the tip of the PuPr statement-surface branch (databricks-sql-kernel#165), which adds kernel_statement_bind_parameter and kernel_executed_statement_query_id — the two C-ABI symbols the bound-parameter (c6) and EXECUTE_STATEMENT-telemetry (c7) commits link against. Re-pin to the squash-merge SHA once #165 lands. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
mani-mathur-arch
force-pushed
the
mani/sea-kernel-consolidated
branch
from
d0c999e to
2caa354
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Consolidates the DAIS gap-closure (#396) and PuPr feature (#397) work for the SEA-via-kernel backend into a single branch based directly on the feature backend (#393), dropping the now-redundant build-recipe PR (#395) — its recipe was folded into #393, so this branch carries the identical files with the temporary CI trigger already stripped.
Ten reviewable feature commits, no build-recipe duplication. Stacked directly on
mani/sea-kernel-backend(#393).Why consolidate
build/kernel-lib.sh,KERNEL_REV, thecgo_<os>.gofiles, thebuild-and-test-kerneljob). This branch drops build(kernel): 2-mode kernel build recipe + tagged CI job #395's three recipe commits viarebase --onto— the recipe files are byte-identical to feat(kernel): SEA-via-kernel backend (opt-in, behind build tag) #393's copy.NewExecutionErrorWithStateoriginally took(ctx, msg, err, sqlState); feat(kernel): SEA-via-kernel backend (opt-in, behind build tag) #393's round-9 revision made it(ctx, msg, err, queryId, sqlState)(pulls both the sqlstate and the server query id off*KernelError, with an empty-queryId→ctx fallback). This consolidation keeps feat(kernel): SEA-via-kernel backend (opt-in, behind build tag) #393's more-correct version and discards the earlier one, soQueryId()is populated on the kernel error path.Commits (bottom → top)
feat(kernel): close deferred DAIS rows — OAuth M2M/U2M, metric-view, initial namespace(feat(kernel): close deferred feature rows — OAuth M2M/U2M, metric-view, initial namespace #396)fix(kernel): address code-review findings — internal auth interfaces, single-source auth, coverage + docs(feat(kernel): close deferred feature rows — OAuth M2M/U2M, metric-view, initial namespace #396)feat(arrowscan): render INTERVAL day-time & year-month(feat(kernel): SEA-via-kernel data types, params, errors & query-id telemetry #397)test(kernel): pin TIMESTAMP vs TIMESTAMP_NTZ parity(feat(kernel): SEA-via-kernel data types, params, errors & query-id telemetry #397)test(kernel): pin VARIANT & GEOMETRY render parity(feat(kernel): SEA-via-kernel data types, params, errors & query-id telemetry #397)fix(kernel): surface kernel errors as DBExecutionError with sqlstate(feat(kernel): SEA-via-kernel data types, params, errors & query-id telemetry #397, reconciled against feat(kernel): SEA-via-kernel backend (opt-in, behind build tag) #393's F1 fix)docs(kernel): document verified inherited features; drop stale interval caveat(feat(kernel): SEA-via-kernel data types, params, errors & query-id telemetry #397)feat(kernel): bind query parameters via the C ABI(feat(kernel): SEA-via-kernel data types, params, errors & query-id telemetry #397)feat(kernel): surface server query id for EXECUTE_STATEMENT telemetry(feat(kernel): SEA-via-kernel data types, params, errors & query-id telemetry #397)build(kernel): pin KERNEL_REV to the PuPr statement-surface rev(feat(kernel): SEA-via-kernel data types, params, errors & query-id telemetry #397)Conflict resolutions of note
kernel_config.go— merged feat(kernel): close deferred feature rows — OAuth M2M/U2M, metric-view, initial namespace #396'skernel.Authrefactor (value-type auth descriptor,resolveKernelAuth, forward namespace + metric-view) with feat(kernel): SEA-via-kernel backend (opt-in, behind build tag) #393's round-7/8 hardening (rejections wrapErrNotSupportedByKernel,WithTransportrejected). The field-disposition map keepsCatalog/Schema/EnableMetricViewMetadataasforwarded(no duplicate-key collision) and listsTransportamong the rejected set.executevia the kernel raw-param C ABI); the staging reject stays.doc.go— merged the OAuth/namespace/metric-view "now supported" prose with the params-supported / staging-still-unsupported wording, dropping the stale INTERVAL caveat.Testing
CGO_ENABLED=0):go build,go vet,go test ./...— 24 packages ok, 0 fail.CGO_ENABLED=1 -tags databricks_kernel, linked against a locally-built kernel.aatKERNEL_REV):go build,go vet,go test ./...— 24 packages ok, 0 fail.gofmtclean across all 24 changed files; no conflict markers.Three issues surfaced by building both paths (not just the default) were fixed and folded into their originating commits: a 4-arg→5-arg
NewExecutionErrorWithStatetest call, a droppederrorsimport in the taggedbackend.go, and a staleErrNotSupportedByKernelassertion in the repurposedTestExecuteHandleLessOpContract.Merge order / gate
Base is #393 (
mani/sea-kernel-backend). Merge order unchanged apart from dropping #395: kernel #163 → #165, then driver #393 → this. Re-pinKERNEL_REV(currently53171cd, #165's tip) to #165's squash-merge SHA at merge time.#395 / #396 / #397 are intentionally left open for now.
This pull request and its description were written by Isaac.