Repository navigation
Operator instrumentation
Implementation reference: native PR #116 at 4300ff6, with the browser observer in dependent PR #124 at 4671499. Commands and APIs below describe these revisions; the PRs show their current merge status.
Instrumentation is opt-in. Ordinary execution emits no measurement program and does not walk rows for counters. Instrumented CPU and elapsed times include the measurement overhead; do not mix them with instrumentation-free timing arms.
qdb --sql --data /path/to/parquet -i query.sql --instrumentation
qdb --sql --data /path/to/parquet -i query.sql --instrumentation group_predicate
qdb --sql --data /path/to/parquet -i query.sql --instrumentation operators,group_predicate
The flag without a list selects operators. The comma-separated list also accepts the --instrumentation=operators,group_predicate spelling. operators prints per-task and per-stage counters after execution; group_predicate prints row-group predicate decisions. Unknown or empty components are rejected. Repeating the option replaces the preceding selection. Output goes to stderr; query results retain their selected format on stdout. The old QDB_DEBUG_ROW_GROUP_PREDICATE environment variable has no effect.
Applications select TSettings::Instrumentation explicitly and provide its output stream. No execution-library code reads the CLI environment.
qdb/exec/instrumentation/operator.{h,cpp} contains rowset and operator counters. qdb/scheduler/instrumentation/task.{h,cpp} binds counters to graph ports and formats the post-run snapshot. Row/byte and hash-storage measurements use the common generated Oz program in qdb/kernel/instrumentation.oz.
Blocking and merge tasks own an ITaskState. Instrumentation is configured by a method on that mutable state before scheduling begins. The immutable prepared code carries no per-query instrumentation callback or counters. States that wrap processors forward setup to their own processor; CTE spool state owns its own observer. Source/unary/sink callbacks retain their existing execution interfaces.
Semantic stage IDs supply output schemas. A synthetic UNION export node has InvalidExecStageId, which does not identify a semantic stage. Routing tasks derive their schema from physical input connections. Inserting the synthetic union schema into the stage map would mis-type shuffles in branches with a different width. The native regression covers two grouped JOIN branches projected to one column before UNION, with four shuffle partitions and both schedulers.
Runtime cardinalities belong to the task that reads or publishes a batch. Instrumentation is opt-in: ordinary execution does not emit the measurement program or visit rows for instrumentation. Prepared measurement code is immutable; selection-aware row and byte computation uses the same Oz program in native and browser execution.
Each task owns its counters and receives its observer explicitly before it can run. Only the worker executing that task writes them. The scheduler's existing single-writer guarantee lets the counters follow a task between workers without locks, atomics per row, thread-local storage, or query-global mutable counters. Snapshots are read and combined only after workers stop. An input counts a batch when fetched successfully; an output counts it once when published successfully. Backpressure retries do not count the same batch again. Source stages have no rowset input; their output is the data actually supplied to the graph.
Active rows respect the selection mask. Bytes measure the logical payload of selected, non-NULL values, including actual string lengths, rather than disk I/O or allocator traffic. Materialized bytes describe output payload at stages that materialize rows; they do not claim every byte was copied. Hash groups, capacity and growth are observed at batch boundaries before state is destroyed. Geometric capacity changes preserve the number of successful rehashes even when several occur inside one batch. Initial reservation is not a rehash.
Buffered exchanges account the full visible payload of retained input owners, including inactive rows in sparse views. Partition views of one batch count that owner once per task. Measurement is cached for the lifetime of a retained owner, so another partition does not trigger another row walk. Current retained payload and its peak remain separate from selected output bytes. Current payload describes ownership at snapshot time; releasing input resets the current value while preserving its peak. State retained until graph teardown can therefore remain current in the post-run snapshot.
Memory snapshots separate retained hash allocations from retained input payload. Their sum is accounted build memory, excluding transient rehash copies, allocator overhead, input-buffer metadata and sort scratch; it is not process RSS. Retained payload can share backing storage. A stage sums task peaks, which need not occur simultaneously. Task cardinalities remain in the snapshot so partition imbalance can be distinguished from a cardinality-estimation error.
Native stage CPU sums thread CPU consumed inside task execution, including instrumentation, and excludes ready-queue waiting and descheduling. CPU counters follow the task between workers; snapshots are read only after workers stop.
Browser observation is a separate dependent PR. It uses the native exporter measurement metadata and the same compiled Oz measurement program. The browser has no thread CPU clock: CPU is unavailable, while task elapsed time includes asynchronous I/O. Its contracted graph does not measure native partition distribution. Unsupported metrics remain absent or null. These browser semantics apply only with the separate JS instrumentation change, not the native-only PR.