diff --git a/eng/pipelines/perf/README.md b/eng/pipelines/perf/README.md new file mode 100644 index 0000000000..2a7ae55dd9 --- /dev/null +++ b/eng/pipelines/perf/README.md @@ -0,0 +1,239 @@ +# SqlClient Performance Test Pipeline + +This directory contains the Azure DevOps pipeline and supporting scripts that run the +Microsoft.Data.SqlClient [BenchmarkDotNet](https://benchmarkdotnet.org/) performance tests on a +dedicated performance test lab (Azure Dedicated Hosts), compare the branch under test against a +released NuGet baseline, and optionally ingest the results into an Azure Data Explorer (Kusto) +database. + +## Contents + +| Path | Purpose | +| ---- | ------- | +| `sqlclient-perf-pipeline.yml` | The pipeline. Extends `v1/Perf.Test.Job.yml@PerfTemplates`. | +| `scripts/run-perf-tests.sh` | Linux on-VM entry point: install SDK, create DB, run benchmarks (interleaved or sequential), compare. | +| `scripts/run-perf-tests.ps1` | Windows equivalent (ProcessorAffinity instead of `taskset`). | +| `scripts/interleave_perf.py` | Interleaved + best-of-N orchestrator: runs each unit baseline↔candidate back-to-back and confirms regressions across N passes. | +| `scripts/compare_perf.py` | Compares baseline vs current BenchmarkDotNet JSON → delta (md + json). Reused by the orchestrator. | +| `scripts/perf_to_kusto.py` | Translates BenchmarkDotNet "full" JSON → Kusto `PerfRun` + `PerfBenchmarkResult` NDJSON. | +| `scripts/ingest_kusto.py` | Queued Kusto ingestion (az CLI auth, runs on the agent). | + +## Architecture + +``` +Queue pipeline (ADO) + │ + ▼ +extends: v1/Perf.Test.Job.yml@PerfTemplates + │ (provisions a dedicated-host VM, SCPs the repo, runs the script over SSH, + │ SCPs back, publishes it, tears the VM down) + ▼ +ON THE VM ── run-perf-tests.{sh,ps1} + 1. Install the .NET SDK pinned by global.json (+ runtimes). + 2. Create the perf database on the VM's SQL Server. + 3. Inject the VM SQL connection string into runnerconfig. + 4. Baseline pass → MDS from NuGet.org (Package mode) → results/baseline/ + 5. Current pass → MDS built from source (ProjectReference) → results/current/ + (both pinned to PERF_CLIENT_CPUS; interleaved per-unit by default, or two full + sequential passes when benchmarkRunMode=sequential) + 6. interleave_perf.py / compare_perf.py → results/comparison/ + results/summary.md + ▼ +ON THE AGENT ── pipeline post-test steps + • Show the BenchmarkDotNet markdown reports in the log. + • perf_to_kusto.py → NDJSON for both passes (published as 'perf-kusto-payloads'). + • (optional) AzureCLI@2 + ingest_kusto.py → Kusto database. +``` + +The extends template only exposes **post-test** steps to consumers (no pre-build hook), so **both +benchmark passes run inside the on-VM script**. Translation and ingestion run on the **agent** +because that is where the pipeline's AAD identity / service connection and the native pipeline +context variables are available (the VM is behind NAT and lacks the pipeline identity). + +## Parameters + +| Parameter | Default | Description | +| --------- | ------- | ----------- | +| `platform` | `linux` | `linux` or `windows` VM + client. | +| `dotnetFramework` | `net9.0` | TFM the benchmarks run against (`net8.0`/`net9.0`/`net10.0`). | +| `testTimeoutMinutes` | `180` | Template timeout waiting for the VM run. | +| `baselineVersion` | `7.0.2` | **Baseline Version** — released MDS the branch is compared against. Empty = current-only (no baseline pass / comparison). | +| `regressionThreshold` | `10` | Percent slowdown (current vs baseline mean) flagged as a regression. | +| `failOnRegression` | `false` | When `true`, a candidate-slower regression **fails** the run (gate). In interleaved mode only **confirmed** regressions (best-of-N majority) fail. Default off. | +| `benchmarkRunMode` | `interleaved` | `interleaved` (per-unit baseline↔candidate + best-of-N confirmation) or `sequential` (legacy two full passes). | +| `confirmationRuns` | `3` | Best-of-N: interleaved passes for a flagged unit before a regression is confirmed. `1` disables confirmation. Interleaved mode only. | +| `enableKustoIngestion` | `true` | **Ingest results into Kusto** — when `false`, the run still benchmarks + compares but skips ingesting into the perf database. When `true`, ingestion additionally requires the `ADX Cluster Variables` group to be populated. | + +The following values are **fixed constants** in the pipeline (not parameters or variables), since they +are invariant for this pipeline: `buildConfiguration = Release`, `sourcesSubDir = dotnet-sqlclient` +(the multi-repo checkout folder for `self`, which must match the ADO repository name), and +`driverName = Microsoft.Data.SqlClient` (recorded on every row as `DriverName` / `DerivedRunId`). + +### Kusto (Azure Data Explorer) ingestion variables + +The ADX ingestion coordinates are **not** pipeline parameters — they come from a pipeline library +variable group named **`ADX Cluster Variables`** so no infrastructure identifiers are hard-coded in +the pipeline. The group must define: + +| Variable | Description | +| -------- | ----------- | +| `KustoClusterUri` | ADX cluster URI, e.g. `https://..kusto.windows.net`. Empty ⇒ ingestion skipped. | +| `KustoDatabase` | Target Kusto database. Empty ⇒ ingestion skipped. | +| `KustoServiceConnection` | Azure DevOps ARM service connection whose SP has ingest rights. Empty ⇒ ingestion skipped. | + +Ingestion is gated at runtime: it only runs when `KustoClusterUri`, `KustoDatabase` and +`KustoServiceConnection` are all non-empty, so the pipeline still runs + compares before the +cluster/service connection exist. + +### Managing the baseline version + +`baselineVersion` is **manually managed**. After each stable release is published to NuGet.org, +bump the `default` in `sqlclient-perf-pipeline.yml` (e.g. `7.0.2` → the next stable). It can also be +overridden at queue time without editing the pipeline. + +## Two-pass build model + +The `PerformanceTests` project references Microsoft.Data.SqlClient two ways, selected by MSBuild: + +- **Current** (default): `ProjectReference` to the in-repo source — the branch under test. +- **Baseline**: `ReferenceType=Package` turns the reference into a `PackageReference`. Because the + repo uses **Central Package Management (CPM)**, the version is pinned with `VersionOverride` via + `-p:MdsPackageVersion=` (a plain `Version` is ignored under CPM). + +The VM's `NuGet.config` exposes only the governed feed, and CPM rejects multiple unmapped sources +(`NU1507`). The baseline pass therefore restores through a **dedicated single-source config** +(`perf-baseline-nuget.config`, generated at runtime) pointing only at `https://api.nuget.org/v3/index.json`. + +## Comparison output + +`compare_perf.py` matches benchmarks by `(Type, Method, Parameters)` and reports, per benchmark, the +baseline/current mean (ms), mean %Δ, allocation %Δ, and a status (`regression` / `improvement` / +`unchanged` / `new` / `removed`). Outputs: + +- `results/comparison/comparison.md` (also copied to `results/summary.md`, which the template + attaches as the run summary), +- `results/comparison/comparison.json` (structured, for tooling). + +## Reducing noise + +The harness applies a set of harness-owned controls to reduce measurement noise. The lab already +supplies the isolated dedicated host, the tuned SQL instance, and the disjoint client CPU set +(`PERF_CLIENT_CPUS`); the run scripts add: + +| Control | What the harness does | +| ------- | --------------------- | +| Client CPU pin | Pins the benchmark process to `PERF_CLIENT_CPUS` (`taskset` on Linux, `ProcessorAffinity` on Windows). | +| Fail loud | Preflight `SELECT 1` before any pass, **and** a post-pass guard that fails the run if a pass produced **zero** benchmark results — so an empty comparison can never be reported green. | +| Warm-up | Touches the target DB in the preflight to warm the buffer pool / plan cache before the first measured benchmark. | +| Allocator tuning (Linux) | Exports `MALLOC_MMAP_THRESHOLD_=128MiB` and `MALLOC_TRIM_THRESHOLD_=-1` so large-buffer benches (`AsyncLargeDataRead`, `SqlBulkCopy`) stop re-`mmap`ing per iteration. | +| Network tuning (Linux) | Best-effort `sysctl` to widen the ephemeral port range and enable `tcp_tw_reuse` for churn benches (`ConnectionPoolStress`, `ParallelAsyncConnection`). Never fails the run. | +| Diagnostics | Writes `results/diagnostics/`: SQL instance config (MAXDOP, memory, affinity, tempdb files, `@@VERSION`), host CPU topology, and per-pass CPU-clock/thermal telemetry (before/after each pass). | +| Regression gate | `failOnRegression` threads `--fail-on-regression`; only a **candidate-slower** delta past the threshold fails, and in interleaved mode only after best-of-N confirmation. Default off. | +| Interleaving | In `interleaved` mode the harness runs **one benchmark unit at a time, baseline then candidate back-to-back**, so both sides see the same host state (see below). | +| Best-of-N confirmation | A unit flagged in the first interleaved pass is re-run `confirmationRuns` times; a regression is **confirmed** only on a strict majority. Unconfirmed flags are reported but never fail the gate. | + +### Interleaving + best-of-N (run model) + +`benchmarkRunMode` selects how the two variants are measured: + +- **`interleaved`** (default) — `interleave_perf.py` orchestrates the run. Both variants are built + **once** into separate output dirs (`perf-build-baseline`, `perf-build-current`), then for each + benchmark unit the baseline and candidate builds run **back-to-back** before moving to the next + unit. Because the same benchmark is measured on both sides within seconds, slow host drift affects + both roughly equally and cancels out of the delta. This relies on the `PerformanceTests` runner + supporting `PERF_LIST_BENCHMARKS` (enumerate enabled units) and `PERF_BENCHMARK=` (run a + single unit) — see `Program.cs`. + + After the first interleaved pass, only the units containing a flagged regression are re-run + `confirmationRuns` times (best-of-N). A regression is **confirmed** only when a strict majority of + the N passes agree `(count * 2 > N)`; otherwise it is reported as `regression (unconfirmed)` and + does **not** fail the `failOnRegression` gate. `confirmationRuns = 1` disables confirmation. + +- **`sequential`** — legacy model: the whole baseline suite runs, then the whole candidate suite, + then `compare_perf.py` diffs them. Kept as a fallback; produces the same `results/baseline`, + `results/current`, and `results/comparison/` layout so Kusto ingestion is identical. + +Both modes emit `results/comparison/comparison.md` + `comparison.json` and copy the markdown to +`results/summary.md`; interleaved mode adds a **Confirm** column and a confirmed/unconfirmed summary. + +### Further tuning (not yet implemented) + +- **Release-grade sampling / relaxed thresholds** — tune BenchmarkDotNet job counts and + significance thresholds in `runnerconfig.jsonc` / `BenchmarkConfig.cs` now that interleaving and + best-of-N are in place. + +## Kusto schema & ingestion + +Two tables: + +- **`PerfRun`** — one row per run (baseline OR current): `DerivedRunId` (PK = + `DriverName|CommitHash|PipelineRunId`), driver/machine/agent, `OperatingSystem` (`Windows`/`Linux`), + `Architecture` (`x64`/`x86`), `RunType` (`Sequential`/`Interweaved`), pipeline id + build URL, + branch + `BranchCategory`, `VersionString`, commit hash/date, `IsComparableBase`, `IngestedAt`. +- **`PerfBenchmarkResult`** — one row per benchmark: `BenchmarkId` (PK = + `DerivedRunId|BenchmarkName|MethodName|ParameterSignature`), timings in **milliseconds** + (BenchmarkDotNet reports nanoseconds; values are divided by 1,000,000), percentiles, throughput, + allocation, runtime/platform, and `DriverSpecificMetrics` (GC collections, lock contentions, …). + +The baseline and current passes share the pipeline run id and (for the current pass) the commit, so +to keep their `DerivedRunId`s distinct the baseline row uses `CommitHash = v` and +`IsComparableBase = true`; the current row uses the real commit and `IsComparableBase = false` with +the triggering branch name. + +Source = BenchmarkDotNet **JSON "full"** exporter files (`*-report-full.json`). The exporter is +enabled in `Config/BenchmarkConfig.cs` (`JsonExporter.Full`). + +### One-time database setup + +Before ingestion can run, create the two tables (`PerfRun`, `PerfBenchmarkResult`) in the target +database, with columns matching the schema summarized above. No server-side ingestion mappings need +to be created: `ingest_kusto.py` sends a **self-contained inline JSON column mapping** built from +each payload's own property names (which are identical to the table's column names), so ingestion +does not depend on any pre-created named mapping existing on the cluster. + +### Authentication + +Ingestion runs in an `AzureCLI@2` task using the ADO **ARM service connection** +(`KustoServiceConnection` from the `ADX Cluster Variables` group). That connection's **service +principal** must be granted, on the target database: + +- **Database Ingestor** — required to queue the ingestion, and +- **Database Viewer** — required for the post-ingestion verification queries. With Ingestor-only + rights the data still lands, but the verify step cannot read it back and logs a warning naming + this missing role. + +`ingest_kusto.py` authenticates to Kusto with `with_az_cli_authentication` (the service connection +is already `az login`'d inside the task) and performs a **queued** ingestion against the +data-management (`ingest-`) endpoint. + +### Running before the cluster exists + +Ingestion is **conditional**: it only runs when the `enableKustoIngestion` parameter is `true` (the +default) **and** `KustoClusterUri`, `KustoDatabase` and `KustoServiceConnection` (from the `ADX +Cluster Variables` group) are all non-empty. Set `enableKustoIngestion` to `false` to opt a run out +of ingestion explicitly. Until a cluster and service connection are +configured, the pipeline still runs both passes, produces the comparison, and publishes the +translated NDJSON as the `perf-kusto-payloads` artifact for manual/backfill ingestion. + +## Running the pipeline + +1. Open the performance test pipeline in Azure DevOps and select **Run pipeline**. +2. Choose the branch to benchmark; override `baselineVersion` only if needed. Ingestion is on by + default (`enableKustoIngestion`) and uses the `ADX Cluster Variables` group — populate + `KustoClusterUri` / `KustoDatabase` / `KustoServiceConnection` there to enable it, leave them empty + to skip ingestion, or untick **Ingest results into Kusto** to skip it for a single run. +3. After the run, review the **run summary** (comparison) and the `perf-results` / + `perf-kusto-payloads` artifacts. When a baseline pass ran, the build is tagged + **`Baseline `** so the baseline used is visible at a glance in the ADO build list. + +## Troubleshooting + +| Symptom | Likely cause / fix | +| ------- | ------------------ | +| `NU1507` during the baseline pass | Multiple NuGet sources under CPM. The baseline uses a single-source config; ensure `perf-baseline-nuget.config` is being passed via `-p:RestoreConfigFile`. | +| Baseline restore fails to find MDS | `baselineVersion` isn't a published NuGet.org version, or the VM has no outbound access to `api.nuget.org`. | +| No comparison / summary | The baseline pass was skipped (empty `baselineVersion`) or one pass produced no `*-report-full.json`. | +| Ingestion step skipped | `enableKustoIngestion` is `false`, or `KustoClusterUri`, `KustoDatabase` or `KustoServiceConnection` (from `ADX Cluster Variables`) is empty (expected until the cluster is provisioned). | +| Ingestion auth error | The service connection's SP lacks **Database Ingestor** on the target database. | +| "Kusto ingestion was queued, but the ingestion principal is not authorized to query the database" | The SP has **Database Ingestor** but not **Database Viewer**. Ingestion succeeded; grant **Database Viewer** so the verify step can confirm the rows landed. | +| `Kusto ingestion did not complete: PerfRun=0/N ... after 300s` while `.show ingestion failures` shows nothing | Data was queued but never landed. Confirm the `PerfRun` / `PerfBenchmarkResult` tables exist with columns matching the schema above. Ingestion uses a self-contained inline JSON mapping, so a missing server-side named mapping is no longer a cause; a schema/column-name mismatch is the remaining one. | +| Benchmarks not CPU-pinned | `PERF_CLIENT_CPUS` was not injected, or `taskset` is unavailable on the VM. | diff --git a/eng/pipelines/perf/scripts/compare_perf.py b/eng/pipelines/perf/scripts/compare_perf.py new file mode 100644 index 0000000000..609af7e8bd --- /dev/null +++ b/eng/pipelines/perf/scripts/compare_perf.py @@ -0,0 +1,253 @@ +#!/usr/bin/env python3 +"""Compare two sets of BenchmarkDotNet "full" JSON reports and emit a delta. + +Reads every ``*-report-full.json`` under a baseline directory and a current +directory, matches benchmarks by (Type, Method, Parameters), and computes the +per-benchmark delta in mean execution time and allocated memory. + +Outputs: + * a GitHub-flavoured markdown table (``--out-md``), and + * a structured JSON document (``--out-json``) + +The script only uses the Python standard library so it can run on the perf VM +without installing any packages. +""" + +import argparse +import glob +import json +import os +import sys + + +NS_PER_MS = 1_000_000.0 + + +def _load_benchmarks(directory): + """Return {key: record} for every benchmark found under *directory*. + + key = "Type.Method(Parameters)". record carries the mean (ns), the + allocated bytes/op, and the display fields used for reporting. + """ + records = {} + pattern = os.path.join(directory, "**", "*-report-full.json") + for path in sorted(glob.glob(pattern, recursive=True)): + try: + with open(path, "r", encoding="utf-8-sig") as handle: + data = json.load(handle) + except (OSError, ValueError) as exc: + print(f"WARNING: could not parse {path}: {exc}", file=sys.stderr) + continue + + for bench in data.get("Benchmarks", []): + btype = bench.get("Type", "") + method = bench.get("Method", "") + params = bench.get("Parameters", "") or "" + stats = bench.get("Statistics") or {} + mean_ns = stats.get("Mean") + if mean_ns is None: + continue + memory = bench.get("Memory") or {} + alloc = memory.get("BytesAllocatedPerOperation") + + key = f"{btype}.{method}({params})" + records[key] = { + "benchmarkName": btype, + "methodName": method, + "parameterSignature": params, + "meanNs": float(mean_ns), + "allocatedBytes": float(alloc) if alloc is not None else None, + } + return records + + +def _pct(baseline, current): + if baseline is None or current is None: + return None + if baseline == 0: + # A percentage change from a zero baseline is undefined (0 -> 0 is no change; 0 -> X is an + # infinite increase). Return 0.0 only for the genuine no-change case; otherwise None, and let + # callers surface the raw before/after values so a 0 -> X regression is still visible. + return 0.0 if current == 0 else None + return (current - baseline) / baseline * 100.0 + + +def build_comparison(baseline_dir, current_dir, threshold_pct): + baseline = _load_benchmarks(baseline_dir) + current = _load_benchmarks(current_dir) + + entries = [] + for key in sorted(set(baseline) | set(current)): + b = baseline.get(key) + c = current.get(key) + ref = c or b + entry = { + "key": key, + "benchmarkName": ref["benchmarkName"], + "methodName": ref["methodName"], + "parameterSignature": ref["parameterSignature"], + "baselineMeanMs": (b["meanNs"] / NS_PER_MS) if b else None, + "currentMeanMs": (c["meanNs"] / NS_PER_MS) if c else None, + "baselineAllocBytes": b["allocatedBytes"] if b else None, + "currentAllocBytes": c["allocatedBytes"] if c else None, + } + + if b and c: + entry["meanDeltaPct"] = _pct(b["meanNs"], c["meanNs"]) + entry["meanRatio"] = (c["meanNs"] / b["meanNs"]) if b["meanNs"] else None + # Compute the allocation delta whenever both sides report a value. A 0-byte baseline is + # valid, so gate on 'is not None' rather than truthiness -- gating on truthiness would drop + # a real 0 -> X allocation regression. The percentage itself is undefined for a 0 baseline + # (see _pct); the raw baseline/current byte counts on the entry keep 0 -> X visible. + if b["allocatedBytes"] is not None and c["allocatedBytes"] is not None: + entry["allocDeltaPct"] = _pct(b["allocatedBytes"], c["allocatedBytes"]) + else: + entry["allocDeltaPct"] = None + delta = entry["meanDeltaPct"] + if delta is None: + entry["status"] = "unknown" + elif delta > threshold_pct: + entry["status"] = "regression" + elif delta < -threshold_pct: + entry["status"] = "improvement" + else: + entry["status"] = "unchanged" + elif c and not b: + entry["meanDeltaPct"] = None + entry["meanRatio"] = None + entry["allocDeltaPct"] = None + entry["status"] = "current-only" + else: + entry["meanDeltaPct"] = None + entry["meanRatio"] = None + entry["allocDeltaPct"] = None + entry["status"] = "baseline-only" + + entries.append(entry) + + # Sort worst-regression first, then by name for stability. + def _sort_key(e): + d = e["meanDeltaPct"] + return (-(d if d is not None else -1e18), e["key"]) + + entries.sort(key=_sort_key) + return entries + + +def _fmt_ms(value): + return f"{value:.4f}" if value is not None else "-" + + +def _fmt_pct(value): + if value is None: + return "-" + return f"{value:+.2f}%" + + +def _fmt_bytes(value): + return f"{int(value)}" if value is not None else "-" + + +def _fmt_alloc(entry): + """Alloc column: a percentage when it is defined, otherwise the raw byte transition so a + 0 -> X regression (undefined as a percentage) is still shown rather than collapsing to '-'.""" + pct = entry.get("allocDeltaPct") + if pct is not None: + return f"{pct:+.2f}%" + base = entry.get("baselineAllocBytes") + cur = entry.get("currentAllocBytes") + if base is not None and cur is not None and base != cur: + return f"{int(base)} → {int(cur)} B" + return "-" + + +def render_markdown(entries, baseline_version, threshold_pct): + regressions = [e for e in entries if e["status"] == "regression"] + improvements = [e for e in entries if e["status"] == "improvement"] + + lines = [] + lines.append("# SqlClient Performance Comparison") + lines.append("") + lines.append(f"Baseline: **{baseline_version}**  |  " + f"Regression threshold: **{threshold_pct:.0f}%**") + lines.append("") + lines.append(f"- Benchmarks compared: **{len(entries)}**") + lines.append(f"- Regressions (slower > {threshold_pct:.0f}%): **{len(regressions)}**") + lines.append(f"- Improvements (faster > {threshold_pct:.0f}%): **{len(improvements)}**") + lines.append("") + lines.append("| Status | Benchmark | Method | Params | Baseline (ms) | " + "Current (ms) | Mean Δ | Alloc Δ |") + lines.append("| ------ | --------- | ------ | ------ | ------------- | " + "------------ | ------ | ------- |") + + icon = { + "regression": "🔴 regression", + "improvement": "🟢 improvement", + "unchanged": "⚪ unchanged", + "current-only": "🆕 new", + "baseline-only": "➖ removed", + "unknown": "❔", + } + for e in entries: + lines.append( + "| {status} | {name} | {method} | {params} | {base} | {cur} | " + "{delta} | {alloc} |".format( + status=icon.get(e["status"], e["status"]), + name=e["benchmarkName"], + method=e["methodName"], + params=e["parameterSignature"] or "-", + base=_fmt_ms(e["baselineMeanMs"]), + cur=_fmt_ms(e["currentMeanMs"]), + delta=_fmt_pct(e["meanDeltaPct"]), + alloc=_fmt_alloc(e), + ) + ) + lines.append("") + return "\n".join(lines) + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--baseline-dir", required=True) + parser.add_argument("--current-dir", required=True) + parser.add_argument("--out-md", required=True) + parser.add_argument("--out-json", required=True) + parser.add_argument("--baseline-version", default="baseline") + parser.add_argument("--threshold", type=float, default=10.0, + help="Percent slowdown that counts as a regression.") + parser.add_argument("--fail-on-regression", action="store_true", + help="Exit non-zero if any regression is detected.") + args = parser.parse_args(argv) + + entries = build_comparison(args.baseline_dir, args.current_dir, args.threshold) + + os.makedirs(os.path.dirname(os.path.abspath(args.out_md)), exist_ok=True) + os.makedirs(os.path.dirname(os.path.abspath(args.out_json)), exist_ok=True) + + markdown = render_markdown(entries, args.baseline_version, args.threshold) + with open(args.out_md, "w", encoding="utf-8") as handle: + handle.write(markdown + "\n") + + with open(args.out_json, "w", encoding="utf-8") as handle: + json.dump( + { + "baselineVersion": args.baseline_version, + "thresholdPct": args.threshold, + "entries": entries, + }, + handle, + indent=2, + ) + + regressions = [e for e in entries if e["status"] == "regression"] + print(f"Compared {len(entries)} benchmarks; {len(regressions)} regression(s).") + print(f"Wrote {args.out_md} and {args.out_json}.") + + if args.fail_on_regression and regressions: + print("Regressions detected; failing as requested.", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/eng/pipelines/perf/scripts/ingest_kusto.py b/eng/pipelines/perf/scripts/ingest_kusto.py new file mode 100644 index 0000000000..52f96c640e --- /dev/null +++ b/eng/pipelines/perf/scripts/ingest_kusto.py @@ -0,0 +1,337 @@ +#!/usr/bin/env python3 +"""Ingest translated perf-results NDJSON files into Azure Data Explorer (Kusto). + +Runs on the pipeline agent inside an ``AzureCLI@2`` task so that the Azure DevOps +service connection's service principal is already logged in via ``az``. The +script therefore authenticates to Kusto with *az CLI* credentials and performs a +queued ingestion of the PerfRun and PerfBenchmarkResult NDJSON files produced by +``perf_to_kusto.py``. + +Queued ingestion is asynchronous: ``ingest_from_file`` only *enqueues* the data +and returns success even when the data later fails to land (e.g. a schema +mismatch, or a *missing* server-side ingestion mapping - which the data-management +service silently retries for ~2 days before recording a failure). Those failures +surface only in Kusto's ingestion failure system, so the pipeline step used to go +green (or hang) while the database stayed empty. To avoid that this script: + + * sends a **self-contained inline JSON column mapping** (built from the payload's + own property names, which match the target table's columns) instead of + referencing a named server-side mapping that must be pre-created on the cluster, + * sets ``flush_immediately`` so the data-management service seals the batch + right away instead of waiting out the (default 5 minute) batching window, and + * verifies, after queuing, that the expected rows actually became queryable -- + polling the target tables and, on timeout, printing ``.show ingestion + failures`` so the real error is visible in the build log (and failing the + step). + +Requires the ``azure-kusto-ingest`` package (``pip install azure-kusto-data +azure-kusto-ingest``); the pipeline step installs it before invoking this script. +""" + +import argparse +import json +import os +import sys +import time + + +def _engine_uri(cluster): + """Return the engine (query) endpoint, stripping any 'ingest-' prefix.""" + if "://" in cluster: + scheme, rest = cluster.split("://", 1) + if rest.startswith("ingest-"): + rest = rest[len("ingest-"):] + return f"{scheme}://{rest}" + return cluster + + +def _ingest_uri(cluster): + """Return the data-management (ingest) endpoint used by queued ingestion.""" + if "://" in cluster: + scheme, rest = cluster.split("://", 1) + if not rest.startswith("ingest-"): + return f"{scheme}://ingest-{rest}" + return cluster + + +def _ndjson_columns(data_file): + """Ordered union of the JSON property names across every record in an NDJSON file. + + The translated payload uses property names that are identical to the target table's + column names, so this doubles as the set of columns to map.""" + columns = [] + seen = set() + with open(data_file, "r", encoding="utf-8") as fh: + for line in fh: + line = line.strip() + if not line: + continue + try: + obj = json.loads(line) + except ValueError: + continue + for key in obj: + if key not in seen: + seen.add(key) + columns.append(key) + return columns + + +def _inline_json_mapping(data_file): + """Build an inline JSON column mapping (column -> $.column) from the payload's own keys. + + Ingestion deliberately does NOT reference a pre-created server-side named mapping: relying on + one silently breaks ingestion whenever the mapping is missing from the cluster (the queued + request is retried for ~2 days before a failure is even recorded). Because the NDJSON property + names match the table column names 1:1, a self-contained inline mapping removes that external + dependency entirely.""" + from azure.kusto.ingest import ColumnMapping + + # An empty column_type lets Kusto use each table column's declared type (works for string, + # numeric, datetime, bool and dynamic columns alike). + return [ColumnMapping(column_name=col, column_type="", path=f"$.{col}") + for col in _ndjson_columns(data_file)] + + +def _ingest(cluster, database, table, data_file): + from azure.kusto.data import KustoConnectionStringBuilder + from azure.kusto.ingest import ( + QueuedIngestClient, + IngestionProperties, + FileDescriptor, + ) + from azure.kusto.data.data_format import DataFormat, IngestionMappingKind + from azure.kusto.ingest import ReportLevel + + if not os.path.exists(data_file) or os.path.getsize(data_file) == 0: + print(f"Skipping {table}: '{data_file}' is missing or empty.") + return 0 + + column_mappings = _inline_json_mapping(data_file) + if not column_mappings: + print(f"Skipping {table}: '{data_file}' has no recognizable JSON records.") + return 0 + + kcsb = KustoConnectionStringBuilder.with_az_cli_authentication(_ingest_uri(cluster)) + client = QueuedIngestClient(kcsb) + + props = IngestionProperties( + database=database, + table=table, + data_format=DataFormat.MULTIJSON, + column_mappings=column_mappings, + ingestion_mapping_kind=IngestionMappingKind.JSON, + report_level=ReportLevel.FailuresAndSuccesses, + # Seal the batch immediately: perf payloads are tiny and we want the rows + # queryable in seconds (and any verification to run promptly), not after + # the default multi-minute batching window. + flush_immediately=True, + ) + + descriptor = FileDescriptor(data_file, os.path.getsize(data_file)) + client.ingest_from_file(descriptor, ingestion_properties=props) + print(f"Queued ingestion of '{data_file}' -> {database}.{table} " + f"(inline JSON mapping, {len(column_mappings)} columns).") + return 0 + + +def _count_rows(path): + """Number of non-empty NDJSON lines in a file (0 when missing).""" + if not os.path.exists(path): + return 0 + with open(path, "r", encoding="utf-8") as fh: + return sum(1 for line in fh if line.strip()) + + +def _pipeline_ids(paths): + """Collect the distinct PipelineRunId values from PerfRun NDJSON files.""" + ids = set() + for path in paths: + if not os.path.exists(path): + continue + with open(path, "r", encoding="utf-8") as fh: + for line in fh: + line = line.strip() + if not line: + continue + try: + pid = json.loads(line).get("PipelineRunId") + except ValueError: + pid = None + if pid: + ids.add(str(pid)) + return ids + + +def _scalar(client, database, query): + resp = client.execute(database, query) + return int(resp.primary_results[0].rows[0][0]) + + +def _is_authorization_error(exc): + """True when an exception looks like a Kusto authorization/permission denial + (as opposed to a transient network error), so the operator can be told to + grant the querying role rather than chase a connectivity problem.""" + if exc is None: + return False + text = str(exc).lower() + needles = ( + "forbidden", "unauthorized", "not authorized", "does not have permission", + "principal", "403", "e_access", "access denied", + ) + return any(n in text for n in needles) + + +def _dump_failures(client, database): + cmd = (".show ingestion failures " + "| where Table in ('PerfRun', 'PerfBenchmarkResult') " + "| where FailedOn > ago(2h) " + "| project FailedOn, Table, ErrorCode, FailureKind, Details, OperationId " + "| order by FailedOn desc | take 25") + try: + resp = client.execute_mgmt(database, cmd) + rows = resp.primary_results[0].rows + if not rows: + print("No recent ingestion failures reported by Kusto " + "(data may still be flushing).", file=sys.stderr) + return + print("Recent Kusto ingestion failures:", file=sys.stderr) + for r in rows: + print(f" {r[0]} {r[1]} [{r[2]}/{r[3]}] {r[4]} (op {r[5]})", + file=sys.stderr) + except Exception as exc: # noqa: BLE001 - diagnostics best-effort + print(f" (could not query ingestion failures: {exc})", file=sys.stderr) + + +def _verify(cluster, database, run_table, results_table, pipeline_ids, + expected_run, expected_results, timeout_s, interval_s): + """Poll the target tables until the expected rows appear. + + Returns 0 on success, 1 when rows are provably missing after the timeout. + When verification queries cannot run at all (e.g. the ingestion principal + lacks query rights) this is best-effort: it warns and returns 0 so the step + is not failed on a permission gap, since the ingestion itself was queued. + """ + if not pipeline_ids or (expected_run == 0 and expected_results == 0): + print("Nothing to verify (no rows were queued).") + return 0 + + from azure.kusto.data import KustoClient, KustoConnectionStringBuilder + + client = KustoClient( + KustoConnectionStringBuilder.with_az_cli_authentication(_engine_uri(cluster))) + + id_list = ", ".join(f'"{i}"' for i in sorted(pipeline_ids)) + run_query = f"{run_table} | where PipelineRunId in ({id_list}) | count" + # PerfBenchmarkResult has no PipelineRunId column, but DerivedRunId embeds it + # as the trailing '|' segment. + res_conds = " or ".join(f'DerivedRunId endswith "|{i}"' for i in sorted(pipeline_ids)) + res_query = f"{results_table} | where {res_conds} | count" + + print(f"Verifying ingestion for PipelineRunId in [{id_list}] " + f"(expecting {run_table}>={expected_run}, {results_table}>={expected_results}); " + f"timeout {timeout_s}s ...") + + deadline = time.time() + timeout_s + query_ever_ok = False + last_exc = None + run_have = res_have = -1 + while True: + try: + run_have = _scalar(client, database, run_query) + res_have = _scalar(client, database, res_query) + query_ever_ok = True + except Exception as exc: # noqa: BLE001 - transient/permission, retried + last_exc = exc + print(f" (verification query failed, will retry: {exc})") + + if query_ever_ok: + print(f" {run_table}={run_have}/{expected_run}, " + f"{results_table}={res_have}/{expected_results}") + if run_have >= expected_run and res_have >= expected_results: + print("Verified: all expected rows are queryable in Kusto.") + return 0 + + if time.time() >= deadline: + break + time.sleep(interval_s) + + if not query_ever_ok: + detail = str(last_exc) if last_exc is not None else "no further detail" + if _is_authorization_error(last_exc): + # Queued ingestion only needs the 'Database Ingestor' role; verification *queries* the + # data back and additionally needs 'Database Viewer'. A principal with Ingestor-only + # rights ingests fine but cannot verify, so name the missing role instead of blaming the + # network. The ingestion itself was queued, so this stays a warning (exit 0). + print("##vso[task.logissue type=warning]Kusto ingestion was queued, but the ingestion " + "principal is not authorized to query the database, so ingestion could not be " + "verified. Grant the service connection's service principal the 'Database Viewer' " + "role on this database (in addition to 'Database Ingestor'). " + f"Details: {detail}", file=sys.stderr) + else: + print("##vso[task.logissue type=warning]Could not verify Kusto ingestion " + "(query endpoint unreachable or the verification query failed). Data was queued; " + f"check the database manually. Details: {detail}", file=sys.stderr) + return 0 + + print(f"##vso[task.logissue type=error]Kusto ingestion did not complete: " + f"{run_table}={run_have}/{expected_run}, " + f"{results_table}={res_have}/{expected_results} after {timeout_s}s.", + file=sys.stderr) + _dump_failures(client, database) + return 1 + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--cluster", required=True, + help="Kusto cluster URI, e.g. https://..kusto.windows.net") + parser.add_argument("--database", required=True) + parser.add_argument("--run-file", required=True, + help="PerfRun NDJSON file(s), comma-separated.") + parser.add_argument("--results-file", required=True, + help="PerfBenchmarkResult NDJSON file(s), comma-separated.") + parser.add_argument("--run-table", default="PerfRun") + parser.add_argument("--results-table", default="PerfBenchmarkResult") + parser.add_argument("--verify-timeout", type=int, + default=int(os.environ.get("KUSTO_VERIFY_TIMEOUT", "300")), + help="Seconds to wait for ingested rows to become queryable " + "(0 disables verification).") + parser.add_argument("--verify-interval", type=int, + default=int(os.environ.get("KUSTO_VERIFY_INTERVAL", "15")), + help="Polling interval in seconds for verification.") + args = parser.parse_args(argv) + + try: + import azure.kusto.ingest # noqa: F401 + except ImportError: + print("ERROR: azure-kusto-ingest is not installed. " + "Run 'pip install azure-kusto-data azure-kusto-ingest'.", + file=sys.stderr) + return 2 + + run_files = [f.strip() for f in args.run_file.split(",") if f.strip()] + results_files = [f.strip() for f in args.results_file.split(",") if f.strip()] + + expected_run = sum(_count_rows(f) for f in run_files) + expected_results = sum(_count_rows(f) for f in results_files) + pipeline_ids = _pipeline_ids(run_files) + + for data_file in run_files: + _ingest(args.cluster, args.database, args.run_table, data_file) + for data_file in results_files: + _ingest(args.cluster, args.database, args.results_table, data_file) + + print("Ingestion requests submitted (queued).") + + if args.verify_timeout <= 0: + print("Verification disabled (--verify-timeout <= 0).") + return 0 + + return _verify(args.cluster, args.database, args.run_table, args.results_table, + pipeline_ids, expected_run, expected_results, + args.verify_timeout, args.verify_interval) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/eng/pipelines/perf/scripts/interleave_perf.py b/eng/pipelines/perf/scripts/interleave_perf.py new file mode 100644 index 0000000000..2489f269dd --- /dev/null +++ b/eng/pipelines/perf/scripts/interleave_perf.py @@ -0,0 +1,413 @@ +#!/usr/bin/env python3 +"""Interleaved, best-of-N benchmark orchestrator for the SqlClient perf pipeline. + +Implements the two structural noise-reduction controls from InternalDriverTools +wiki 339 ("Reducing Noise in Performance Tests"): + +* **Interleaving (§2.2 / §2.3)** — instead of running the whole baseline suite and + then the whole candidate suite (so a given benchmark is measured tens of minutes + apart), this runs *one benchmark unit at a time*, executing the baseline build and + the candidate build back-to-back for that unit before moving on. Any slow host + drift then affects both sides of each comparison roughly equally. + +* **Best-of-N confirmation (§2.6)** — a single measurement can flag a regression + purely from noise. After the first interleaved pass, only the units that showed a + regression are re-run N-1 more times (still interleaved); a regression is + *confirmed* only when a strict majority of the N passes agree. This is what makes + the ``--fail-on-regression`` gate trustworthy. + +The benchmark executable is the ``PerformanceTests`` app built two ways (same test +sources, different Microsoft.Data.SqlClient reference). It understands two env vars +added for this orchestrator: + +* ``PERF_LIST_BENCHMARKS`` — print the enabled unit names and exit. +* ``PERF_BENCHMARK=`` — run only that unit. + +Only the Python standard library is used so it runs on the perf VM unchanged. +""" + +import argparse +import glob +import json +import os +import shutil +import subprocess +import sys + + +SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, SCRIPT_DIR) +import compare_perf # noqa: E402 (local module, sits next to this script) + + +# -------------------------------------------------------------------------------------------------- +# CPU affinity (client pinning, wiki §2.4) — best effort, cross platform. +# -------------------------------------------------------------------------------------------------- +def parse_cpus(spec): + """Parse a CPU spec like "16-31" or "0,2,4" or "0-3,8" into a sorted int list. + + CPU pinning is a best-effort optimisation, never a gate, so a malformed spec must not + fail the run: on any unparseable segment this warns and returns [] (i.e. no pinning).""" + if not spec: + return [] + cpus = set() + try: + for part in spec.split(","): + part = part.strip() + if not part: + continue + if "-" in part: + lo, hi = part.split("-", 1) + cpus.update(range(int(lo), int(hi) + 1)) + else: + cpus.add(int(part)) + except ValueError: + print(f"WARNING: could not parse CPU spec {spec!r}; running without CPU pinning.", + file=sys.stderr) + return [] + return sorted(cpus) + + +def apply_affinity(proc, cpus): + """Pin *proc* to *cpus*. Never raises — pinning is an optimisation, not a gate.""" + if not cpus: + return + try: + if hasattr(os, "sched_setaffinity"): # Linux + os.sched_setaffinity(proc.pid, set(cpus)) + return + if os.name == "nt": # Windows + import ctypes + + mask = 0 + for c in cpus: + mask |= (1 << c) + handle = int(proc._handle) # noqa: SLF001 (Popen exposes the OS handle here) + if ctypes.windll.kernel32.SetProcessAffinityMask(handle, ctypes.c_size_t(mask)) == 0: + print(f"WARNING: SetProcessAffinityMask failed for pid {proc.pid}.", file=sys.stderr) + except Exception as exc: # noqa: BLE001 + print(f"WARNING: could not pin pid {getattr(proc, 'pid', '?')} to CPUs {cpus}: {exc}", + file=sys.stderr) + + +# -------------------------------------------------------------------------------------------------- +# Running one unit and collecting its artifacts. +# -------------------------------------------------------------------------------------------------- +def run_unit_process(exe_dir, assembly, unit, cwd, cpus, log_path): + """Run one benchmark *unit* from the build at *exe_dir* in *cwd*. + + Returns the subprocess return code. Kept as a small seam so tests can substitute + a fake runner. + """ + os.makedirs(cwd, exist_ok=True) + cmd = ["dotnet", os.path.join(exe_dir, assembly)] + env = dict(os.environ) + env["PERF_BENCHMARK"] = unit + env.pop("PERF_LIST_BENCHMARKS", None) + + with open(log_path, "w", encoding="utf-8") as log: + proc = subprocess.Popen(cmd, cwd=cwd, env=env, stdout=log, + stderr=subprocess.STDOUT) + apply_affinity(proc, cpus) + return proc.wait() + + +def _report_files(root): + return sorted(glob.glob(os.path.join(root, "**", "*-report-full.json"), recursive=True)) + + +def collect_results(cwd, dest): + """Copy the BenchmarkDotNet reports produced under *cwd* into *dest*. + + Returns the set of benchmark ``Type`` names found (used to map a unit to the + report rows it produced, so best-of-N can re-run the right unit for a flagged key). + """ + os.makedirs(dest, exist_ok=True) + types = set() + artifacts = os.path.join(cwd, "BenchmarkDotNet.Artifacts") + for path in _report_files(artifacts): + shutil.copy2(path, os.path.join(dest, os.path.basename(path))) + try: + with open(path, "r", encoding="utf-8-sig") as fh: + for bench in json.load(fh).get("Benchmarks", []): + if bench.get("Type"): + types.add(bench["Type"]) + except (OSError, ValueError): + pass + # Also keep the human-readable GitHub markdown reports alongside, best effort. + for md in glob.glob(os.path.join(artifacts, "**", "*-report-github.md"), recursive=True): + shutil.copy2(md, os.path.join(dest, os.path.basename(md))) + return types + + +# -------------------------------------------------------------------------------------------------- +# Interleaved passes. +# -------------------------------------------------------------------------------------------------- +class Runner: + """Holds the invariant run parameters and performs interleaved unit passes.""" + + def __init__(self, baseline_dir, current_dir, assembly, work_dir, cpus): + self.baseline_dir = baseline_dir + self.current_dir = current_dir + self.assembly = assembly + self.work_dir = work_dir + self.cpus = cpus + + def list_units(self): + cmd = ["dotnet", os.path.join(self.current_dir, self.assembly)] + env = dict(os.environ) + env["PERF_LIST_BENCHMARKS"] = "1" + env.pop("PERF_BENCHMARK", None) + out = subprocess.check_output(cmd, cwd=self.work_dir, env=env, text=True) + return [line.strip() for line in out.splitlines() if line.strip()] + + def _run_one(self, variant, exe_dir, unit, rep, agg_dir): + cwd = os.path.join(self.work_dir, f"rep{rep}", variant, unit) + if os.path.isdir(cwd): + shutil.rmtree(cwd) + os.makedirs(cwd, exist_ok=True) + log_path = os.path.join(cwd, "run.log") + rc = run_unit_process(exe_dir, self.assembly, unit, cwd, self.cpus, log_path) + if rc != 0: + _tail(log_path) + raise RuntimeError(f"benchmark unit '{unit}' ({variant}, rep {rep}) failed (exit {rc}).") + types = collect_results(cwd, agg_dir) + if not types: + _tail(log_path) + raise RuntimeError( + f"benchmark unit '{unit}' ({variant}, rep {rep}) produced no results " + f"(a broken benchmark must not be reported as a pass).") + return types + + def interleave(self, units, rep, baseline_agg, current_agg): + """Run each unit baseline-then-candidate; aggregate reports per variant. + + Returns a dict mapping unit name -> set of report Type names it produced. + """ + unit_types = {} + for unit in units: + print(f"[rep {rep}] {unit}: baseline -> candidate", flush=True) + # Interleaved: measure baseline and candidate for this unit back-to-back. + self._run_one("baseline", self.baseline_dir, unit, rep, baseline_agg) + types = self._run_one("current", self.current_dir, unit, rep, current_agg) + unit_types[unit] = types + return unit_types + + +def _tail(path, n=40): + try: + with open(path, "r", encoding="utf-8", errors="replace") as fh: + lines = fh.readlines() + sys.stderr.write("".join(lines[-n:])) + except OSError: + pass + + +# -------------------------------------------------------------------------------------------------- +# Comparison + confirmation. +# -------------------------------------------------------------------------------------------------- +def _regression_keys(baseline_dir, current_dir, threshold): + entries = compare_perf.build_comparison(baseline_dir, current_dir, threshold) + return entries, {e["key"] for e in entries if e["status"] == "regression"} + + +def orchestrate(runner, units, results_dir, threshold, reps): + baseline_agg = os.path.join(results_dir, "baseline") + current_agg = os.path.join(results_dir, "current") + comparison_dir = os.path.join(results_dir, "comparison") + os.makedirs(comparison_dir, exist_ok=True) + + # --- Pass 1: interleave every enabled unit ----------------------------------------------------- + unit_types = runner.interleave(units, 1, baseline_agg, current_agg) + type_to_unit = {} + for unit, types in unit_types.items(): + for t in types: + type_to_unit[t] = unit + + entries, reg_keys_1 = _regression_keys(baseline_agg, current_agg, threshold) + + # Which units contain a rep-1 regression? Those are the best-of-N candidates. + candidate_units = [] + for e in entries: + if e["status"] == "regression": + unit = type_to_unit.get(e["benchmarkName"]) + if unit and unit not in candidate_units: + candidate_units.append(unit) + + # Per-key regression tally across reps (rep 1 counts once). + reg_counts = {k: 1 for k in reg_keys_1} + + # --- Passes 2..N: re-run only the candidate units, still interleaved --------------------------- + if reps > 1 and candidate_units: + print(f"Best-of-{reps}: confirming {len(candidate_units)} candidate unit(s): " + f"{', '.join(candidate_units)}", flush=True) + for rep in range(2, reps + 1): + b_dir = os.path.join(results_dir, "reps", f"rep{rep}", "baseline") + c_dir = os.path.join(results_dir, "reps", f"rep{rep}", "current") + runner.interleave(candidate_units, rep, b_dir, c_dir) + _, reg_keys_r = _regression_keys(b_dir, c_dir, threshold) + for k in reg_keys_1: + if k in reg_keys_r: + reg_counts[k] = reg_counts.get(k, 0) + 1 + + # --- Confirmation verdict (strict majority of N) ----------------------------------------------- + total_reps = reps + for e in entries: + key = e["key"] + if e["status"] == "regression": + count = reg_counts.get(key, 0) + confirmed = (count * 2) > total_reps + e["regressionReps"] = count + e["totalReps"] = total_reps + e["confirmedRegression"] = confirmed + if not confirmed: + e["status"] = "regression-unconfirmed" + else: + e["regressionReps"] = 0 + e["totalReps"] = total_reps + e["confirmedRegression"] = False + + confirmed = [e for e in entries if e.get("confirmedRegression")] + unconfirmed = [e for e in entries if e["status"] == "regression-unconfirmed"] + return entries, confirmed, unconfirmed + + +# -------------------------------------------------------------------------------------------------- +# Rendering. +# -------------------------------------------------------------------------------------------------- +_ICON = { + "regression": "🔴 regression", + "regression-unconfirmed": "🟠 regression (unconfirmed)", + "improvement": "🟢 improvement", + "unchanged": "⚪ unchanged", + "current-only": "🆕 new", + "baseline-only": "➖ removed", + "unknown": "❔", +} + + +def render_markdown(entries, confirmed, unconfirmed, baseline_version, threshold, reps): + improvements = [e for e in entries if e["status"] == "improvement"] + lines = [] + lines.append("# SqlClient Performance Comparison (interleaved, best-of-N)") + lines.append("") + lines.append(f"Baseline: **{baseline_version}**  |  " + f"Regression threshold: **{threshold:.0f}%**  |  " + f"Confirmation runs: **N = {reps}**") + lines.append("") + lines.append(f"- Benchmarks compared: **{len(entries)}**") + lines.append(f"- **Confirmed** regressions (majority of {reps}): **{len(confirmed)}**") + lines.append(f"- Unconfirmed (flagged once, not reproduced): **{len(unconfirmed)}**") + lines.append(f"- Improvements (faster > {threshold:.0f}%): **{len(improvements)}**") + lines.append("") + lines.append("| Status | Benchmark | Method | Params | Baseline (ms) | " + "Current (ms) | Mean Δ | Alloc Δ | Confirm |") + lines.append("| ------ | --------- | ------ | ------ | ------------- | " + "------------ | ------ | ------- | ------- |") + for e in entries: + if e["totalReps"] > 1 and (e["confirmedRegression"] or e["status"] == "regression-unconfirmed"): + confirm = f"{e['regressionReps']}/{e['totalReps']}" + else: + confirm = "-" + lines.append( + "| {status} | {name} | {method} | {params} | {base} | {cur} | " + "{delta} | {alloc} | {confirm} |".format( + status=_ICON.get(e["status"], e["status"]), + name=e["benchmarkName"], + method=e["methodName"], + params=e["parameterSignature"] or "-", + base=compare_perf._fmt_ms(e["baselineMeanMs"]), + cur=compare_perf._fmt_ms(e["currentMeanMs"]), + delta=compare_perf._fmt_pct(e["meanDeltaPct"]), + alloc=compare_perf._fmt_pct(e["allocDeltaPct"]), + confirm=confirm, + ) + ) + lines.append("") + return "\n".join(lines) + + +# -------------------------------------------------------------------------------------------------- +# Entry point. +# -------------------------------------------------------------------------------------------------- +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--baseline-exe-dir", required=True, + help="Directory of the baseline (released package) build.") + parser.add_argument("--current-exe-dir", required=True, + help="Directory of the candidate (from-source) build.") + parser.add_argument("--assembly", default="PerformanceTests.dll") + parser.add_argument("--results-dir", required=True) + parser.add_argument("--work-dir", default=None, + help="Scratch directory for per-unit run dirs (default: /../perf-interleave-work).") + parser.add_argument("--threshold", type=float, default=10.0) + parser.add_argument("--reps", type=int, default=3, + help="Total interleaved passes for a flagged unit (best-of-N). 1 disables confirmation.") + parser.add_argument("--baseline-version", default="baseline") + parser.add_argument("--client-cpus", default=os.environ.get("PERF_CLIENT_CPUS", ""), + help="CPU set to pin the benchmark client to, e.g. '16-31'.") + parser.add_argument("--fail-on-regression", action="store_true", + help="Exit non-zero if any CONFIRMED regression is detected.") + args = parser.parse_args(argv) + + if args.reps < 1: + parser.error("--reps must be >= 1") + + results_dir = os.path.abspath(args.results_dir) + work_dir = os.path.abspath(args.work_dir) if args.work_dir \ + else os.path.join(os.path.dirname(results_dir), "perf-interleave-work") + os.makedirs(work_dir, exist_ok=True) + os.makedirs(results_dir, exist_ok=True) + + cpus = parse_cpus(args.client_cpus) + if cpus: + print(f"Pinning benchmark client to CPUs {args.client_cpus} ({len(cpus)} core(s)).") + else: + print("PERF_CLIENT_CPUS not set; running without CPU pinning.", file=sys.stderr) + + runner = Runner(os.path.abspath(args.baseline_exe_dir), + os.path.abspath(args.current_exe_dir), + args.assembly, work_dir, cpus) + + units = runner.list_units() + if not units: + print("ERROR: no enabled benchmark units were reported by the current build.", file=sys.stderr) + return 1 + print(f"Enabled units ({len(units)}): {', '.join(units)}") + + entries, confirmed, unconfirmed = orchestrate( + runner, units, results_dir, args.threshold, args.reps) + + # Outputs. + comparison_dir = os.path.join(results_dir, "comparison") + md = render_markdown(entries, confirmed, unconfirmed, args.baseline_version, args.threshold, args.reps) + with open(os.path.join(comparison_dir, "comparison.md"), "w", encoding="utf-8") as fh: + fh.write(md + "\n") + with open(os.path.join(comparison_dir, "comparison.json"), "w", encoding="utf-8") as fh: + json.dump({ + "baselineVersion": args.baseline_version, + "thresholdPct": args.threshold, + "confirmationRuns": args.reps, + "confirmedRegressions": len(confirmed), + "unconfirmedRegressions": len(unconfirmed), + "entries": entries, + }, fh, indent=2) + # Surface the comparison as the top-level run summary (collect-results attaches results/*.md). + shutil.copyfile(os.path.join(comparison_dir, "comparison.md"), + os.path.join(results_dir, "summary.md")) + + print(f"Interleaved comparison complete: {len(confirmed)} confirmed regression(s), " + f"{len(unconfirmed)} unconfirmed.") + + if args.fail_on_regression and confirmed: + print("Confirmed regressions detected; failing as requested.", file=sys.stderr) + for e in confirmed: + print(f" {e['key']}: {compare_perf._fmt_pct(e['meanDeltaPct'])} " + f"({e['regressionReps']}/{e['totalReps']} reps)", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/eng/pipelines/perf/scripts/perf_to_kusto.py b/eng/pipelines/perf/scripts/perf_to_kusto.py new file mode 100644 index 0000000000..5e10b08d90 --- /dev/null +++ b/eng/pipelines/perf/scripts/perf_to_kusto.py @@ -0,0 +1,336 @@ +#!/usr/bin/env python3 +"""Translate BenchmarkDotNet "full" JSON reports into Kusto perf-results rows. + +Implements the mapping documented on the InternalDriverTools wiki, page 270 +("Performance Results Database Specification") and the SqlClient conversion +subpage. Produces two newline-delimited JSON (NDJSON) files ready for Kusto +ingestion: + + * PerfRun - one row describing this run (baseline OR current). + * PerfBenchmarkResult - one row per benchmark method/parameter combination. + +All benchmark timings in BenchmarkDotNet are expressed in NANOSECONDS; the +schema stores milliseconds, so every time value is divided by 1,000,000. + +Only the Python standard library is required. +""" + +import argparse +import datetime +import glob +import json +import os +import re +import sys + + +NS_PER_MS = 1_000_000.0 + + +def _utcnow_iso(): + return datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ") + + +# Top-level runner-config keys that must never be captured into PerfRun.Config: +# * ConnectionString - contains server/credential details (secret; not a comparable knob). +# * Benchmarks - per-runner tuning object, not a boolean feature flag. +_CONFIG_EXCLUDED_KEYS = frozenset(("ConnectionString", "Benchmarks")) + + +def _load_runner_config(path): + """Return the runner's boolean feature flags as a dict for the PerfRun.Config column. + + The benchmark runner is driven by a .jsonc config (see PerformanceTests/runnerconfig.jsonc). + Only its top-level *boolean* flags describe the SqlClient behaviour a run exercised + (e.g. UseManagedSniOnWindows, UseOptimizedAsyncBehaviour), so those are the values worth + recording alongside the results. ConnectionString and Benchmarks are excluded, and any + non-boolean top-level entry is ignored so the shape stays a flat {flag: bool} map. + + Returns an empty dict (and warns) when the file is missing or unparseable, so a config + problem degrades to an empty Config rather than failing the whole translation. + """ + if not path: + return {} + try: + with open(path, "r", encoding="utf-8-sig") as handle: + raw = handle.read() + except OSError as exc: + print(f"WARNING: could not read runner config {path}: {exc}", file=sys.stderr) + return {} + # Strip //-style line comments so the .jsonc content parses as plain JSON. The config uses + # only whole-line comments, so a leading-whitespace match is sufficient and never touches a + # '//' embedded in a string value. + no_comments = re.sub(r"(?m)^\s*//.*$", "", raw) + try: + cfg = json.loads(no_comments) + except ValueError as exc: + print(f"WARNING: could not parse runner config {path}: {exc}", file=sys.stderr) + return {} + if not isinstance(cfg, dict): + print(f"WARNING: runner config {path} is not a JSON object; ignoring.", file=sys.stderr) + return {} + return {key: value for key, value in cfg.items() + if key not in _CONFIG_EXCLUDED_KEYS and isinstance(value, bool)} + + +def _normalize_os(value): + """Map an OS descriptor to the schema's canonical 'Windows' or 'Linux'.""" + if not value: + return "" + v = value.strip().lower() + if "win" in v: + return "Windows" + if any(tok in v for tok in ("linux", "unix", "ubuntu", "debian", "centos", "alpine")): + return "Linux" + return value + + +def _normalize_arch(value): + """Map a process-architecture descriptor to the schema's lowercase form (e.g. X64 -> x64).""" + if not value: + return "" + return value.strip().lower() + + +def _normalize_run_type(value): + """Map the harness run mode to the schema's 'Sequential' or 'Interweaved'.""" + if not value: + return "" + v = value.strip().lower() + if v.startswith("seq"): + return "Sequential" + if v.startswith("inter"): # interleaved (harness) -> Interweaved (schema spelling) + return "Interweaved" + return value + + +def _branch_category(branch_name): + """Map a git ref to one of the schema's BranchCategory buckets.""" + if not branch_name: + return "other" + name = branch_name + for prefix in ("refs/heads/", "refs/"): + if name.startswith(prefix): + name = name[len(prefix):] + break + if name.startswith("pull/") or branch_name.startswith("refs/pull/"): + return "pull_request" + if name == "main" or name == "master": + return "main" + if name.startswith("release/"): + return "release" + if name.startswith("dev/"): + return "dev" + if name.startswith("feat/") or name.startswith("feature/"): + return "feature" + return "other" + + +def _parse_parameters(parameters): + """Parse BenchmarkDotNet's 'Parameters' string into a structured bag. + + Example input: "RowCount=1000, UseAsync=True" + """ + bag = {} + if not parameters: + return bag + for part in parameters.split(","): + part = part.strip() + if not part or "=" not in part: + continue + key, _, value = part.partition("=") + bag[key.strip()] = value.strip() + return bag + + +def _driver_specific_metrics(bench): + metrics = {} + memory = bench.get("Memory") or {} + for field in ("Gen0Collections", "Gen1Collections", "Gen2Collections", + "TotalOperations", "BytesAllocatedPerOperation"): + if field in memory and memory[field] is not None: + metrics[field] = memory[field] + for metric in bench.get("Metrics", []) or []: + descriptor = metric.get("Descriptor") or {} + legend = descriptor.get("Legend") or descriptor.get("Id") + if legend is not None and metric.get("Value") is not None: + metrics[legend] = metric["Value"] + return metrics + + +def _percentile(stats, name): + pct = stats.get("Percentiles") or {} + value = pct.get(name) + return (value / NS_PER_MS) if value is not None else None + + +def translate(input_dir, ctx): + """Return (run_row, [result_rows]) for every full-report JSON in input_dir.""" + derived_run_id = "{driver}|{commit}|{pipeline}".format( + driver=ctx["driver_name"], + commit=ctx["commit_hash"], + pipeline=ctx["pipeline_run_id"], + ) + now = _utcnow_iso() + + run_row = { + "DerivedRunId": derived_run_id, + "DriverName": ctx["driver_name"], + "MachineName": ctx["machine_name"], + "AgentName": ctx["agent_name"], + "OperatingSystem": _normalize_os(ctx.get("operating_system")), + "Architecture": _normalize_arch(ctx.get("architecture")), + "RunType": _normalize_run_type(ctx.get("run_type")), + "PipelineRunId": ctx["pipeline_run_id"], + "BuildUrl": ctx["build_url"], + "RunDate": ctx["run_date"] or now, + "BranchName": ctx["branch_name"], + "BranchCategory": _branch_category(ctx["branch_name"]), + "VersionString": ctx["version_string"], + "CommitHash": ctx["commit_hash"], + "CommitDate": ctx["commit_date"] or None, + "IsComparableBase": bool(ctx["is_comparable_base"]), + "Config": ctx.get("config") or {}, + "IngestedAt": now, + } + + result_rows = [] + runtime_seen = None + platform_seen = None + os_seen = None + + pattern = os.path.join(input_dir, "**", "*-report-full.json") + for path in sorted(glob.glob(pattern, recursive=True)): + try: + with open(path, "r", encoding="utf-8-sig") as handle: + data = json.load(handle) + except (OSError, ValueError) as exc: + print(f"WARNING: could not parse {path}: {exc}", file=sys.stderr) + continue + + host = data.get("HostEnvironmentInfo") or {} + runtime = host.get("RuntimeVersion") + architecture = host.get("Architecture") + runtime_seen = runtime_seen or runtime + platform_seen = platform_seen or architecture + os_seen = os_seen or host.get("OsVersion") + + for bench in data.get("Benchmarks", []): + stats = bench.get("Statistics") or {} + mean_ns = stats.get("Mean") + if mean_ns is None: + continue + mean_ms = mean_ns / NS_PER_MS + + btype = bench.get("Type", "") + method = bench.get("Method", "") + params = bench.get("Parameters", "") or "" + memory = bench.get("Memory") or {} + + benchmark_id = "{run}|{name}|{method}|{sig}".format( + run=derived_run_id, name=btype, method=method, sig=params) + + result_rows.append({ + "BenchmarkId": benchmark_id, + "DerivedRunId": derived_run_id, + "BenchmarkName": btype, + "MethodName": method, + "ParameterSignature": params, + "ParameterBag": _parse_parameters(params), + "JobName": (bench.get("Job") or bench.get("JobConfig") or ""), + "MeanMs": mean_ms, + "P50Ms": _percentile(stats, "P50"), + "P95Ms": _percentile(stats, "P95"), + "P99Ms": _percentile(stats, "P99"), + "ThroughputOpsPerSec": (1000.0 / mean_ms) if mean_ms else None, + "ErrorMs": (stats.get("StandardError") / NS_PER_MS) + if stats.get("StandardError") is not None else None, + "StdDevMs": (stats.get("StandardDeviation") / NS_PER_MS) + if stats.get("StandardDeviation") is not None else None, + "MemoryAllocatedBytes": memory.get("BytesAllocatedPerOperation"), + "Runtime": runtime, + "Platform": architecture, + "DriverSpecificMetrics": _driver_specific_metrics(bench), + "IngestedAt": now, + }) + + # Fill OperatingSystem / Architecture from the benchmark host environment when the pipeline did + # not supply explicit overrides, normalizing to the schema's canonical values. + if not run_row["OperatingSystem"]: + run_row["OperatingSystem"] = _normalize_os(os_seen) + if not run_row["Architecture"]: + run_row["Architecture"] = _normalize_arch(platform_seen) + + return run_row, result_rows + + +def _write_ndjson(path, rows): + os.makedirs(os.path.dirname(os.path.abspath(path)), exist_ok=True) + with open(path, "w", encoding="utf-8") as handle: + for row in rows: + handle.write(json.dumps(row, separators=(",", ":")) + "\n") + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--input-dir", required=True, + help="Directory containing *-report-full.json for one pass.") + parser.add_argument("--out-run", required=True, help="PerfRun NDJSON output path.") + parser.add_argument("--out-results", required=True, + help="PerfBenchmarkResult NDJSON output path.") + parser.add_argument("--driver-name", default="Microsoft.Data.SqlClient") + parser.add_argument("--machine-name", default="") + parser.add_argument("--agent-name", default="") + parser.add_argument("--operating-system", default="", + help="OS the run executed on (Windows/Linux). " + "Derived from the benchmark host env when omitted.") + parser.add_argument("--architecture", default="", + help="Process architecture the run executed on (x64/x86). " + "Derived from the benchmark host env when omitted.") + parser.add_argument("--run-type", default="", + help="Benchmark execution ordering (Sequential/Interweaved).") + parser.add_argument("--pipeline-run-id", required=True) + parser.add_argument("--build-url", default="") + parser.add_argument("--run-date", default="") + parser.add_argument("--branch-name", default="") + parser.add_argument("--version-string", default="") + parser.add_argument("--commit-hash", required=True) + parser.add_argument("--commit-date", default="") + parser.add_argument("--is-comparable-base", default="false") + parser.add_argument("--runner-config", default="", + help="Path to the benchmark runner's .jsonc config. Its top-level boolean " + "flags (excluding ConnectionString and Benchmarks) are recorded in " + "PerfRun.Config.") + args = parser.parse_args(argv) + + ctx = { + "driver_name": args.driver_name, + "machine_name": args.machine_name, + "agent_name": args.agent_name, + "operating_system": args.operating_system, + "architecture": args.architecture, + "run_type": args.run_type, + "pipeline_run_id": args.pipeline_run_id, + "build_url": args.build_url, + "run_date": args.run_date, + "branch_name": args.branch_name, + "version_string": args.version_string, + "commit_hash": args.commit_hash, + "commit_date": args.commit_date, + "is_comparable_base": str(args.is_comparable_base).lower() in ("1", "true", "yes"), + "config": _load_runner_config(args.runner_config), + } + + run_row, result_rows = translate(args.input_dir, ctx) + _write_ndjson(args.out_run, [run_row]) + _write_ndjson(args.out_results, result_rows) + + print(f"Wrote 1 PerfRun row -> {args.out_run}") + print(f"Wrote {len(result_rows)} PerfBenchmarkResult row(s) -> {args.out_results}") + if not result_rows: + print("WARNING: no benchmark results were translated.", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/eng/pipelines/perf/scripts/run-perf-tests.ps1 b/eng/pipelines/perf/scripts/run-perf-tests.ps1 new file mode 100644 index 0000000000..d72b915b21 --- /dev/null +++ b/eng/pipelines/perf/scripts/run-perf-tests.ps1 @@ -0,0 +1,536 @@ +#################################################################################################### +# Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this +# file to you under the MIT license. See the LICENSE file in the project root for more information. +#################################################################################################### +# +# run-perf-tests.ps1 +# +# Entry point executed ON the Perf Test Lab Windows VM by the InternalDriverTools/PerfTest extends +# template (v1/Perf.Test.Job.yml). The template SCPs the driver source tree to the VM, runs this +# script over SSH, then SCPs the results sub-directory back and publishes it as a pipeline artifact. +# +# This is the Windows counterpart of run-perf-tests.sh. See that file for the full description of +# responsibilities. On Windows the benchmark client is pinned to the reserved CPU set via the +# process ProcessorAffinity mask (derived from PERF_CLIENT_CPUS) instead of taskset. +# +# Environment variables injected by the template (see wiki "Performance Test Automation"): +# SQL_SERVER Host/IP of the SQL Server on the perf VM (e.g. localhost). +# SQL_PASSWORD SQL Server 'sa' password. +# PERF_CLIENT_CPUS Core range reserved for the test client, e.g. "16-31". +# PERF_SQL_CPUS Core range SQL Server is pinned to, e.g. "0-15" (informational). +# +[CmdletBinding()] +param( + [string]$Configuration = "Release", + [string]$Framework = "net9.0", + [string]$ResultsSubdir = "perf-results", + [string]$BaselineVersion = "", + [string]$RegressionThreshold = "10", + # When set, a candidate-slower-than-baseline regression fails the run (wiki 339 §3 gate). + # Off by default so deltas are reported without blocking until the gate is trusted. + [switch]$FailOnRegression, + # Benchmark run model (wiki 339 §2.2/§2.3/§2.6): + # interleaved -> run one unit at a time, baseline and candidate back-to-back, with best-of-N + # confirmation of flagged regressions (the noise-resistant default). + # sequential -> legacy: run the whole baseline suite, then the whole candidate suite, compare. + [ValidateSet("interleaved", "sequential")] + [string]$RunMode = "interleaved", + # Best-of-N: total interleaved passes for a flagged unit before a regression is confirmed. + [int]$ConfirmationRuns = 3 +) + +$ErrorActionPreference = "Stop" +Set-StrictMode -Version Latest + +#################################################################################################### +# Native-command error handling +# +# The perf VM runs Windows PowerShell 5.1. There, with $ErrorActionPreference = 'Stop', ANY write +# to stderr by a native command (dotnet, python) is promoted to a TERMINATING error - even when the +# command's exit code is 0. The .NET CLI and Python routinely emit non-fatal diagnostics (SDK +# resolution notes, restore/build warnings, progress) to stderr, so an unguarded 'dotnet ...' or +# 'python3 ...' aborts the whole run and surfaces the tool's stderr text as the failure. +# +# PowerShell 7.3+ exposes $PSNativeCommandUseErrorActionPreference to opt out; set it where present. +# On 5.1 there is no such switch, so native tools are invoked through Invoke-Native, which relaxes +# the preference for the duration of the call and judges success solely by the process exit code. +#################################################################################################### + +if (Test-Path Variable:\PSNativeCommandUseErrorActionPreference) { + $PSNativeCommandUseErrorActionPreference = $false +} + +# Run a native command (in a scriptblock) with stderr-as-terminating-error suppressed, then throw +# $FailureMessage if it exited non-zero. Use for native calls whose non-zero exit must fail the run. +function Invoke-Native { + [CmdletBinding()] + param( + [Parameter(Mandatory, Position = 0)][scriptblock] $Command, + [Parameter(Position = 1)][string] $FailureMessage = "Native command failed" + ) + $previousPreference = $ErrorActionPreference + $ErrorActionPreference = 'Continue' + try { + & $Command + } finally { + $ErrorActionPreference = $previousPreference + } + if ($LASTEXITCODE -ne 0) { + throw "$FailureMessage (exit $LASTEXITCODE)." + } +} + +#################################################################################################### +# Resolve paths +#################################################################################################### + +# This script lives at /eng/pipelines/perf/scripts/run-perf-tests.ps1, so the repo root is +# four levels up. +$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path +$RepoRoot = (Resolve-Path (Join-Path $ScriptDir "..\..\..\..")).Path +$PerfProject = Join-Path $RepoRoot "src\Microsoft.Data.SqlClient\tests\PerformanceTests\Microsoft.Data.SqlClient.PerformanceTests.csproj" +$PerfDir = Split-Path -Parent $PerfProject +$ResultsDir = Join-Path $RepoRoot $ResultsSubdir + +$SqlServer = if ($env:SQL_SERVER) { $env:SQL_SERVER } else { "localhost" } +$SqlPassword = $env:SQL_PASSWORD +$DbName = "sqlclient-perf-db" + +Write-Host "==================================================================" +Write-Host " SqlClient Performance Tests" +Write-Host "==================================================================" +Write-Host " Repo root : $RepoRoot" +Write-Host " Perf project : $PerfProject" +Write-Host " Configuration : $Configuration" +Write-Host " Framework : $Framework" +Write-Host " Results dir : $ResultsDir" +Write-Host " Run mode : $RunMode (confirmation runs: $ConfirmationRuns)" +Write-Host " Baseline ver : $(if ($BaselineVersion) { $BaselineVersion } else { '' })" +Write-Host " SQL_SERVER : $SqlServer" +Write-Host " PERF_CLIENT_CPUS: $($env:PERF_CLIENT_CPUS)" +Write-Host " PERF_SQL_CPUS : $($env:PERF_SQL_CPUS)" +Write-Host "==================================================================" + +if (-not (Test-Path $PerfProject)) { + throw "Performance test project not found at $PerfProject" +} +if ([string]::IsNullOrEmpty($SqlPassword)) { + throw "SQL_PASSWORD environment variable is not set (expected from the perf template)." +} + +New-Item -ItemType Directory -Force -Path $ResultsDir | Out-Null + +# Record VM-side run metadata (e.g. the perf VM hostname) for the agent-side Kusto translation. +"MACHINE_NAME=$env:COMPUTERNAME" | Set-Content -Path (Join-Path $ResultsDir "runinfo.env") -Encoding ASCII + +$env:DOTNET_NOLOGO = "1" +$env:DOTNET_CLI_TELEMETRY_OPTOUT = "1" +$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE = "1" + +#################################################################################################### +# 1. Install the .NET SDK (pinned by global.json) and the runtimes for the target frameworks. +#################################################################################################### + +function Install-DotNet { + $globalJson = Get-Content (Join-Path $RepoRoot "global.json") -Raw + # Strip // comments so ConvertFrom-Json accepts the file. + $globalJson = ($globalJson -split "`n" | ForEach-Object { $_ -replace '//.*$', '' }) -join "`n" + $sdkVersion = (ConvertFrom-Json $globalJson).sdk.version + if ([string]::IsNullOrEmpty($sdkVersion)) { + throw "Could not determine SDK version from global.json" + } + + $dotnetRoot = Join-Path $env:USERPROFILE ".dotnet" + $env:DOTNET_ROOT = $dotnetRoot + $env:PATH = "$dotnetRoot;$dotnetRoot\tools;$env:PATH" + + Write-Host "Installing .NET SDK $sdkVersion into $dotnetRoot ..." + $installScript = Join-Path $env:TEMP "dotnet-install.ps1" + Invoke-WebRequest -UseBasicParsing "https://dot.net/v1/dotnet-install.ps1" -OutFile $installScript + + & $installScript -Version $sdkVersion -InstallDir $dotnetRoot + foreach ($channel in @("8.0", "9.0", "10.0")) { + & $installScript -Channel $channel -Runtime dotnet -InstallDir $dotnetRoot + } +} + +$hasNet10Sdk = $false +if (Get-Command dotnet -ErrorAction SilentlyContinue) { + # 'dotnet --version' evaluated from the repo root honours global.json (including rollForward), so + # it succeeds only when the pinned SDK is actually installed. A bare '10.0.*' match would accept + # the wrong SDK band and skip installing the pinned one. + Push-Location $RepoRoot + try { + # Relax Stop here: a missing/mismatched pinned SDK makes 'dotnet --version' write to stderr + # and exit non-zero, which under Stop would abort before we can fall through to Install-DotNet. + $previousPreference = $ErrorActionPreference + $ErrorActionPreference = 'Continue' + try { + dotnet --version *> $null + if ($LASTEXITCODE -eq 0) { $hasNet10Sdk = $true } + } finally { + $ErrorActionPreference = $previousPreference + } + } finally { + Pop-Location + } +} +if ($hasNet10Sdk) { + Write-Host "Using pre-installed dotnet: $((Get-Command dotnet).Source)" +} else { + Install-DotNet +} + +# Informational only: never let 'dotnet --info' (or its stderr) fail the run. +try { Invoke-Native { dotnet --info } "dotnet --info failed" } catch { Write-Warning $_.Exception.Message } + +#################################################################################################### +# 2. Create the perf database on the VM's SQL Server. +# +# The benchmark runners create their own tables but not the database, so create it here +# (idempotently) using sqlcmd. sqlcmd is required on the VM; if it is not present the script fails +# fast (throws below) rather than continuing on to run benchmarks against a missing database. +#################################################################################################### + +Write-Host "Ensuring database [$DbName] exists on $SqlServer ..." + +$sqlcmd = Get-Command sqlcmd -ErrorAction SilentlyContinue +if ($sqlcmd) { + # Relax Stop around the native sqlcmd call so a benign stderr write cannot abort the run before + # the explicit exit-code check below (Windows PowerShell 5.1 promotes native stderr under Stop). + $previousPreference = $ErrorActionPreference + $ErrorActionPreference = 'Continue' + try { + & $sqlcmd.Source -S $SqlServer -U sa -P $SqlPassword -C -b -l 30 ` + -Q "IF DB_ID('$DbName') IS NULL CREATE DATABASE [$DbName];" + } finally { + $ErrorActionPreference = $previousPreference + } + if ($LASTEXITCODE -ne 0) { throw "sqlcmd failed to create database [$DbName] (exit $LASTEXITCODE)." } + Write-Host "Database [$DbName] is ready." +} else { + throw "sqlcmd was not found on the VM; cannot create the perf database [$DbName]." +} + +#################################################################################################### +# Noise-reduction controls (InternalDriverTools wiki 339, "Reducing Noise in Performance Tests"). +# +# The Perf Test Lab already provides the isolated dedicated host, the tuned SQL instance and the +# disjoint client CPU set (PERF_CLIENT_CPUS, pinned per pass below). These are the remaining +# harness-owned controls: per-run diagnostics, a fail-loud preflight and a warm-up, so a run's +# mean/variance is steadier and a broken run cannot masquerade as a pass. (The glibc allocator and +# sysctl tuning from the Linux harness are Linux-only and intentionally omitted here.) +#################################################################################################### + +$DiagDir = Join-Path $ResultsDir "diagnostics" +New-Item -ItemType Directory -Force -Path $DiagDir | Out-Null + +# --- §2.11 Capture host CPU topology (static, once per run) --------------------------------------- +try { + Get-CimInstance Win32_Processor | + Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed, CurrentClockSpeed | + Format-List | Out-File -FilePath (Join-Path $DiagDir "cpu-info.txt") -Encoding UTF8 +} catch { Write-Warning "Could not capture CPU info: $_" } + +# --- §2.11 Capture the SQL instance configuration (confirm the lab tuning actually took effect) --- +try { + & $sqlcmd.Source -S $SqlServer -U sa -P $SqlPassword -C -b -l 30 -h -1 -W ` + -Q "SET NOCOUNT ON; + SELECT name, value_in_use FROM sys.configurations + WHERE name IN ('max degree of parallelism','cost threshold for parallelism', + 'max server memory (MB)','min server memory (MB)','affinity mask', + 'affinity I/O mask'); + SELECT 'tempdb_data_files' AS setting, COUNT(*) AS value FROM tempdb.sys.database_files WHERE type = 0; + SELECT @@VERSION;" ` + *> (Join-Path $DiagDir "sql-config.txt") + Write-Host "Captured SQL instance config -> $(Join-Path $DiagDir 'sql-config.txt')" +} catch { Write-Warning "Could not capture SQL instance config: $_" } + +# --- §2.10 / §2.5 Fail loud on an unreachable server, and warm the buffer pool / plan cache ------- +# A benchmark suite that "skips" when the server is down produces an empty comparison that reads +# green; verify connectivity up front and touch the target DB before the first measured benchmark. +$previousPreference = $ErrorActionPreference +$ErrorActionPreference = 'Continue' +try { + & $sqlcmd.Source -S $SqlServer -U sa -P $SqlPassword -C -b -l 15 ` + -Q "SET NOCOUNT ON; USE [$DbName]; SELECT 1;" *> $null +} finally { + $ErrorActionPreference = $previousPreference +} +if ($LASTEXITCODE -ne 0) { + throw "SQL Server $SqlServer (db $DbName) is unreachable; refusing to run so an empty perf comparison cannot be reported as a pass." +} +Write-Host "Preflight: SQL Server $SqlServer (db $DbName) is reachable and warmed." + +#################################################################################################### +# 3. Inject the VM's SQL Server connection string into the benchmark runner config. +#################################################################################################### + +$RunnerConfig = Join-Path $RepoRoot "perf-runnerconfig.json" +$env:RUNNER_CONFIG = $RunnerConfig + +# The perf app also loads datatypes.json via the DATATYPES_CONFIG env var, falling back to +# "datatypes.json" in the working directory. Each pass runs from an otherwise-empty +# perf-run-