diff --git a/src/current/_includes/v25.4/sidebar-data/troubleshooting.json b/src/current/_includes/v25.4/sidebar-data/troubleshooting.json index 886d5297ac6..f8b50de2d2a 100644 --- a/src/current/_includes/v25.4/sidebar-data/troubleshooting.json +++ b/src/current/_includes/v25.4/sidebar-data/troubleshooting.json @@ -112,6 +112,12 @@ "urls": [ "/${VERSION}/automatic-go-execution-tracer.html" ] + }, + { + "title": "Transaction Diagnostics", + "urls": [ + "/${VERSION}/transaction-diagnostics.html" + ] } ] } diff --git a/src/current/_includes/v25.4/ui/transaction-details.md b/src/current/_includes/v25.4/ui/transaction-details.md index 5fe709b04b6..5b8696bf354 100644 --- a/src/current/_includes/v25.4/ui/transaction-details.md +++ b/src/current/_includes/v25.4/ui/transaction-details.md @@ -5,6 +5,7 @@ The details displayed on the **Transaction Details** page reflect the [time inte - The _transaction fingerprint_ is displayed as a list of the individual [SQL statement fingerprints]({{ page_prefix }}statements-page.html#sql-statement-fingerprints) in the transaction. - The **Mean transaction time**: The mean average time it took to execute the transaction within the aggregation interval. - The **Application name**: The name specified by the [`application_name`]({{ link_prefix }}show-vars.html#supported-variables) session setting. +- The **Fingerprint ID**: The hex-encoded ID of the transaction fingerprint. - **Transaction resource usage** shows overall statistics about the transaction. - **Mean rows/bytes read**: The mean average number of rows and bytes [read from the storage layer]({{ link_prefix }}architecture/life-of-a-distributed-transaction.html#reads-from-the-storage-layer) during the execution of the transaction within the specified aggregation interval. - **Bytes read over network**: The amount of [data transferred over the network]({{ link_prefix }}architecture/reads-and-writes-overview.html) for this transaction within the aggregation interval.

If this value is 0, the statement was executed on a single node. diff --git a/src/current/images/v25.4/transaction-diagnostics-1.png b/src/current/images/v25.4/transaction-diagnostics-1.png new file mode 100644 index 00000000000..38a5485b2bd Binary files /dev/null and b/src/current/images/v25.4/transaction-diagnostics-1.png differ diff --git a/src/current/images/v25.4/transaction-diagnostics-2.png b/src/current/images/v25.4/transaction-diagnostics-2.png new file mode 100644 index 00000000000..bc7bb89a7d4 Binary files /dev/null and b/src/current/images/v25.4/transaction-diagnostics-2.png differ diff --git a/src/current/images/v25.4/transaction-diagnostics-3.png b/src/current/images/v25.4/transaction-diagnostics-3.png new file mode 100644 index 00000000000..5373a8d488e Binary files /dev/null and b/src/current/images/v25.4/transaction-diagnostics-3.png differ diff --git a/src/current/images/v25.4/transaction-diagnostics-4.png b/src/current/images/v25.4/transaction-diagnostics-4.png new file mode 100644 index 00000000000..e75858152c6 Binary files /dev/null and b/src/current/images/v25.4/transaction-diagnostics-4.png differ diff --git a/src/current/v25.4/transaction-diagnostics.md b/src/current/v25.4/transaction-diagnostics.md new file mode 100644 index 00000000000..6aa28a13944 --- /dev/null +++ b/src/current/v25.4/transaction-diagnostics.md @@ -0,0 +1,166 @@ +--- +title: Transaction Diagnostics +summary: Use the built-in function `crdb_internal.request_transaction_bundle` to request a transaction diagnostics bundle for a specified transaction fingerprint ID. +toc: true +--- + +Transaction diagnostics allows operators and support engineers to investigate issues involving [transactions]({% link {{ page.version.version }}/transactions.md %}) in user workloads. Use the built-in function `crdb_internal.request_transaction_bundle` to request a transaction diagnostics bundle for a specified [transaction fingerprint ID]({% link {{ page.version.version }}/ui-transactions-page.md %}). + +{{site.data.alerts.callout_info}} +Transaction diagnostics introduces a performance overhead. This feature is primarily intended for use under the guidance of [Cockroach Labs Support](https://support.cockroachlabs.com/). +{{site.data.alerts.end}} + +## Required privileges + +To use this function on a [secure cluster]({% link {{ page.version.version }}/secure-a-cluster.md %}), you must be an [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or a SQL user with the [`VIEWACTIVITY`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewactivity) or [`VIEWACTIVITYREDACTED`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewactivityredacted) [system privilege]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges). If the user has only `VIEWACTIVITYREDACTED`, they can request only redacted bundles. + +## Function signature + +~~~ +crdb_internal.request_transaction_bundle( + transaction_fingerprint_id: string, + sampling_probability: float, + min_execution_latency: interval, + expires_after: interval, + redacted: bool +) -> (request_id: int, created: bool) +~~~ + +## Parameters + +- `transaction_fingerprint_id`: A hex-encoded ID of the transaction fingerprint to capture. The fingerprint ID must exist in `crdb_internal.transaction_statistics`, which is the system of record for transaction fingerprints. +- `sampling_probability`: A probability value (between 0 and 1) for sampling whether a transaction bundle should be recorded. +- `min_execution_latency`: The minimum execution time required for the transaction to be considered. If `sampling_probability` is non-zero, this value must also be non-zero. +- `expires_after`: The duration for which the request remains active. A value of 0 keeps the request open until fulfilled or canceled. +- `redacted`: Specifies whether the resulting bundle should be redacted. + +## Return values + +- `request_id`: The ID of the generated request, or `NULL` if the request could not be created. +- `created`: Returns `true` if the request is successfully created, or `false` if the specified fingerprint ID does not exist. + +## Troubleshooting example + +To troubleshoot with a transaction diagnostics bundle, follow these steps: + +1. [Identify the transaction fingerprint ID](#step-1-identify-the-transaction-fingerprint-id) to generate a bundle for. +1. [Create a bundle request](#step-2-create-a-bundle-request) for that fingerprint ID. +1. [Download the bundle](#step-3-download-the-bundle) from [DB Console]({% link {{ page.version.version }}/ui-overview.md %}). +1. [Inspect the bundle](#step-4-inspect-the-bundle). + +### Setup + +For this example, set the [`application_name`]({% link {{ page.version.version }}/map-sql-activity-to-app.md %}) to enable filtering: + +{% include_cached copy-clipboard.html %} +~~~sql +SET application_name = 'cockroachdb_test'; +~~~ + +Run the following explicit transaction, which sleeps for 10 seconds: + +{% include_cached copy-clipboard.html %} +~~~sql +BEGIN; SELECT pg_sleep(10), 'cockroachdb_test' ; COMMIT; +~~~ + +### Step 1. Identify the transaction fingerprint ID + +Identify the transaction fingerprint ID of the example transaction by navigating to the [**SQL Activity**]({% link {{ page.version.version }}/ui-overview.md %}#sql-activity) page in the [DB Console]({% link {{ page.version.version }}/ui-overview.md %}). On the [**Transactions**]({% link {{ page.version.version }}/ui-transactions-page.md %}) tab, search the Top `500` by `Transaction Time` in the `Past Hour` and click **Apply** to list transactions. Filter the transactions by **Application Name**: `cockroachdb_test`. + +In the **Transactions** column, click the transaction fingerprint `SELECT pg_sleep(_), _` to open the [**Transaction Details**]({% link {{ page.version.version }}/ui-transactions-page.md %}#transaction-details-page) page for that fingerprint. + +Transactions tab + +From the **Transaction Details** page, copy the hexadecimal **Fingerprint ID** for this transaction, `afdd4059a899442e`. + +Transactions Details page + +Note the decimal equivalent of the fingerprint ID in the browser's address bar. In this case, the URL may look like `https://127.0.0.1:29001/#/transaction/12672355680315327534?appNames=cockroachdb_test`. The decimal value of the fingerprint is `12672355680315327534`. + +### Step 2. Create a bundle request + +Create a transaction diagnostics request to be fulfilled the next time the relevant transaction is executed: + +{% include_cached copy-clipboard.html %} +~~~sql +SELECT * FROM crdb_internal.request_transaction_bundle('afdd4059a899442e', 0, '0', '0', false); +~~~ + +where + +- `transaction_fingerprint_id`: `'afdd4059a899442e'`, the hexadecimal fingerprint ID from [Step 1](#step-1-identify-the-transaction-fingerprint-id). +- `sampling_probability`: `0`, which disables sampling. +- `min_execution_latency`: `'0'`, which sets no minimum execution time. +- `expires_after`: `'0'`, which keeps the request open until fulfilled or canceled. +- `redacted`: `false`, which does not redact the bundle. + +Calling the function will return a `request_id` and a `created` boolean flag. This will create an entry in the `system.transaction_diagnostics_request` table. + +~~~sql +> SELECT * FROM crdb_internal.request_transaction_bundle('afdd4059a899442e', 0, '0', '0', false); + request_id | created +----------------------+---------- + 1113728276333035521 | t + +> SELECT * FROM system.transaction_diagnostics_requests; + id | completed | transaction_fingerprint_id | statement_fingerprint_ids | transaction_diagnostics_id | requested_at | min_execution_latency | expires_at | sampling_probability | redacted | username +----------------------+-----------+----------------------------+---------------------------+----------------------------+-------------------------------+-----------------------+------------------------+----------------------+----------+------------ + 1113386248552742913 | t | \xafdd4059a899442e | {"\\x00befd152e98f3f1"} | 1113386693458034689 | 2025-10-07 14:42:18.234632+00 | NULL | NULL | NULL | f | roachprod +~~~ + +In the DB Console, go to [**Advanced Debug**]({% link {{ page.version.version }}/ui-debug-pages.md %}) > [**Diagnostics History**]({% link {{ page.version.version }}/ui-debug-pages.md %}#diagnostics-history). Under the **Transactions** tab, there will be a row for the bundle request. You could also use the URL `https://{host}:{port}/#/reports/diagnosticshistory?tab=Transactions`. The page initially displays the following information: + +- the date and time the request was **Activated on** +- `Transaction 12672355680315327534` (the decimal form of the transaction fingerprint ID from [Step 1](#step-1-identify-the-transaction-fingerprint-id)) +- a **Status** of `WAITING` +- a button to **Cancel request** (Use this if a transaction diagnostics bundle is no longer needed.) + +Diagnostics History, Transactions, Status Waiting + +### Step 3. Download the bundle + +To fulfill the request, run the explicit transaction again. Note that the constant values in the transaction do not have to exactly match the original run. In the second execution of the transaction, the number of seconds differs from the original `10` and the string differs from the original `'cockroachdb_test'`. + +{% include_cached copy-clipboard.html %} +~~~sql +BEGIN; SELECT pg_sleep(12), 'cockroachdb_test_2' ; COMMIT; +~~~ + +Navigate to the [**Advanced Debug**]({% link {{ page.version.version }}/ui-debug-pages.md %}) > [**Diagnostics History**]({% link {{ page.version.version }}/ui-debug-pages.md %}#diagnostics-history) page in the DB Console. Under the **Transactions** tab, the row for the bundle request now shows: + +- the date and time the request was **Activated on** +- the statements for the transaction fingerprint +- a **Status** of `READY` +- a **Bundle.zip** link + +Diagnostics History, Transactions, Status Ready + +Click the **Bundle.zip** link to download the completed bundle, which will be named `txn-bundle-1113386693458034689.zip` using the `transaction_diagnostics_id` from the `system.transaction_diagnostics_requests` table. + +### Step 4. Inspect the bundle + +Unzip the transaction diagnostic bundle and examine its contents. It contains [statement diagnostic bundle]({% link {{ page.version.version }}/ui-statements-page.md %}#diagnostics) `zip` files for each statement as well as a `zip` file of the transaction traces: + +``` +1-SELECT.zip +2-COMMIT.zip +transaction-1113386693458034689.zip +``` + +Unzip the `zip` file `transaction-1113386693458034689.zip`. It consists of transaction traces in various formats (including a JSON file that can be [imported to Jaeger]({% link {{ page.version.version }}/query-behavior-troubleshooting.md %}#visualize-statement-traces-in-jaeger)): + +``` +trace-jaeger.json +trace.json +trace.txt +``` + +## See also + +- [Transactions]({% link {{ page.version.version }}/transactions.md %}) +- [DB Console Transactions page]({% link {{ page.version.version }}/ui-transactions-page.md %}) +- [DB Console Statements page, Diagnostics]({% link {{ page.version.version }}/ui-statements-page.md %}#diagnostics) +- [`cockroach statement-diag`]({% link {{ page.version.version }}/cockroach-statement-diag.md %}) +- [Identify slow queries]({% link {{ page.version.version }}/query-behavior-troubleshooting.md %}#identify-slow-queries) +- [Visualize statement traces in Jaeger]({% link {{ page.version.version }}/query-behavior-troubleshooting.md %}#visualize-statement-traces-in-jaeger) diff --git a/src/current/v25.4/troubleshooting-overview.md b/src/current/v25.4/troubleshooting-overview.md index e4152c431ad..e4b911aadbd 100644 --- a/src/current/v25.4/troubleshooting-overview.md +++ b/src/current/v25.4/troubleshooting-overview.md @@ -33,4 +33,5 @@ If you experience an issue when using CockroachDB, try these steps to resolve th - [`cockroach debug zip`]({% link {{ page.version.version }}/cockroach-debug-zip.md %}) - [`cockroach debug tsdump`]({% link {{ page.version.version }}/cockroach-debug-tsdump.md %}) - [Automatic CPU Profiler]({% link {{ page.version.version }}/automatic-cpu-profiler.md %}) - - [Automatic Go Execution Tracer]({% link {{ page.version.version }}/automatic-go-execution-tracer.md %}) \ No newline at end of file + - [Automatic Go Execution Tracer]({% link {{ page.version.version }}/automatic-go-execution-tracer.md %}) + - [Transaction Diagnostics]({% link {{ page.version.version }}/transaction-diagnostics.md %}) \ No newline at end of file diff --git a/src/current/v25.4/ui-debug-pages.md b/src/current/v25.4/ui-debug-pages.md index 0758d18d462..3f5c86e2523 100644 --- a/src/current/v25.4/ui-debug-pages.md +++ b/src/current/v25.4/ui-debug-pages.md @@ -33,7 +33,7 @@ Report | Description | Access level [Custom Time Series Chart]({% link {{ page.version.version }}/ui-custom-chart-debug-page.md %}) | Create a custom chart of time series data. | All users. Problem Ranges | View ranges in your cluster that are unavailable, under-replicated, slow, paused, or have other problems. | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or a SQL user with the [`VIEWCLUSTERMETADATA`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewclustermetadata) [system privilege]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges). Data Distribution and Zone Configs | View the distribution of table data across nodes and verify [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}). | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or a SQL user with the [`VIEWCLUSTERMETADATA`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewclustermetadata) [system privilege]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges). -Statement Diagnostics History | Diagnostic bundles for all statements executed on the cluster. | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) only. +Diagnostics History | Diagnostic bundles for statements and transactions executed on the cluster. | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or a SQL user with the [`VIEWACTIVITY`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewactivity) or [`VIEWACTIVITYREDACTED`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewactivityredacted) [system privilege]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges). ## Configuration