From 3395fb169caf6d190e068c04ce191c5fe10b6a2f Mon Sep 17 00:00:00 2001 From: Amruta Ranade Date: Fri, 16 Oct 2020 18:25:47 -0400 Subject: [PATCH 1/8] Documented redact-logs flag --- v20.2/cockroach-debug-merge-logs.md | 1 + v20.2/cockroach-debug-zip.md | 1 + 2 files changed, 2 insertions(+) diff --git a/v20.2/cockroach-debug-merge-logs.md b/v20.2/cockroach-debug-merge-logs.md index c5d302ad4b4..61474753176 100644 --- a/v20.2/cockroach-debug-merge-logs.md +++ b/v20.2/cockroach-debug-merge-logs.md @@ -31,6 +31,7 @@ Flag | Description `--filter` | Limit the results to the specified regular expression `--from` | Start time for the time range filter. `--to` | End time for the time range filter. +`--redact-logs` | Redact sensitive PII data from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed ablve) collected by the `debug zip` command may still contain sensitive information. ## Example diff --git a/v20.2/cockroach-debug-zip.md b/v20.2/cockroach-debug-zip.md index 589d602456b..6a3c88af6f3 100644 --- a/v20.2/cockroach-debug-zip.md +++ b/v20.2/cockroach-debug-zip.md @@ -81,6 +81,7 @@ Flag | Description `--port`
`-p` | The server port to connect to.

**Env Variable:** `COCKROACH_PORT`
**Default:** `26257` `--nodes` | New in v20.2: Specify nodes to inspect as a comma-separated list or range of node IDs. For example:

`--nodes=1,10,13-15` `--exclude-nodes` | New in v20.2: Specify nodes to exclude from inspection as a comma-separated list or range of node IDs. For example:

`--nodes=1,10,13-15` +`--redact-logs` | Redact sensitive PII data from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed ablve) collected by the `debug zip` command may still contain sensitive information. ### Client connection From 86a75055ebdaf5a9c21c6fb7992cddb3cc503d07 Mon Sep 17 00:00:00 2001 From: Amruta Ranade Date: Mon, 19 Oct 2020 14:39:35 -0400 Subject: [PATCH 2/8] Added example to redact logs --- v20.2/cockroach-debug-zip.md | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/v20.2/cockroach-debug-zip.md b/v20.2/cockroach-debug-zip.md index 6a3c88af6f3..8dc07a6895e 100644 --- a/v20.2/cockroach-debug-zip.md +++ b/v20.2/cockroach-debug-zip.md @@ -99,20 +99,41 @@ If you need to troubleshoot this command's behavior, you can also change its [lo ### Generate a debug zip file +Generate the debug zip file for an insecure cluster: + {% include copy-clipboard.html %} ~~~ shell -# Generate the debug zip file for an insecure cluster: $ cockroach debug zip ./cockroach-data/logs/debug.zip --insecure --host=200.100.50.25 ~~~ +Generate the debug zip file for a secure cluster: + {% include copy-clipboard.html %} ~~~ shell -# Generate the debug zip file for a secure cluster: $ cockroach debug zip ./cockroach-data/logs/debug.zip --host=200.100.50.25 ~~~ {{site.data.alerts.callout_info}}Secure examples assume you have the appropriate certificates in the default certificate directory, ${HOME}/.cockroach-certs/.{{site.data.alerts.end}} +### Redact sensitive information from the logs + +Example of a log string without redaction enabled: + +~~~ +server/server.go:1423 ⋮ password of user ‹admin› was set to ‹"s3cr34?!@x_"› +~~~ + +Enable log redaction: + +{% include copy-clipboard.html %} +~~~ shell +$ cockroach debug zip ./cockroach-data/logs/debug.zip -- redact-logs --insecure --host=200.100.50.25 +~~~ + +~~~ +server/server.go:1423 ⋮ password of user ‹×› was set to ‹×› +~~~ + ## See also - [File an Issue](file-an-issue.html) From 91e7eb0923c503d7e619f1e8681c28a7a7e3418c Mon Sep 17 00:00:00 2001 From: Amruta Ranade Date: Mon, 26 Oct 2020 18:19:05 -0400 Subject: [PATCH 3/8] Worked on Raphael's comments --- v20.2/cockroach-debug-merge-logs.md | 2 +- v20.2/cockroach-debug-zip.md | 4 ++-- v20.2/debug-and-error-logs.md | 17 +++++++++++++++++ 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/v20.2/cockroach-debug-merge-logs.md b/v20.2/cockroach-debug-merge-logs.md index 61474753176..3697abf8485 100644 --- a/v20.2/cockroach-debug-merge-logs.md +++ b/v20.2/cockroach-debug-merge-logs.md @@ -9,7 +9,7 @@ key: debug-merge-logs.html The `cockroach debug merge-logs` [command](cockroach-commands.html) merges log files from multiple nodes into a single time-ordered stream of messages with an added per-message prefix to indicate the corresponding node. You can use it in conjunction with logs collected using the [`debug zip`](https://www.cockroachlabs.com/docs/stable/cockroach-debug-zip.html) command to aid in debugging. {{site.data.alerts.callout_danger}} -The file produced by `cockroach debug merge-log` can contain highly sensitive, unanonymized information, such as usernames, passwords, and possibly your table's data. You should share this data only with Cockroach Labs developers and only after determining the most secure method of delivery. +The file produced by `cockroach debug zip` can contain highly [sensitive, unanonymized information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact-logs`](#redact-sensitive-information-from-the-logs) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. {{site.data.alerts.end}} ## Subcommands diff --git a/v20.2/cockroach-debug-zip.md b/v20.2/cockroach-debug-zip.md index 8dc07a6895e..a974ef7d25b 100644 --- a/v20.2/cockroach-debug-zip.md +++ b/v20.2/cockroach-debug-zip.md @@ -30,7 +30,7 @@ The `cockroach debug zip` [command](cockroach-commands.html) connects to your cl Additionally, you can run the [`debug merge-logs`](cockroach-debug-merge-logs.html) command to merge the collected logs in one file, making it easier to parse them to locate an issue with your cluster. {{site.data.alerts.callout_danger}} -The file produced by `cockroach debug zip` can contain highly sensitive, unanonymized information, such as usernames, hashed passwords, and possibly your table's data. You should share this data only with Cockroach Labs developers and only after determining the most secure method of delivery. +The file produced by `cockroach debug zip` can contain highly [sensitive, unanonymized information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact-logs`](#redact-sensitive-information-from-the-logs) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. {{site.data.alerts.end}} ## Details @@ -81,7 +81,7 @@ Flag | Description `--port`
`-p` | The server port to connect to.

**Env Variable:** `COCKROACH_PORT`
**Default:** `26257` `--nodes` | New in v20.2: Specify nodes to inspect as a comma-separated list or range of node IDs. For example:

`--nodes=1,10,13-15` `--exclude-nodes` | New in v20.2: Specify nodes to exclude from inspection as a comma-separated list or range of node IDs. For example:

`--nodes=1,10,13-15` -`--redact-logs` | Redact sensitive PII data from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed ablve) collected by the `debug zip` command may still contain sensitive information. +`--redact-logs` | Redact [sensitive data](debug-and-error-logs.html#redacted-logs) from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed ablve) collected by the `debug zip` command may still contain sensitive information. ### Client connection diff --git a/v20.2/debug-and-error-logs.md b/v20.2/debug-and-error-logs.md index 2eaa35543c5..b6bffdfb21d 100644 --- a/v20.2/debug-and-error-logs.md +++ b/v20.2/debug-and-error-logs.md @@ -112,6 +112,23 @@ These logging flags are used with [`cockroach` commands](cockroach-commands.html {% include {{ page.version.version }}/misc/logging-flags.md %} +## Redacted logs + +If you contact CockroachDB Support for troubleshooting help, you might be asked to run [`cockroach debug zip`](cockroach-debug-zip.html) and share the resulting file with the CockroachDB team. The log files created by `cockroach debug zip` may contain highly sensitive, unanonymized information, such as usernames, hashed passwords, and possibly your table's data. + +New in v20.2 You can run `cockroach debug zip` with the [`redact-logs` flag](cockroach-debug-zip.html#redact-sensitive-information-from-the-logs) to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. Redactable sensitive data includes but is not limited to: + +- Stored values +- Text of SQL statements, especially the values embedded therein +- Result rows +- The dynamic part of error messages that includes application-provided parameters. +- IP addresses or hostnames +- Database, schema, table, or column names +- Cluster IDs +- File names of stored data or log files +- User/role names +- Hashed passwords + ## See also - [SQL logging](query-behavior-troubleshooting.html#sql-logging) From a7c38ed19a5183af859110815185b2e264f99475 Mon Sep 17 00:00:00 2001 From: Amruta Ranade Date: Mon, 26 Oct 2020 18:37:41 -0400 Subject: [PATCH 4/8] Fixed nits --- v20.2/cockroach-debug-merge-logs.md | 11 +++++++++-- v20.2/cockroach-debug-zip.md | 2 +- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/v20.2/cockroach-debug-merge-logs.md b/v20.2/cockroach-debug-merge-logs.md index 3697abf8485..82d290cded5 100644 --- a/v20.2/cockroach-debug-merge-logs.md +++ b/v20.2/cockroach-debug-merge-logs.md @@ -9,7 +9,7 @@ key: debug-merge-logs.html The `cockroach debug merge-logs` [command](cockroach-commands.html) merges log files from multiple nodes into a single time-ordered stream of messages with an added per-message prefix to indicate the corresponding node. You can use it in conjunction with logs collected using the [`debug zip`](https://www.cockroachlabs.com/docs/stable/cockroach-debug-zip.html) command to aid in debugging. {{site.data.alerts.callout_danger}} -The file produced by `cockroach debug zip` can contain highly [sensitive, unanonymized information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact-logs`](#redact-sensitive-information-from-the-logs) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. +The file produced by `cockroach debug zip` can contain highly [sensitive, unanonymized information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact-logs`](#example) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. {{site.data.alerts.end}} ## Subcommands @@ -31,7 +31,7 @@ Flag | Description `--filter` | Limit the results to the specified regular expression `--from` | Start time for the time range filter. `--to` | End time for the time range filter. -`--redact-logs` | Redact sensitive PII data from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed ablve) collected by the `debug zip` command may still contain sensitive information. +`--redact-logs` | Redact [sensitive data](debug-and-error-logs.html#redacted-logs) from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed above) collected by the `debug zip` command may still contain sensitive information. ## Example @@ -70,6 +70,13 @@ You can also filter the merged logs for a regular expression: cockroach debug merge-logs debug/nodes/*/logs/* --filter="RUNNING IN INSECURE MODE" ~~~ +You can redact sensitive information from the merged logs: + +{% include copy-clipboard.html %} +~~~ shell +cockroach debug merge-logs debug/nodes/*/logs/* --redact-logs +~~~ + ## See also - [File an Issue](file-an-issue.html) diff --git a/v20.2/cockroach-debug-zip.md b/v20.2/cockroach-debug-zip.md index a974ef7d25b..7acf211a020 100644 --- a/v20.2/cockroach-debug-zip.md +++ b/v20.2/cockroach-debug-zip.md @@ -81,7 +81,7 @@ Flag | Description `--port`
`-p` | The server port to connect to.

**Env Variable:** `COCKROACH_PORT`
**Default:** `26257` `--nodes` | New in v20.2: Specify nodes to inspect as a comma-separated list or range of node IDs. For example:

`--nodes=1,10,13-15` `--exclude-nodes` | New in v20.2: Specify nodes to exclude from inspection as a comma-separated list or range of node IDs. For example:

`--nodes=1,10,13-15` -`--redact-logs` | Redact [sensitive data](debug-and-error-logs.html#redacted-logs) from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed ablve) collected by the `debug zip` command may still contain sensitive information. +`--redact-logs` | Redact [sensitive data](debug-and-error-logs.html#redacted-logs) from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed above) collected by the `debug zip` command may still contain sensitive information. ### Client connection From 00d0c046439654d298fddcc6892cc758b364f26f Mon Sep 17 00:00:00 2001 From: Amruta Ranade Date: Tue, 27 Oct 2020 09:14:34 -0400 Subject: [PATCH 5/8] Fixed a bug --- v20.2/cockroach-debug-merge-logs.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/v20.2/cockroach-debug-merge-logs.md b/v20.2/cockroach-debug-merge-logs.md index 82d290cded5..d42e2a2d005 100644 --- a/v20.2/cockroach-debug-merge-logs.md +++ b/v20.2/cockroach-debug-merge-logs.md @@ -9,7 +9,7 @@ key: debug-merge-logs.html The `cockroach debug merge-logs` [command](cockroach-commands.html) merges log files from multiple nodes into a single time-ordered stream of messages with an added per-message prefix to indicate the corresponding node. You can use it in conjunction with logs collected using the [`debug zip`](https://www.cockroachlabs.com/docs/stable/cockroach-debug-zip.html) command to aid in debugging. {{site.data.alerts.callout_danger}} -The file produced by `cockroach debug zip` can contain highly [sensitive, unanonymized information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact-logs`](#example) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. +The file produced by `cockroach debug zip` can contain highly [sensitive, unanonymized information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact`](#example) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. {{site.data.alerts.end}} ## Subcommands @@ -31,7 +31,7 @@ Flag | Description `--filter` | Limit the results to the specified regular expression `--from` | Start time for the time range filter. `--to` | End time for the time range filter. -`--redact-logs` | Redact [sensitive data](debug-and-error-logs.html#redacted-logs) from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed above) collected by the `debug zip` command may still contain sensitive information. +`--redact` | Redact [sensitive data](debug-and-error-logs.html#redacted-logs) from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed above) collected by the `debug zip` command may still contain sensitive information. ## Example @@ -74,7 +74,7 @@ You can redact sensitive information from the merged logs: {% include copy-clipboard.html %} ~~~ shell -cockroach debug merge-logs debug/nodes/*/logs/* --redact-logs +cockroach debug merge-logs --redact debug/nodes/*/logs/* ~~~ ## See also From 584601878c70bef5fe22805dd370f6026fa3955a Mon Sep 17 00:00:00 2001 From: Amruta Ranade Date: Tue, 27 Oct 2020 09:36:04 -0400 Subject: [PATCH 6/8] Worked on Raphael's comments --- v20.2/cockroach-debug-merge-logs.md | 6 +++--- v20.2/cockroach-debug-zip.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/v20.2/cockroach-debug-merge-logs.md b/v20.2/cockroach-debug-merge-logs.md index d42e2a2d005..a48e0d9ea27 100644 --- a/v20.2/cockroach-debug-merge-logs.md +++ b/v20.2/cockroach-debug-merge-logs.md @@ -9,7 +9,7 @@ key: debug-merge-logs.html The `cockroach debug merge-logs` [command](cockroach-commands.html) merges log files from multiple nodes into a single time-ordered stream of messages with an added per-message prefix to indicate the corresponding node. You can use it in conjunction with logs collected using the [`debug zip`](https://www.cockroachlabs.com/docs/stable/cockroach-debug-zip.html) command to aid in debugging. {{site.data.alerts.callout_danger}} -The file produced by `cockroach debug zip` can contain highly [sensitive, unanonymized information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact`](#example) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. +The file produced by `cockroach debug zip` can contain highly [sensitive, identifiable information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact`](#example) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. {{site.data.alerts.end}} ## Subcommands @@ -31,7 +31,7 @@ Flag | Description `--filter` | Limit the results to the specified regular expression `--from` | Start time for the time range filter. `--to` | End time for the time range filter. -`--redact` | Redact [sensitive data](debug-and-error-logs.html#redacted-logs) from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed above) collected by the `debug zip` command may still contain sensitive information. +`--redact` | Redact [sensitive data](debug-and-error-logs.html#redacted-logs) from the log files. ## Example @@ -74,7 +74,7 @@ You can redact sensitive information from the merged logs: {% include copy-clipboard.html %} ~~~ shell -cockroach debug merge-logs --redact debug/nodes/*/logs/* +cockroach debug merge-logs --redact debug/nodes/*/logs/* ~~~ ## See also diff --git a/v20.2/cockroach-debug-zip.md b/v20.2/cockroach-debug-zip.md index 7acf211a020..0075e12f5f2 100644 --- a/v20.2/cockroach-debug-zip.md +++ b/v20.2/cockroach-debug-zip.md @@ -30,7 +30,7 @@ The `cockroach debug zip` [command](cockroach-commands.html) connects to your cl Additionally, you can run the [`debug merge-logs`](cockroach-debug-merge-logs.html) command to merge the collected logs in one file, making it easier to parse them to locate an issue with your cluster. {{site.data.alerts.callout_danger}} -The file produced by `cockroach debug zip` can contain highly [sensitive, unanonymized information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact-logs`](#redact-sensitive-information-from-the-logs) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. +The file produced by `cockroach debug zip` can contain highly [sensitive, identifiable information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact-logs`](#redact-sensitive-information-from-the-logs) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. {{site.data.alerts.end}} ## Details From 5798a60eaff5e857c1676dbda8c7be228139dc1a Mon Sep 17 00:00:00 2001 From: Amruta Ranade Date: Mon, 2 Nov 2020 11:18:57 -0500 Subject: [PATCH 7/8] Working on Rich's comments --- v20.2/cockroach-debug-merge-logs.md | 2 +- v20.2/cockroach-debug-zip.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/v20.2/cockroach-debug-merge-logs.md b/v20.2/cockroach-debug-merge-logs.md index a48e0d9ea27..cd0aea3027e 100644 --- a/v20.2/cockroach-debug-merge-logs.md +++ b/v20.2/cockroach-debug-merge-logs.md @@ -9,7 +9,7 @@ key: debug-merge-logs.html The `cockroach debug merge-logs` [command](cockroach-commands.html) merges log files from multiple nodes into a single time-ordered stream of messages with an added per-message prefix to indicate the corresponding node. You can use it in conjunction with logs collected using the [`debug zip`](https://www.cockroachlabs.com/docs/stable/cockroach-debug-zip.html) command to aid in debugging. {{site.data.alerts.callout_danger}} -The file produced by `cockroach debug zip` can contain highly [sensitive, identifiable information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact`](#example) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. +The file produced by `cockroach debug zip` can contain highly [sensitive, identifiable information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`--redact`](#example) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. {{site.data.alerts.end}} ## Subcommands diff --git a/v20.2/cockroach-debug-zip.md b/v20.2/cockroach-debug-zip.md index 0075e12f5f2..cef7ab52728 100644 --- a/v20.2/cockroach-debug-zip.md +++ b/v20.2/cockroach-debug-zip.md @@ -30,7 +30,7 @@ The `cockroach debug zip` [command](cockroach-commands.html) connects to your cl Additionally, you can run the [`debug merge-logs`](cockroach-debug-merge-logs.html) command to merge the collected logs in one file, making it easier to parse them to locate an issue with your cluster. {{site.data.alerts.callout_danger}} -The file produced by `cockroach debug zip` can contain highly [sensitive, identifiable information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`redact-logs`](#redact-sensitive-information-from-the-logs) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. +The file produced by `cockroach debug zip` can contain highly [sensitive, identifiable information](debug-and-error-logs.html#redacted-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`--redact-logs`](#redact-sensitive-information-from-the-logs) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. {{site.data.alerts.end}} ## Details From 9100781b55e229631c4e2bc46b0dbdbc2255e418 Mon Sep 17 00:00:00 2001 From: Amruta Ranade Date: Mon, 2 Nov 2020 11:54:50 -0500 Subject: [PATCH 8/8] Fixed nit --- v20.2/debug-and-error-logs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/v20.2/debug-and-error-logs.md b/v20.2/debug-and-error-logs.md index b6bffdfb21d..147574d7f8f 100644 --- a/v20.2/debug-and-error-logs.md +++ b/v20.2/debug-and-error-logs.md @@ -114,7 +114,7 @@ These logging flags are used with [`cockroach` commands](cockroach-commands.html ## Redacted logs -If you contact CockroachDB Support for troubleshooting help, you might be asked to run [`cockroach debug zip`](cockroach-debug-zip.html) and share the resulting file with the CockroachDB team. The log files created by `cockroach debug zip` may contain highly sensitive, unanonymized information, such as usernames, hashed passwords, and possibly your table's data. +If you contact CockroachDB Support for troubleshooting help, you might be asked to run [`cockroach debug zip`](cockroach-debug-zip.html) and share the resulting file with the CockroachDB team. The log files created by `cockroach debug zip` may contain highly sensitive, identifiable information, such as usernames, hashed passwords, and possibly your table's data. New in v20.2 You can run `cockroach debug zip` with the [`redact-logs` flag](cockroach-debug-zip.html#redact-sensitive-information-from-the-logs) to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. Redactable sensitive data includes but is not limited to: