Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: create new system observability tables and update job #104714

Merged
merged 1 commit into from
Oct 6, 2023
Merged

sql: create new system observability tables and update job #104714

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions docs/generated/metrics/metrics.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1077,6 +1077,18 @@
<tr><td>APPLICATION</td><td>jobs.migration.resume_completed</td><td>Number of migration jobs which successfully resumed to completion</td><td>jobs</td><td>COUNTER</td><td>COUNT</td><td>AVG</td><td>NON_NEGATIVE_DERIVATIVE</td></tr>
<tr><td>APPLICATION</td><td>jobs.migration.resume_failed</td><td>Number of migration jobs which failed with a non-retriable error</td><td>jobs</td><td>COUNTER</td><td>COUNT</td><td>AVG</td><td>NON_NEGATIVE_DERIVATIVE</td></tr>
<tr><td>APPLICATION</td><td>jobs.migration.resume_retry_error</td><td>Number of migration jobs which failed with a retriable error</td><td>jobs</td><td>COUNTER</td><td>COUNT</td><td>AVG</td><td>NON_NEGATIVE_DERIVATIVE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.currently_idle</td><td>Number of mvcc_statistics_update jobs currently considered Idle and can be freely shut down</td><td>jobs</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.currently_paused</td><td>Number of mvcc_statistics_update jobs currently considered Paused</td><td>jobs</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.currently_running</td><td>Number of mvcc_statistics_update jobs currently running in Resume or OnFailOrCancel state</td><td>jobs</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.expired_pts_records</td><td>Number of expired protected timestamp records owned by mvcc_statistics_update jobs</td><td>records</td><td>COUNTER</td><td>COUNT</td><td>AVG</td><td>NON_NEGATIVE_DERIVATIVE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.fail_or_cancel_completed</td><td>Number of mvcc_statistics_update jobs which successfully completed their failure or cancelation process</td><td>jobs</td><td>COUNTER</td><td>COUNT</td><td>AVG</td><td>NON_NEGATIVE_DERIVATIVE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.fail_or_cancel_failed</td><td>Number of mvcc_statistics_update jobs which failed with a non-retriable error on their failure or cancelation process</td><td>jobs</td><td>COUNTER</td><td>COUNT</td><td>AVG</td><td>NON_NEGATIVE_DERIVATIVE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.fail_or_cancel_retry_error</td><td>Number of mvcc_statistics_update jobs which failed with a retriable error on their failure or cancelation process</td><td>jobs</td><td>COUNTER</td><td>COUNT</td><td>AVG</td><td>NON_NEGATIVE_DERIVATIVE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.protected_age_sec</td><td>The age of the oldest PTS record protected by mvcc_statistics_update jobs</td><td>seconds</td><td>GAUGE</td><td>SECONDS</td><td>AVG</td><td>NONE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.protected_record_count</td><td>Number of protected timestamp records held by mvcc_statistics_update jobs</td><td>records</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.resume_completed</td><td>Number of mvcc_statistics_update jobs which successfully resumed to completion</td><td>jobs</td><td>COUNTER</td><td>COUNT</td><td>AVG</td><td>NON_NEGATIVE_DERIVATIVE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.resume_failed</td><td>Number of mvcc_statistics_update jobs which failed with a non-retriable error</td><td>jobs</td><td>COUNTER</td><td>COUNT</td><td>AVG</td><td>NON_NEGATIVE_DERIVATIVE</td></tr>
<tr><td>APPLICATION</td><td>jobs.mvcc_statistics_update.resume_retry_error</td><td>Number of mvcc_statistics_update jobs which failed with a retriable error</td><td>jobs</td><td>COUNTER</td><td>COUNT</td><td>AVG</td><td>NON_NEGATIVE_DERIVATIVE</td></tr>
<tr><td>APPLICATION</td><td>jobs.new_schema_change.currently_idle</td><td>Number of new_schema_change jobs currently considered Idle and can be freely shut down</td><td>jobs</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
<tr><td>APPLICATION</td><td>jobs.new_schema_change.currently_paused</td><td>Number of new_schema_change jobs currently considered Paused</td><td>jobs</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
<tr><td>APPLICATION</td><td>jobs.new_schema_change.currently_running</td><td>Number of new_schema_change jobs currently running in Resume or OnFailOrCancel state</td><td>jobs</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
Expand Down
2 changes: 1 addition & 1 deletion docs/generated/settings/settings-for-tenants.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,4 +315,4 @@ trace.snapshot.rate duration 0s if non-zero, interval at which background trace
trace.span_registry.enabled boolean true if set, ongoing traces can be seen at https://<ui>/#/debug/tracez application
trace.zipkin.collector string the address of a Zipkin instance to receive traces, as <host>:<port>. If no port is specified, 9411 will be used. application
ui.display_timezone enumeration etc/utc the timezone used to format timestamps in the ui [etc/utc = 0, america/new_york = 1] application
version version 1000023.1-28 set the active cluster version in the format '<major>.<minor>' application
version version 1000023.1-32 set the active cluster version in the format '<major>.<minor>' application
2 changes: 1 addition & 1 deletion docs/generated/settings/settings.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,6 @@
<tr><td><div id="setting-trace-span-registry-enabled" class="anchored"><code>trace.span_registry.enabled</code></div></td><td>boolean</td><td><code>true</code></td><td>if set, ongoing traces can be seen at https://&lt;ui&gt;/#/debug/tracez</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
<tr><td><div id="setting-trace-zipkin-collector" class="anchored"><code>trace.zipkin.collector</code></div></td><td>string</td><td><code></code></td><td>the address of a Zipkin instance to receive traces, as &lt;host&gt;:&lt;port&gt;. If no port is specified, 9411 will be used.</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
<tr><td><div id="setting-ui-display-timezone" class="anchored"><code>ui.display_timezone</code></div></td><td>enumeration</td><td><code>etc/utc</code></td><td>the timezone used to format timestamps in the ui [etc/utc = 0, america/new_york = 1]</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
<tr><td><div id="setting-version" class="anchored"><code>version</code></div></td><td>version</td><td><code>1000023.1-28</code></td><td>set the active cluster version in the format &#39;&lt;major&gt;.&lt;minor&gt;&#39;</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
<tr><td><div id="setting-version" class="anchored"><code>version</code></div></td><td>version</td><td><code>1000023.1-32</code></td><td>set the active cluster version in the format &#39;&lt;major&gt;.&lt;minor&gt;&#39;</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
</tbody>
</table>
9 changes: 9 additions & 0 deletions pkg/ccl/backupccl/system_schema.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -818,6 +818,15 @@ var systemTableBackupConfiguration = map[string]systemBackupConfiguration{
systemschema.RegionLivenessTable.GetName(): {
shouldIncludeInClusterBackup: optOutOfClusterBackup,
},
systemschema.SystemMVCCStatisticsTable.GetName(): {
shouldIncludeInClusterBackup: optOutOfClusterBackup,
},
systemschema.StatementExecInsightsTable.GetName(): {
shouldIncludeInClusterBackup: optOutOfClusterBackup,
},
systemschema.TransactionExecInsightsTable.GetName(): {
shouldIncludeInClusterBackup: optOutOfClusterBackup,
},
}

func rekeySystemTable(
Expand Down
4 changes: 2 additions & 2 deletions pkg/ccl/logictestccl/testdata/logic_test/crdb_internal_tenant
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,12 +351,12 @@ txn_id txn_fingerprint_id query implicit_txn session_id start_time end_tim
query ITTI
SELECT range_id, start_pretty, end_pretty, lease_holder FROM crdb_internal.ranges
----
65 /Tenant/10 /Tenant/11 1
68 /Tenant/10 /Tenant/11 1

query ITT
SELECT range_id, start_pretty, end_pretty FROM crdb_internal.ranges_no_leases
----
65 /Tenant/10 /Tenant/11
68 /Tenant/10 /Tenant/11

query IT
SELECT zone_id, target FROM crdb_internal.zones ORDER BY 1
Expand Down
2 changes: 1 addition & 1 deletion pkg/ccl/logictestccl/testdata/logic_test/multi_region_remote_access_error
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -463,7 +463,7 @@ skipif config multiregion-9node-3region-3azs-vec-off
query I retry
SELECT DISTINCT range_id FROM [SHOW RANGES FROM TABLE messages_rbr]
----
67
70

# Update does not fail when accessing all rows in messages_rbr because lookup
# join does not error out the lookup table in phase 1.
Expand Down
4 changes: 2 additions & 2 deletions pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_query_behavior
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,7 @@ ap-southeast-2 23
query TT
SELECT start_key, end_key FROM [SHOW RANGE FROM TABLE regional_by_row_table FOR ROW ('ap-southeast-2', 1)]
----
<before:/Table/62> …
<before:/Table/65> …

query TIIII
SELECT crdb_region, pk, pk2, a, b FROM regional_by_row_table
Expand Down Expand Up @@ -402,7 +402,7 @@ SELECT start_key, end_key, replicas, lease_holder FROM [SHOW RANGES FROM INDEX r
ORDER BY 1
----
start_key end_key replicas lease_holder
<before:/Table/62> …/"\x80"/0 {1} 1
<before:/Table/65> …/"\x80"/0 {1} 1
…/"\x80"/0 …/"\xc0"/0 {4} 4
…/"\xc0"/0 <after:/Table/110/5> {7} 7

Expand Down
12 changes: 12 additions & 0 deletions pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/basic
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ upsert /Table/{59-60} database system (host)
upsert /Table/6{0-1} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
upsert /Table/6{1-2} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
upsert /Table/6{2-3} database system (host)
upsert /Table/6{3-4} database system (host)
upsert /Table/6{4-5} database system (host)
upsert /Table/6{5-6} database system (host)

exec-sql
CREATE DATABASE db;
Expand Down Expand Up @@ -122,6 +125,9 @@ state offset=47
/Table/6{0-1} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{1-2} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{2-3} database system (host)
/Table/6{3-4} database system (host)
/Table/6{4-5} database system (host)
/Table/6{5-6} database system (host)
/Table/10{6-7} num_replicas=7 num_voters=5
/Table/10{7-8} num_replicas=7
/Table/11{2-3} num_replicas=7
Expand Down Expand Up @@ -233,6 +239,12 @@ delete /Table/{59-60}
upsert /Table/{59-60} ttl_seconds=100 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
delete /Table/6{2-3}
upsert /Table/6{2-3} ttl_seconds=100 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
delete /Table/6{3-4}
upsert /Table/6{3-4} ttl_seconds=100 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
delete /Table/6{4-5}
upsert /Table/6{4-5} ttl_seconds=100 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
delete /Table/6{5-6}
upsert /Table/6{5-6} ttl_seconds=100 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true

state offset=5 limit=42
----
Expand Down
15 changes: 15 additions & 0 deletions pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/indexes
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ state offset=47
/Table/6{0-1} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{1-2} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{2-3} database system (host)
/Table/6{3-4} database system (host)
/Table/6{4-5} database system (host)
/Table/6{5-6} database system (host)
/Table/10{6-7} range default

exec-sql
Expand Down Expand Up @@ -74,6 +77,9 @@ state offset=47
/Table/6{0-1} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{1-2} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{2-3} database system (host)
/Table/6{3-4} database system (host)
/Table/6{4-5} database system (host)
/Table/6{5-6} database system (host)
/Table/106{-/2} num_replicas=7
/Table/106/{2-3} num_replicas=7 num_voters=5
/Table/10{6/3-7} num_replicas=7
Expand Down Expand Up @@ -113,6 +119,9 @@ state offset=47
/Table/6{0-1} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{1-2} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{2-3} database system (host)
/Table/6{3-4} database system (host)
/Table/6{4-5} database system (host)
/Table/6{5-6} database system (host)
/Table/106{-/2} ttl_seconds=3600 num_replicas=7
/Table/106/{2-3} ttl_seconds=25 num_replicas=7 num_voters=5
/Table/10{6/3-7} ttl_seconds=3600 num_replicas=7
Expand Down Expand Up @@ -142,6 +151,9 @@ state offset=47
/Table/6{0-1} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{1-2} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{2-3} database system (host)
/Table/6{3-4} database system (host)
/Table/6{4-5} database system (host)
/Table/6{5-6} database system (host)
/Table/106{-/2} ttl_seconds=3600 num_replicas=9
/Table/106/{2-3} ttl_seconds=25 num_replicas=9 num_voters=5
/Table/10{6/3-7} ttl_seconds=3600 num_replicas=9
Expand Down Expand Up @@ -185,3 +197,6 @@ state offset=46
/Table/6{0-1} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{1-2} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{2-3} database system (host)
/Table/6{3-4} database system (host)
/Table/6{4-5} database system (host)
/Table/6{5-6} database system (host)
18 changes: 18 additions & 0 deletions pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/multitenant/basic
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ state offset=47
/Table/6{0-1} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{1-2} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{2-3} database system (host)
/Table/6{3-4} database system (host)
/Table/6{4-5} database system (host)
/Table/6{5-6} database system (host)
/Tenant/10{-\x00} database system (tenant)
/Tenant/11{-\x00} database system (tenant)
/Tenant/12{-\x00} database system (tenant)
Expand Down Expand Up @@ -95,6 +98,9 @@ upsert /Tenant/10/Table/5{6-7} database system (tenant)
upsert /Tenant/10/Table/5{7-8} database system (tenant)
upsert /Tenant/10/Table/5{8-9} database system (tenant)
upsert /Tenant/10/Table/{59-60} database system (tenant)
upsert /Tenant/10/Table/6{0-1} database system (tenant)
upsert /Tenant/10/Table/6{1-2} database system (tenant)
upsert /Tenant/10/Table/6{2-3} database system (tenant)

state offset=47
----
Expand All @@ -115,6 +121,9 @@ state offset=47
/Table/6{0-1} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{1-2} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{2-3} database system (host)
/Table/6{3-4} database system (host)
/Table/6{4-5} database system (host)
/Table/6{5-6} database system (host)
/Tenant/10{-/Table/4} database system (tenant)
/Tenant/10/Table/{4-5} database system (tenant)
/Tenant/10/Table/{5-6} database system (tenant)
Expand Down Expand Up @@ -161,6 +170,9 @@ state offset=47
/Tenant/10/Table/5{7-8} database system (tenant)
/Tenant/10/Table/5{8-9} database system (tenant)
/Tenant/10/Table/{59-60} database system (tenant)
/Tenant/10/Table/6{0-1} database system (tenant)
/Tenant/10/Table/6{1-2} database system (tenant)
/Tenant/10/Table/6{2-3} database system (tenant)
/Tenant/11{-\x00} database system (tenant)
/Tenant/12{-\x00} database system (tenant)

Expand All @@ -185,6 +197,9 @@ upsert /Tenant/10/Table/11{3-4} range default
state offset=81
----
...
/Tenant/10/Table/2{4-5} database system (tenant)
/Tenant/10/Table/2{5-6} database system (tenant)
/Tenant/10/Table/2{6-7} database system (tenant)
/Tenant/10/Table/2{7-8} database system (tenant)
/Tenant/10/Table/2{8-9} database system (tenant)
/Tenant/10/NamespaceTable/{30-Max} database system (tenant)
Expand Down Expand Up @@ -213,6 +228,9 @@ state offset=81
/Tenant/10/Table/5{7-8} database system (tenant)
/Tenant/10/Table/5{8-9} database system (tenant)
/Tenant/10/Table/{59-60} database system (tenant)
/Tenant/10/Table/6{0-1} database system (tenant)
/Tenant/10/Table/6{1-2} database system (tenant)
/Tenant/10/Table/6{2-3} database system (tenant)
/Tenant/10/Table/10{6-7} range default
/Tenant/10/Table/10{7-8} range default
/Tenant/10/Table/11{2-3} range default
Expand Down
6 changes: 6 additions & 0 deletions pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/multitenant/protectedts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ state offset=47
/Table/6{0-1} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{1-2} ttl_seconds=3600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
/Table/6{2-3} database system (host)
/Table/6{3-4} database system (host)
/Table/6{4-5} database system (host)
/Table/6{5-6} database system (host)
/Tenant/10{-\x00} database system (tenant)
/Tenant/11{-\x00} database system (tenant)

Expand Down Expand Up @@ -111,6 +114,9 @@ upsert /Tenant/10/Table/5{6-7} database system (tenant)
upsert /Tenant/10/Table/5{7-8} database system (tenant)
upsert /Tenant/10/Table/5{8-9} database system (tenant)
upsert /Tenant/10/Table/{59-60} database system (tenant)
upsert /Tenant/10/Table/6{0-1} database system (tenant)
upsert /Tenant/10/Table/6{1-2} database system (tenant)
upsert /Tenant/10/Table/6{2-3} database system (tenant)

exec-sql tenant=10
CREATE DATABASE db;
Expand Down
Loading
Loading