Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stats: tables with table-level auto stats are not refreshed on node startup #87247

Closed
rytaft opened this issue Sep 1, 2022 · 0 comments · Fixed by #88673
Closed

stats: tables with table-level auto stats are not refreshed on node startup #87247

rytaft opened this issue Sep 1, 2022 · 0 comments · Fixed by #88673
Assignees
@msirek
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-qa T-sql-queries SQL Queries Team
Projects
SQL Queries

Comments

@rytaft
Copy link
Collaborator

rytaft commented Sep 1, 2022 •
edited by cockroach-jira-scripts

Describe the problem

If the cluster setting sql.stats.automatic_collection.enabled is false, tables that have the table-level setting sql_stats_automatic_collection_enabled set to true should have their stats refreshed when a node restarts (assuming their stats have not been refreshed recently). This is not currently happening.

To Reproduce

On master, start a single node cluster using ./cockroach start-single-node --insecure. In a separate window, connect to the cluster with ./cockroach sql --insecure.

Run the following:

SET CLUSTER SETTING sql.stats.automatic_collection.enabled=false; 
CREATE TABLE tab (x INT) WITH (sql_stats_automatic_collection_enabled = true);
INSERT INTO tab SELECT generate_series(1,1000);
DELETE FROM system.table_statistics WHERE true;

Disconnect from the cluster and restart the node in the other window. Now, reconnect to the cluster with ./cockroach sql --insecure. After a minute, check whether any stats exist for tab by running SHOW STATISTICS FOR TABLE tab;. They will not exist.

Expected behavior

Restarting the node should cause stats to be refreshed for tables that have auto stats enabled and don't already have stats (or have stats that are old). This works when the cluster setting sql.stats.automatic_collection.enabled is true, but does not currently work for the table-level setting.

Jira issue: CRDB-19232
@rytaft rytaft added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-qa labels Sep 1, 2022
@rytaft rytaft added this to Triage in SQL Queries via automation Sep 1, 2022
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Sep 1, 2022
@mgartner mgartner moved this from Triage to 23.1 Release in SQL Queries Sep 20, 2022
@msirek msirek mentioned this issue Sep 25, 2022
Merged
msirek pushed a commit to msirek/cockroach that referenced this issue Sep 25, 2022
stats: fix missing autostats on cluster startup
6fae797 
Fixes cockroachdb#87247

Previously, automatic statistics may fail to be collected on tables
with no stats at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false
but table storage parameter `sql_stats_automatic_collection_enabled`
is true.

Method `ensureAllTables` does not put entries into the settingOverrides
map, and since storage parameter info is not normally looked up in the
table header in auto stats processing, we fall back on the cluster
setting to determine if auto stats should be collected.

To address this, this patch modifies auto stats to flag whether the
current batch of tables saved in the mutationCounts map comes from
cluster startup processing, and if so, ensures that storage
parameters controlling auto stats are always looked up in the table
header during that processing.

Release note (bug fix): This patch fixes missing automatic statistics
collection at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false,
but there are tables with storage parameter
`sql_stats_automatic_collection_enabled` set to true.
msirek pushed a commit to msirek/cockroach that referenced this issue Sep 25, 2022
stats: fix missing autostats on cluster startup
53ec9ca 
Fixes cockroachdb#87247

Previously, automatic statistics may fail to be collected on tables
with no stats at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false
but table storage parameter `sql_stats_automatic_collection_enabled`
is true.

Method `ensureAllTables` does not put entries into the settingOverrides
map, and since storage parameter info is not normally looked up in the
table header in auto stats processing, we fall back on the cluster
setting to determine if auto stats should be collected.

To address this, this patch modifies auto stats to flag whether the
current batch of tables saved in the mutationCounts map comes from
cluster startup processing, and if so, ensures that storage
parameters controlling auto stats are always looked up in the table
header during that processing.

Release note (bug fix): This patch fixes missing automatic statistics
collection at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false,
but there are tables with storage parameter
`sql_stats_automatic_collection_enabled` set to true.
msirek pushed a commit to msirek/cockroach that referenced this issue Sep 26, 2022
stats: fix missing autostats on cluster startup
4911ca5 
Fixes cockroachdb#87247

Previously, automatic statistics may fail to be collected on tables
with no stats at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false
but table storage parameter `sql_stats_automatic_collection_enabled`
is true.

Method `ensureAllTables` does not put entries into the settingOverrides
map, and since storage parameter info is not normally looked up in the
table header in auto stats processing, we fall back on the cluster
setting to determine if auto stats should be collected.

To address this, this patch modifies auto stats to flag whether the
current batch of tables saved in the mutationCounts map comes from
cluster startup processing, and if so, ensures that storage
parameters controlling auto stats are always looked up in the table
header during that processing.

Release note (bug fix): This patch fixes missing automatic statistics
collection at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false,
but there are tables with storage parameter
`sql_stats_automatic_collection_enabled` set to true.
craig bot pushed a commit that referenced this issue Sep 26, 2022
@jbowens @yuzefovich
@amyyq2 @erikgrinaker
Merge #88582 #88615 #88673 #88718 #88724 #88730
089d9c0 
88582: vendor: bump Pebble to 829c25fa5db3 r=nicktrav a=jbowens

```
829c25fa db: fix RangeKeyChanged and -WithLimit interaction
0f464e9c sstable: fix interaction between bpf and monotonic bounds optimization
46a6a539 sstable: remove commented code
9a552270 db: fix TestRangeKeyMaskingRandomized
8418ebc2 build: bump min go version to 1.17
5f8eb821 *: remove references to `ioutil`
05a4b29c db: expand iter_histories test coverage
2855ba7c db: refactor TestRangeKeys into TestIterHistories
```

Release note: None

88615: bench: add a benchmark for update with assigment casts r=yuzefovich a=yuzefovich

Informs: #88525.

Release note: None

88673: stats: fix missing autostats on cluster startup r=msirek a=msirek

Fixes #87247

Previously, automatic statistics may fail to be collected on tables
with no stats at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false
but table storage parameter `sql_stats_automatic_collection_enabled`
is true.

Method `ensureAllTables` does not put entries into the settingOverrides
map, and since storage parameter info is not normally looked up in the
table header in auto stats processing, we fall back on the cluster
setting to determine if auto stats should be collected.

To address this, this patch modifies auto stats to flag whether the
current batch of tables saved in the mutationCounts map comes from
cluster startup processing, and if so, ensures that storage
parameters controlling auto stats are always looked up in the table
header during that processing.

Release note (bug fix): This patch fixes missing automatic statistics
collection at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false,
but there are tables with storage parameter
`sql_stats_automatic_collection_enabled` set to true.


88718: storage: add log scoping to MVCC benchmarks r=erikgrinaker a=jbowens

This prevents logs from being interleaved within benchmark output when using `--stream-output` with `dev bench`. Since the Pebble logs are chatty, this might have a non-neglible impact on the benchmark performance. I'll backport this to prior releases so that comparisons between the HEADs of the various release branches are comparing apples to apples.

Release note: None

88724: ui: fix ui on dbconsole table detail page with very long index name r=amyyq2 a=amyyq2

Previously, when there was a very long index name, it would run off the edge of the summary card and cause the Index Stats table to scroll for a long time. This change wraps the index name text so the user can see the entirety of the name on the summary card and in the table. This change also fixed the width of the Index Stats table to be aligned with the other summary cards on the page.

<img width="1203" alt="cluster-ui" src="https://user-images.githubusercontent.com/54999459/192319099-71c7734d-3ed1-4ee0-b6ee-568bd46f0dba.png">

Partially Fixes #86559

Release justification: Category 2: Bug fixes and
low-risk updates to new functionality

Release note: None

88730: cli: deprecate `debug unsafe-remove-dead-replicas` r=aliher1911 a=erikgrinaker

This patch marks `unsafe-remove-dead-replicas` as deprecated, and refers users to the new `recover` commands. It will be removed in v23.1.

Resolves #86543.

Release note (cli change): The `debug unsafe-remove-dead-replicas` CLI command has been deprecated, and will be removed in v23.1. Users should use the new `debug recover` set of commands instead.

Co-authored-by: Jackson Owens <jackson@cockroachlabs.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Mark Sirek <sirek@cockroachlabs.com>
Co-authored-by: Amy Qian <amy.qian@cockroachlabs.com>
Co-authored-by: Erik Grinaker <grinaker@cockroachlabs.com>
@craig craig bot closed this as completed in b2e0f43 Sep 26, 2022
SQL Queries automation moved this from 23.1 Release to Done Sep 26, 2022
blathers-crl bot pushed a commit that referenced this issue Sep 26, 2022
stats: fix missing autostats on cluster startup
b8bae0d 
Fixes #87247

Previously, automatic statistics may fail to be collected on tables
with no stats at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false
but table storage parameter `sql_stats_automatic_collection_enabled`
is true.

Method `ensureAllTables` does not put entries into the settingOverrides
map, and since storage parameter info is not normally looked up in the
table header in auto stats processing, we fall back on the cluster
setting to determine if auto stats should be collected.

To address this, this patch modifies auto stats to flag whether the
current batch of tables saved in the mutationCounts map comes from
cluster startup processing, and if so, ensures that storage
parameters controlling auto stats are always looked up in the table
header during that processing.

Release note (bug fix): This patch fixes missing automatic statistics
collection at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false,
but there are tables with storage parameter
`sql_stats_automatic_collection_enabled` set to true.
This was referenced Sep 26, 2022
Merged
Merged
blathers-crl bot pushed a commit that referenced this issue Sep 26, 2022
stats: fix missing autostats on cluster startup
50ea1e6 
Fixes #87247

Previously, automatic statistics may fail to be collected on tables
with no stats at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false
but table storage parameter `sql_stats_automatic_collection_enabled`
is true.

Method `ensureAllTables` does not put entries into the settingOverrides
map, and since storage parameter info is not normally looked up in the
table header in auto stats processing, we fall back on the cluster
setting to determine if auto stats should be collected.

To address this, this patch modifies auto stats to flag whether the
current batch of tables saved in the mutationCounts map comes from
cluster startup processing, and if so, ensures that storage
parameters controlling auto stats are always looked up in the table
header during that processing.

Release note (bug fix): This patch fixes missing automatic statistics
collection at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false,
but there are tables with storage parameter
`sql_stats_automatic_collection_enabled` set to true.
msirek pushed a commit that referenced this issue Sep 26, 2022
stats: fix missing autostats on cluster startup
c031226 
Fixes #87247

Previously, automatic statistics may fail to be collected on tables
with no stats at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false
but table storage parameter `sql_stats_automatic_collection_enabled`
is true.

Method `ensureAllTables` does not put entries into the settingOverrides
map, and since storage parameter info is not normally looked up in the
table header in auto stats processing, we fall back on the cluster
setting to determine if auto stats should be collected.

To address this, this patch modifies auto stats to flag whether the
current batch of tables saved in the mutationCounts map comes from
cluster startup processing, and if so, ensures that storage
parameters controlling auto stats are always looked up in the table
header during that processing.

Release note (bug fix): This patch fixes missing automatic statistics
collection at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false,
but there are tables with storage parameter
`sql_stats_automatic_collection_enabled` set to true.
msirek pushed a commit to msirek/cockroach that referenced this issue Oct 4, 2022
stats: fix missing autostats on cluster startup
921d3a4 
Fixes cockroachdb#87247

Previously, automatic statistics may fail to be collected on tables
with no stats at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false
but table storage parameter `sql_stats_automatic_collection_enabled`
is true.

Method `ensureAllTables` does not put entries into the settingOverrides
map, and since storage parameter info is not normally looked up in the
table header in auto stats processing, we fall back on the cluster
setting to determine if auto stats should be collected.

To address this, this patch modifies auto stats to flag whether the
current batch of tables saved in the mutationCounts map comes from
cluster startup processing, and if so, ensures that storage
parameters controlling auto stats are always looked up in the table
header during that processing.

Release note (bug fix): This patch fixes missing automatic statistics
collection at cluster startup when the
`sql.stats.automatic_collection.enabled` cluster setting is false,
but there are tables with storage parameter
`sql_stats_automatic_collection_enabled` set to true.
@msirek msirek mentioned this issue Oct 4, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msirek msirek

Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-qa T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
SQL Queries
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

stats: fix missing autostats on cluster startup msirek/cockroach
2 participants
@rytaft @msirek