Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logictest: failure in distsql_stats for virtual columns #121424

Closed
cockroach-teamcity opened this issue Mar 31, 2024 · 2 comments · Fixed by #125150
Closed

logictest: failure in distsql_stats for virtual columns #121424

cockroach-teamcity opened this issue Mar 31, 2024 · 2 comments · Fixed by #125150
Assignees
Labels
branch-master Failures on the master branch. branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-queries SQL Queries Team
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 31, 2024

pkg/sql/logictest/tests/5node/5node_test.TestLogic_distsql_stats failed on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

    logic.go:3021: let $only_null_stat = [{"avg_size": 0, "columns": ["a"], "created_at": "2024-03-31 06:16:43.778278", "distinct_count": 1, "histo_col_type": "INT8", "histo_version": 3, "name": "only_null_stat", "null_count": 3, "row_count": 3}]
    logic.go:3021: let $hist_id_1 = 956155180531318789
    logic.go:3021: let $hist_d = 956155181367066629
    logic.go:3021: let $hist_e = 956155181506822149
    logic.go:3021: let $hist_f = 956155181652082693
[06:16:44] --- progress: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/5node/5node_test_/5node_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/distsql_stats: 340 statements
    logic.go:3021: let $hist_crdb_internal_idx_expr = 956155182815477765
    logic.go:3021: let $hist_d = 956155182964211717
    logic.go:3021: let $hist_e = 956155183102590981
    logic.go:3021: let $hist_crdb_internal_idx_expr = 956155183254274053
[06:17:01] --- progress: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/5node/5node_test_/5node_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/distsql_stats: 357 statements
    logic.go:2975: 
         
        /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/5node/5node_test_/5node_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/distsql_stats:2808: EXPLAIN SELECT * FROM mno WHERE n = 1 AND o = 9
        expected:
            distribution: full
            vectorized: true
            ·
            • render
            │
            └── • filter
                │ estimated row count: 1
                │ filter: n = 1
                │
                └── • scan
                      estimated row count: 18 (1.8% of the table; stats collected <hidden> ago)
                      table: mno@mno_o_idx
                      spans: [/9 - /9]
            
        but found (query options: "") :
            distribution: full
            vectorized: true
            ·
            • render
            │
            └── • filter
                │ filter: sqrt(m::FLOAT8)::INT8 = 9
                │
                └── • scan
                      missing stats
                      table: mno@mno_n_idx
                      spans: [/1 - /1]
            
[06:17:01] --- done: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/5node/5node_test_/5node_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/distsql_stats with config 5node: 366 tests, 1 failures
    logic.go:4145: 
        /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/5node/5node_test_/5node_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/distsql_stats:2825: error while processing
    logic.go:4145: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/5node/5node_test_/5node_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/distsql_stats:2825: too many errors encountered, skipping the rest of the input
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestLogic_distsql_stats2861301879
--- FAIL: TestLogic_distsql_stats (30.14s)

Parameters:

  • attempt=1
  • run=7
  • shard=11
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-37243

@cockroach-teamcity cockroach-teamcity added branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team labels Mar 31, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Mar 31, 2024
@yuzefovich yuzefovich added GA-blocker and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 1, 2024
@yuzefovich yuzefovich changed the title pkg/sql/logictest/tests/5node/5node_test: TestLogic_distsql_stats failed logictest: failure in distsql_stats for virtual columns Apr 1, 2024
michae2 added a commit to michae2/cockroach that referenced this issue Apr 2, 2024
This failure happens when we plan the query without stats. Add a SHOW
HISTOGRAM to make sure we've got the right stats ready to go.

Informs: cockroachdb#121424

Release note: None
@yuzefovich
Copy link
Member

I'm going to drop GA-blocker label since this is a test flake.

@yuzefovich
Copy link
Member

@michae2 this has been failing somewhat frequently, so it'd be nice to fix this sooner than later.

@michae2 michae2 added the branch-master Failures on the master branch. label Jun 5, 2024
craig bot pushed a commit that referenced this issue Jun 5, 2024
125150: sql: deflake distsql_stats logic test r=yuzefovich a=michae2

I stressed this for a while and was not able to reproduce it. However, I have a theory: the failing statement is racing the rangefeed that invalidates the stats cache, and occasionally starts planning before the rangefeed causes the stale entry to be evicted from the cache.

(SHOW STATISTICS does not go through the stats cache, which is why we can hit this even after that statement succeeds.)

A retry on the first non-SHOW-STATISTICS statement after ANALYZE should work. (This is also what we did in #81560.)

Fixes: #121424

Release note: None

125156: acceptance: preserve logs for TestDockerCLI tests in remote execution r=rickystewart a=rafiss

The cleanup logic in (*cluster.DockerCluster).Cleanup takes care to preserve the logs directory if requested. Under remote execution, we need to make sure to use a temporary directory that will be captured by the remote executor.

fixes #125096
Release note: None

Co-authored-by: Michael Erickson <michae2@cockroachlabs.com>
Co-authored-by: Rafi Shamim <rafi@cockroachlabs.com>
@craig craig bot closed this as completed in 3486bea Jun 5, 2024
blathers-crl bot pushed a commit that referenced this issue Jun 5, 2024
I stressed this for a while and was not able to reproduce it. However, I
have a theory: the failing statement is racing the rangefeed that
invalidates the stats cache, and occasionally starts planning before the
rangefeed causes the stale entry to be evicted from the cache.

(SHOW STATISTICS does not go through the stats cache, which is why we
can hit this even after that statement succeeds.)

A retry on the first non-SHOW-STATISTICS statement after ANALYZE should
work. (This is also what we did in #81560.)

Fixes: #121424

Release note: None
Dhruv-Sachdev1313 pushed a commit to Dhruv-Sachdev1313/cockroach that referenced this issue Jun 7, 2024
I stressed this for a while and was not able to reproduce it. However, I
have a theory: the failing statement is racing the rangefeed that
invalidates the stats cache, and occasionally starts planning before the
rangefeed causes the stale entry to be evicted from the cache.

(SHOW STATISTICS does not go through the stats cache, which is why we
can hit this even after that statement succeeds.)

A retry on the first non-SHOW-STATISTICS statement after ANALYZE should
work. (This is also what we did in cockroachdb#81560.)

Fixes: cockroachdb#121424

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures on the master branch. branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-queries SQL Queries Team
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants