Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql/stats: TestDefaultColumns failed #81513

Closed
cockroach-teamcity opened this issue May 19, 2022 · 7 comments · Fixed by #82519
Closed

sql/stats: TestDefaultColumns failed #81513

cockroach-teamcity opened this issue May 19, 2022 · 7 comments · Fixed by #82519
Assignees
Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team
Projects

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 19, 2022

sql/stats.TestDefaultColumns failed with artifacts on master @ ce66acdbba801de88f1dd645eaedeb3834f23dbd:

=== RUN   TestDefaultColumns
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/68790eb7cfaf4e8cc805e2d551cc66f6/logTestDefaultColumns4214364350
    test_log_scope.go:80: use -show-logs to present logs inline
    automatic_stats_test.go:696: query 'SELECT count(*) FROM [SHOW STATISTICS FOR TABLE t.a] WHERE statistics_name = 's'': expected:
        101
        
        got:
        41
        
    automatic_stats_test.go:701: -- test log scope end --
--- FAIL: TestDefaultColumns (66.31s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss,deadlock

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-15109

@cockroach-teamcity cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels May 19, 2022
@cockroach-teamcity cockroach-teamcity added this to Triage in SQL Queries May 19, 2022
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label May 19, 2022
@jlinder jlinder added sync-me and removed sync-me labels May 20, 2022
@cockroach-teamcity
Copy link
Member Author

sql/stats.TestDefaultColumns failed with artifacts on master @ 7dd1c51f6b5802e32bafd82e46747f349836592f:

=== RUN   TestDefaultColumns
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/68790eb7cfaf4e8cc805e2d551cc66f6/logTestDefaultColumns1681085004
    test_log_scope.go:80: use -show-logs to present logs inline
    automatic_stats_test.go:696: query 'SELECT count(*) FROM [SHOW STATISTICS FOR TABLE t.a] WHERE statistics_name = 's'': expected:
        101
        
        got:
        44
        
    automatic_stats_test.go:701: -- test log scope end --
--- FAIL: TestDefaultColumns (62.75s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss,deadlock

This test on roachdash | Improve this report!

@rharding6373
Copy link
Collaborator

It looks like TestDefaultColumns is flaking, but I haven't been able to reproduce it with the random seeds above or through stress testing 150 runs.

@cockroach-teamcity
Copy link
Member Author

sql/stats.TestDefaultColumns failed with artifacts on master @ 48e48db89eb3cd6bc38f3631364c516181811280:

=== RUN   TestDefaultColumns
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/68790eb7cfaf4e8cc805e2d551cc66f6/logTestDefaultColumns415704010
    test_log_scope.go:80: use -show-logs to present logs inline
    automatic_stats_test.go:696: query 'SELECT count(*) FROM [SHOW STATISTICS FOR TABLE t.a] WHERE statistics_name = 's'': expected:
        101
        
        got:
        42
        
    automatic_stats_test.go:701: -- test log scope end --
--- FAIL: TestDefaultColumns (65.58s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss,deadlock

This test on roachdash | Improve this report!

@cucaroach
Copy link
Contributor

I wonder if that check should be wrapped in a retry thingee.

@yuzefovich yuzefovich moved this from Triage to 22.2 High Likelihood (90%) in SQL Queries May 24, 2022
@yuzefovich yuzefovich moved this from 22.2 High Likelihood (90%) to 22.2 May Milestone in SQL Queries May 24, 2022
@rytaft
Copy link
Collaborator

rytaft commented May 25, 2022

I can't reproduce this, but it looks really similar to #44208, which makes me think that somehow auto stats is getting enabled on this table. I'm going to use the same approach as that issue, and add some debug code, similar to #41037.

rytaft added a commit to rytaft/cockroach that referenced this issue May 25, 2022
This commit adds some logging to the TestDefaultColumns test in order
to help debug a test failure.

Informs cockroachdb#81513

Release note: None
craig bot pushed a commit that referenced this issue May 25, 2022
81737: cli/demo: fix enterprise features for multitenancy r=rafiss a=knz

First commit from #81762.

Fixes #80270.
Fixes #40434.

This patch achieves two things:

- it ensures that enterprise features work properly in multi-tenant `cockroach
  demo` when `--disable-demo-license` is not passed as argument.
- it ensures `cockroach demo` does not require a working licensing
  endpoint to run (by not retrieving a license over the network on startup).

The net result will be less spurious failures in CI, as well as
avoiding startup delays/errors during interactive uses, which provide
poor UX.

Release note: None

81797: sql: assert that table writers use root txns r=yuzefovich a=yuzefovich

We assume that all mutation statements use the Root txn, but this was
never enforced in practice. This commit adds an explicit check when
initializing the table writer that the root txn is provided.

Release note: None

81800: stats: add logging to TestDefaultColumns to help debug flake r=rytaft a=rytaft

This commit adds some logging to the TestDefaultColumns test in order
to help debug a test failure.

Informs #81513

Release note: None

Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Rebecca Taft <becca@cockroachlabs.com>
@cockroach-teamcity
Copy link
Member Author

sql/stats.TestDefaultColumns failed with artifacts on master @ 1e2cc61b58dc14386bb68dca59814874648931c2:

        __auto__, {c18}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996169461761
        __auto__, {c19}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996214714369
        __auto__, {c20}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996262653953
        __auto__, {c21}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996306628609
        __auto__, {c22}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996383567873
        __auto__, {c23}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996456640513
        __auto__, {c24}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996505104385
        __auto__, {c25}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996549177345
        __auto__, {c26}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996593446913
        __auto__, {c27}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996636438529
        __auto__, {c28}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996679757825
        __auto__, {c29}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996724977665
        __auto__, {c30}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996770426881
        __auto__, {c31}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996816007169
        __auto__, {c32}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996855230465
        __auto__, {c33}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996899958785
        __auto__, {c34}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996945833985
        __auto__, {c35}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645996989579265
        __auto__, {c36}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997050822657
        __auto__, {c37}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997095747585
        __auto__, {c38}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997140606977
        __auto__, {c39}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997193265153
        __auto__, {c40}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997236387841
        __auto__, {c41}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997282066433
        __auto__, {c42}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997327384577
        __auto__, {c43}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997373784065
        __auto__, {c44}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997422575617
        __auto__, {c45}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997466255361
        __auto__, {c46}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997510885377
        __auto__, {c47}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997556727809
        __auto__, {c48}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997606240257
        __auto__, {c49}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997656047617
        __auto__, {c50}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997705560065
        __auto__, {c51}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997753565185
        __auto__, {c52}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997807206401
        __auto__, {c53}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997855735809
        __auto__, {c54}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997905248257
        __auto__, {c55}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997951877121
        __auto__, {c56}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645997998931969
        __auto__, {c57}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645998047068161
        __auto__, {c58}, 2022-05-28 08:36:53.404574 +0000 +0000, 0, 0, 0, 0, 765645998088945665
        
    automatic_stats_test.go:707: query 'SELECT count(*) FROM [SHOW STATISTICS FOR TABLE t.a] WHERE statistics_name = 's'': expected:
        101
        
        got:
        42
        
    automatic_stats_test.go:712: -- test log scope end --
--- FAIL: TestDefaultColumns (65.35s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss,deadlock

This test on roachdash | Improve this report!

@rytaft
Copy link
Collaborator

rytaft commented Jun 7, 2022

Yes, auto stats are definitely enabled here. I'm not sure what exactly is causing the race condition, but I'll update the test to use the new table-level auto stats settings.

craig bot pushed a commit that referenced this issue Jun 7, 2022
81782: sql: use declarative schemachange for add column sequence exprs r=fqazi a=fqazi

Fixes: #81781

Previously, the declarative schema changer was disabled for
add column expressions with sequence references (i.e. default,
on update, computed) because we were missing telemetry
from the legacy schema changer for during backfill related failures.
This was inadequate because the lack of telemetry from these
areas could be seen as a usability issue. To address this,
this patch adds in support for missing telemetry and enables
supports for add column operations with sequence operations.

Release note: None

82388: sql: remove date/intervalstyle_enabled from code r=otan,mgartner a=rafiss

fixes #81529
 
No release note is needed, since v22.1 already included a release note
about how these are both hardcoded to true and cannot be changed.

The session variables and cluster settings remain in the code, but are
marked as retired.

Release note: None

82451: kvserver/rangefeed: fix off-by-one in `NewCatchUpIterator()` r=miretskiy a=erikgrinaker

**kvserver/rangefeed: fix off-by-one in NewCatchUpIterator()**

`NewCatchUpIterator` created the `MVCCIncrementalIterator` with
`RangeFeedRequest.Header.Timestamp.Prev()`, because it assumed the given
timestamp was inclusive. However, the rest of the rangefeed code treats
this timestamp as exclusive.

This patch uses the correct, exclusive timestamp when creating the
`MVCCIncrementalIterator`. This change has no externally visible effect:
even though the previous code would cause `MVCCIncrementalIterator` to emit
keys at `Timestamp`, the `CatchUpScan()` method would discard those
events if they were at or below `Timestamp`.

An integration test is also added to verify that the start timestamp of
a rangefeed catchup scan is exclusive. The test passes both with the new
and the old code, as expected.

Release note: None

**rangefeed: emphasize that start time is exclusive**

This patch adds comments for rangefeed-related APIs and components
emphasizing that the start timestamp of rangefeeds is exclusive. In
other words, the first possible emitted event (including catchup scans)
will be at `RangeFeedRequest.Header.Timestamp.Next()`. Several
parameters are also renamed to emphasize this.

There are no functional or semantic changes.

Touches #82488.

Release note: None

82466: dev: have `doctor` advise to set a particular `tmpdir` r=rail a=rickystewart

Bazel's default behavior of rooting the `tmpdir` to an "in-sandbox"
directory has been a point of confusion for CRL developers. The sandbox
directory does not exist after the test is run (unless `--sandbox_debug`
is provided), which is sometimes confusing for folks who expect their
test's temp files to be present where the logs suggest they should be
(see #82413). Furthermore, the long `tmpdir` used in these cases breaks
tests that create Unix sockets on OS's where Unix sockets have a maximum
path length.

Avoid these problems by having `doctor` just tell you to manually set a
`test_tmpdir`. We add `/tmp` to `gitignore` in case people want to root
it at the `tmp` directory in their checkout.

Closes #72918.
Closes #82413.

Release note: None

82476: update cluster-ui to v22.2.0-prerelease-1 r=maryliag a=maryliag

Update cluster-ui to latest published version

Release note: None

82519: stats: fix flaky test TestDefaultColumns r=rytaft a=rytaft

TestDefaultColumns creates statistics on a table with 110 columns
using the command `CREATE STATISTICS s FROM t.a`. It then checks
that there are exactly 101 column statistics on table t.a with
statistics_name = 's' (one stat for the primary index, plus 100
other column stats). However, this test may be flaky if automatic
statistics are running, since each new automatic stat will cause
other stats to be deleted.

Although the test disables automatic stats at the beginning, it seems
that some sort of race condition may cause it to be reenabled. This
commit fixes the problem by disabling automatic stats using the new table
level settings, ensuring that the 101 column stats are not deleted after
they have been created.

Fixes #81513

Release note: None

Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com>
Co-authored-by: Rafi Shamim <rafi@cockroachlabs.com>
Co-authored-by: Erik Grinaker <grinaker@cockroachlabs.com>
Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com>
Co-authored-by: Marylia Gutierrez <marylia@cockroachlabs.com>
Co-authored-by: Rebecca Taft <becca@cockroachlabs.com>
@craig craig bot closed this as completed in a8451fc Jun 7, 2022
SQL Queries automation moved this from Active to Done Jun 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

5 participants