Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/changefeedccl: TestChangefeedSchemaChangeBackfillCheckpoint failed #119375

Closed
cockroach-teamcity opened this issue Feb 20, 2024 · 7 comments · May be fixed by #121934
Closed

ccl/changefeedccl: TestChangefeedSchemaChangeBackfillCheckpoint failed #119375

cockroach-teamcity opened this issue Feb 20, 2024 · 7 comments · May be fixed by #121934
Assignees
@asg0451
Labels
A-cdc Change Data Capture branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-cdc
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Feb 20, 2024 •
edited by jlinder
Loading

ccl/changefeedccl.TestChangefeedSchemaChangeBackfillCheckpoint failed with artifacts on release-23.1 @ b102a6dd3f49ef7bb92217d005ad890105f84d07:

=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/905465f697dc04c39f7d8a82cb057bfe/logTestChangefeedSchemaChangeBackfillCheckpoint3530097046
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestChangefeedSchemaChangeBackfillCheckpoint
    changefeed_test.go:2511: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/905465f697dc04c39f7d8a82cb057bfe/logTestChangefeedSchemaChangeBackfillCheckpoint3530097046
--- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint (51.43s)
=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint/cloudstorage
    helpers_test.go:823: making server as secondary tenant
    helpers_test.go:901: making cloudstorage feed factory
    changefeed_test.go:2402: condition failed to evaluate within 45s: waiting for checkpoint
    --- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint/cloudstorage (51.41s)

Parameters:

  • TAGS=bazel,gss
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-36171
@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc labels Feb 20, 2024
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Feb 20, 2024
@blathers-crl blathers-crl bot added the A-cdc Change Data Capture label Feb 20, 2024
@rharding6373 rharding6373 removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Feb 20, 2024
@rharding6373
Copy link
Collaborator

Removing release blocker label since the error message seems to indicate flaky test infra, but Wenyi agreed to investigate.

@cockroach-teamcity
Copy link
Member Author

ccl/changefeedccl.TestChangefeedSchemaChangeBackfillCheckpoint failed with artifacts on release-23.1 @ 333253387f07fcbfc20697df5556224d44ca86c2:

=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/b239b1bb590a7eed1fa5a8194849ef12/logTestChangefeedSchemaChangeBackfillCheckpoint3595143624
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestChangefeedSchemaChangeBackfillCheckpoint
    changefeed_test.go:2511: -- test log scope end --
--- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint (53.40s)
=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint/pubsub
    helpers_test.go:823: making server as secondary tenant
    helpers_test.go:901: making pubsub feed factory
    changefeed_test.go:2402: condition failed to evaluate within 45s: waiting for checkpoint
    --- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint/pubsub (53.37s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@wenyihu6
Copy link
Contributor

wenyihu6 commented Feb 21, 2024 •
edited
Loading

Adding release blocker back - doesn't seem to be a a test infra flake. Removing release blocker again. It has been a flaky test for a while https://github.com/cockroachdb/cockroach/issues?q=TestChangefeedSchemaChangeBackfillCheckpoint. But I will continue the investigation.

@wenyihu6 wenyihu6 added release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. P-1 Issues/test failures with a fix SLA of 1 month and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Feb 21, 2024
@rharding6373 rharding6373 removed the P-1 Issues/test failures with a fix SLA of 1 month label Feb 26, 2024
@rharding6373 rharding6373 added the P-2 Issues/test failures with a fix SLA of 3 months label Mar 20, 2024
@rharding6373
Copy link
Collaborator

Summary of offline discussion: Plan is to add more logs for future debugability and close this issue.

@andyyang890
Copy link
Collaborator

I suspect this is similar to #120744 and we should consider increasing the batch size here:

cockroach/pkg/ccl/changefeedccl/changefeed_test.go

Line 2198 in 8914fa9

b.Header.MaxSpanRequestKeys = 10

@wenyihu6 wenyihu6 mentioned this issue Apr 9, 2024
Draft
wenyihu6 added a commit to wenyihu6/cockroach that referenced this issue Apr 10, 2024
@wenyihu6
changefeedccl: deflake TestChangefeedSchemaChangeBackfillCheckpoint
360916e 
Previously, TestChangefeedSchemaChangeBackfillCheckpoint could fail when the
batch size is too small, leading to a timeout while waiting for initial scan to
complete. This patch deflakes TestChangefeedSchemaChangeBackfillCheckpoint by
increasing the batch size.

Fixes: cockroachdb#119375
Release note: none
@cockroach-teamcity
Copy link
Member Author

ccl/changefeedccl.TestChangefeedSchemaChangeBackfillCheckpoint failed with artifacts on release-23.1 @ bb0d40a1c8334ab6584189c004f358c63bd95aa7:

=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/905465f697dc04c39f7d8a82cb057bfe/logTestChangefeedSchemaChangeBackfillCheckpoint3391357835
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestChangefeedSchemaChangeBackfillCheckpoint
    changefeed_test.go:2511: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/905465f697dc04c39f7d8a82cb057bfe/logTestChangefeedSchemaChangeBackfillCheckpoint3391357835
--- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint (51.32s)
=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint/enterprise
    helpers_test.go:823: making server as secondary tenant
    helpers_test.go:901: making enterprise feed factory
    helpers_test.go:915: pgURL enterprise root
    changefeed_test.go:2402: condition failed to evaluate within 45s: waiting for checkpoint
    --- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint/enterprise (51.31s)

Parameters:

  • TAGS=bazel,gss
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@exalate-issue-sync exalate-issue-sync bot assigned asg0451 and unassigned wenyihu6 Jun 10, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue Jun 15, 2024
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Jul 12, 2024
Closed
@asg0451 asg0451 mentioned this issue Sep 18, 2024
Merged
@asg0451
Copy link
Contributor

asg0451 commented Sep 19, 2024

should be fixed by #130966

@asg0451 asg0451 closed this as completed Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@asg0451 asg0451

Labels
A-cdc Change Data Capture branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-cdc
Projects
None yet
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

changefeedccl: deflake TestChangefeedSchemaChangeBackfillCheckpoint wenyihu6/cockroach
5 participants
@asg0451 @rharding6373 @cockroach-teamcity @andyyang890 @wenyihu6