Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/changefeedccl: TestChangefeedSchemaChangeBackfillCheckpoint failed #108084

Closed
cockroach-teamcity opened this issue Aug 3, 2023 · 2 comments · Fixed by #108143 or #108379
Closed

ccl/changefeedccl: TestChangefeedSchemaChangeBackfillCheckpoint failed #108084

cockroach-teamcity opened this issue Aug 3, 2023 · 2 comments · Fixed by #108143 or #108379
Assignees
@jayshrivastava
Labels
A-cdc Change Data Capture branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-cdc
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 3, 2023 •
edited by exalate-issue-sync bot

ccl/changefeedccl.TestChangefeedSchemaChangeBackfillCheckpoint failed with artifacts on master @ 4fe2a80d81c6fc5a3da3c7c44c5fc38da67e0367:

=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/905465f697dc04c39f7d8a82cb057bfe/logTestChangefeedSchemaChangeBackfillCheckpoint1752657638
    test_log_scope.go:81: use -show-logs to present logs inline
    changefeed_test.go:1870: random seed: -7347133967698241246
=== CONT  TestChangefeedSchemaChangeBackfillCheckpoint
    changefeed_test.go:2115: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/905465f697dc04c39f7d8a82cb057bfe/logTestChangefeedSchemaChangeBackfillCheckpoint1752657638
--- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint (9.54s)
=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint/cloudstorage
    helpers_test.go:862: making server as secondary tenant
    helpers_test.go:943: making cloudstorage feed factory
    changefeed_test.go:2100: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:2100
        	            				github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1134
        	Error:      	Should be false
        	Test:       	TestChangefeedSchemaChangeBackfillCheckpoint/cloudstorage
        	Messages:   	span should not have been resolved: /Tenant/10/Table/106/{2-3}
    --- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint/cloudstorage (9.52s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-30307

Epic CRDB-11732
@cockroach-teamcity cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-cdc labels Aug 3, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Aug 3, 2023
@blathers-crl blathers-crl bot added the A-cdc Change Data Capture label Aug 3, 2023
@jayshrivastava jayshrivastava self-assigned this Aug 3, 2023
@jayshrivastava jayshrivastava mentioned this issue Aug 3, 2023
Merged
craig bot pushed a commit that referenced this issue Aug 3, 2023
@jayshrivastava
Merge #108143
a20cf2d 
108143: changefeedccl: deflake TestChangefeedSchemaChangeBackfillCheckpoint r=miretskiy a=jayshrivastava

When using the declarative schema changer, we should refresh the span to assert at the end because it may change due to a primary index swap during the schema change.

Release note: None
Fixes: #108084
Epic: None

Co-authored-by: Jayant Shrivastava <jayants@cockroachlabs.com>
@craig craig bot closed this as completed in 157cffd Aug 3, 2023
@pav-kv
Copy link
Collaborator

pav-kv commented Aug 8, 2023

I've seen this failure again today in this CI build.

@pav-kv pav-kv reopened this Aug 8, 2023
@jayshrivastava
Copy link
Contributor

Ack. I can take a look.

@jayshrivastava jayshrivastava mentioned this issue Aug 8, 2023
Merged
craig bot pushed a commit that referenced this issue Aug 8, 2023
@Shivs11 @otan
@Xiang-Gu @yuzefovich @raggar @jayshrivastava
Merge #101260 #108160 #108300 #108328 #108341 #108370 #108379
69bc4c6 
101260: sql: replicating JSON empty array ordering found in Postgres r=mgartner a=Shivs11

Currently, #97928 and #99275 are responsible for laying out a
lexicographical ordering for JSON columns to be forward indexable in
nature. This ordering is based on the rules posted by Postgres and is
in #99849.

However, Postgres currently sorts the empty JSON array before any other
JSON values. A Postgres bug report for this has been opened:
https://www.postgresql.org/message-id/17873-826fdc8bbcace4f1%40postgresql.org

This PR intends on replicating the Postgres behavior.

Fixes #105668

Epic: CRDB-24501

Release note: None


108160: roachtest/awsdms: run once a week instead r=Jeremyyang920 a=otan

Save a bit of mad dosh by running awsdms once a weekly instead of daily. We don't need this tested every week.

Epic: None

Release note: None

108300: schemachanger: Unskip some backup tests r=Xiang-Gu a=Xiang-Gu

Randomly skip subtests in the BACKUP/RESTORE suites before parallelizing them.

Epic: None
Release note: None

108328: rowexec: fix TestUncertaintyErrorIsReturned under race r=yuzefovich a=yuzefovich

We just saw a case when `TestUncertaintyErrorIsReturned` failed under race because we got a different DistSQL plan. This seems plausible in case the range cache population (which the test does explicitly) isn't quick enough for some reason, so this commit allows for the DistSQL plan to match the expectation via `SucceedsSoon` (if we happen to get a bad plan, then the following query execution should have the up-to-date range cache).

Fixes: #108250.

Release note: None

108341: importer: fix stale comment on mysqlStrToDatum r=mgartner,DrewKimball a=otan

Release note: None
Epic: None

From #108286 (review)

108370: go.mod: bump Pebble to fffe02a195e3 r=RahulAggarwal1016 a=RahulAggarwal1016

fffe02a1 db: simplify ScanInternal()
df7e2ae1 vfs: deflake TestDiskHealthChecking_Filesystem ff5c929a Rate Limit Scan Statistics
af8c5f27 internal/cache: mark panic messages as redaction-safe

Epic: none
Release note: none

108379: changefeedccl: deflake TestChangefeedSchemaChangeBackfillCheckpoint r=miretskiy a=jayshrivastava

Previously, the test `TestChangefeedSchemaChangeBackfillCheckpoint` would fail because it would read a table span too early. A schema change using the delcarative schema changer will update a table span to point to a new set of ranges. Previously, this test would use the span from before the schema change, which is incorrect. This change makes it use the span from after the schema change.

I stress tested this 30k times under the new schema changer and 10k times under the legacy schema changer to ensure the test is not flaky anymore.

Fixes: #108084
Release note: None
Epic: None

Co-authored-by: Shivam Saraf <shivam.saraf@cockroachlabs.com>
Co-authored-by: Oliver Tan <otan@cockroachlabs.com>
Co-authored-by: Xiang Gu <xiang@cockroachlabs.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Rahul Aggarwal <rahul.aggarwal@cockroachlabs.com>
Co-authored-by: Jayant Shrivastava <jayants@cockroachlabs.com>
@craig craig bot closed this as completed in c44ffa8 Aug 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jayshrivastava jayshrivastava

Labels
A-cdc Change Data Capture branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-cdc
Projects
None yet
Milestone
23.2
3 participants
@pav-kv @jayshrivastava @cockroach-teamcity