Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/changefeedccl: TestChangefeedLaggingRangesMetrics failed #110155

Closed
cockroach-teamcity opened this issue Sep 7, 2023 · 0 comments · Fixed by #110215
Closed

ccl/changefeedccl: TestChangefeedLaggingRangesMetrics failed #110155

cockroach-teamcity opened this issue Sep 7, 2023 · 0 comments · Fixed by #110215
Assignees
@miretskiy
Labels
A-cdc Change Data Capture branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 7, 2023 •
edited by exalate-issue-sync bot
Loading

ccl/changefeedccl.TestChangefeedLaggingRangesMetrics failed with artifacts on master @ 4e79c65617e3f75dd39da5a9cb5544b445ee1c39:

      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x86

Previous write at 0x00c0055d1ebc by goroutine 428433:
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.makeOptions.func1.1()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:659 +0x37
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.(*rangefeed).addEventsToBuffer()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/physical_kv_feed.go:123 +0x3f0
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.(*rangefeed).addEventsToBuffer-fm()
      <autogenerated>:1 +0x4d
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x4c
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x86

Goroutine 428624 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x12e
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:167 +0x109
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).establishMuxConnection()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:341 +0x24a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*activeMuxRangeFeed).start()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:299 +0x1104
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).startSingleRangeFeed()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:227 +0x3dd
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).startSingleRangeFeed-fm()
      <autogenerated>:1 +0x184
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.divideSpanOnRangeBoundaries()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:449 +0x597
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.divideAllSpansOnRangeBoundaries.func2.1()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:290 +0x219
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x4c
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x86

Goroutine 428433 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x12e
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:167 +0x109
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.rangefeedFactory.Run()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/physical_kv_feed.go:77 +0x335
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.(*kvFeed).runUntilTableEvent.func4()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:581 +0x157
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x4c
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x86
==================

Parameters: TAGS=bazel,gss,race , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-31287

Epic CRDB-11732
@cockroach-teamcity cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc labels Sep 7, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Sep 7, 2023
@blathers-crl blathers-crl bot added the A-cdc Change Data Capture label Sep 7, 2023
@miretskiy miretskiy self-assigned this Sep 7, 2023
@miretskiy miretskiy mentioned this issue Sep 7, 2023
Merged
craig bot pushed a commit that referenced this issue Sep 8, 2023
@msbutler
Merge #110048 #110215
1184f96 
110048: c2c: write all span config updates from intial scan in one transaction r=stevendanna a=msbutler

Previously, the spanConfigIngestor treated updates from the rangefeed initial scan the same way as regular updates: the ingestor would incrementally update the destination side span config table by flushing updates with same source side transaction timestamp. To undestand why this is problematic, recall that an initial scan contains all the latest span config records for the relevant tenant. So, some of these updates may have already been written to the destination side. Further, the initial scan does not replicate source side span config updates that have since been written over by later updates. To summarize, its not possible to replicate a consistent view of the span config table by writing incremental updates surfaced by the initial scan.

In this patch, the ingestor now buffers all updates surfaced by the initial scan and flushes these updates in one transaction which also deletes all existing span config records for the replicating tenant.

Fixes #106823

Release note: None

110215: changefeedccl: Fix test data race r=miretskiy a=miretskiy

Fixes #110155

Release note: None

Co-authored-by: Michael Butler <butler@cockroachlabs.com>
Co-authored-by: Yevgeniy Miretskiy <yevgeniy@cockroachlabs.com>
@craig craig bot closed this as completed in a3f0373 Sep 8, 2023
@blathers-crl blathers-crl bot mentioned this issue Sep 20, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Sep 20, 2023
changefeedccl: Fix test data race
c49a9db 
Fixes #110155

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@miretskiy miretskiy

Labels
A-cdc Change Data Capture branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Projects
None yet
Milestone
23.2
Development

Successfully merging a pull request may close this issue.

changefeedccl: Fix test data race miretskiy/cockroach
2 participants
@cockroach-teamcity @miretskiy