Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lightning failed at CreateOrUpdateRegionLabelRule when one PD node is unavailable #46688

Closed
lance6716 opened this issue Sep 5, 2023 · 1 comment · Fixed by #46726
Closed

lightning failed at CreateOrUpdateRegionLabelRule when one PD node is unavailable #46688

lance6716 opened this issue Sep 5, 2023 · 1 comment · Fixed by #46726
Labels
affects-6.5 affects-7.1 component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@lance6716
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

randomly inject some PD error to one node

2. What did you expect to see? (Required)

lightning import success

3. What did you see instead (Required)

Post "http://tc-a-pd:2379/pd/api/v1/config/region-label/rule": dial tcp 10.67.250.186:2379: connect: connection refused
github.com/pingcap/errors.AddStack
	/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174
github.com/pingcap/errors.Trace
	/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15
github.com/pingcap/tidb/br/pkg/pdutil.pdRequestWithCode
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:202
github.com/pingcap/tidb/br/pkg/pdutil.pdRequest
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:153
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).CreateOrUpdateRegionLabelRule
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:948
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).pauseSchedulerByKeyRangeWithTTL
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:1010
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).PauseSchedulersByKeyRange
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:990
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*local).ImportEngine
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1645
github.com/pingcap/tidb/br/pkg/lightning/backend.(*ClosedEngine).Import
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/backend.go:477
github.com/pingcap/tidb/br/pkg/lightning/restore.(*TableRestore).importKV
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/restore/table_restore.go:1039
github.com/pingcap/tidb/br/pkg/lightning/restore.(*TableRestore).importEngine
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/restore/table_restore.go:705
github.com/pingcap/tidb/br/pkg/lightning/restore.(*TableRestore).restoreEngines.func3
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/restore/table_restore.go:333
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1594

4. What is your TiDB version? (Required)
@lance6716 lance6716 added the type/bug The issue is confirmed as a bug. label Sep 5, 2023
@lance6716
Copy link
Contributor Author

similar to #43436

@lance6716 lance6716 mentioned this issue Sep 6, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 7, 2023
@lance6716
lightning: always get latest PD leader when access PD after initializ…
41d1ec0 
…ed (#46726)

close #43436, close #46688
This was referenced Sep 7, 2023
Merged
Merged
@jebter jebter added the component/lightning This issue is related to Lightning of TiDB. label Sep 11, 2023
ti-chi-bot bot pushed a commit that referenced this issue Sep 11, 2023
@ti-chi-bot
lightning: always get latest PD leader when access PD after initializ…
a4aec47 
…ed (#46726) (#46757)

close #43436, close #46688
@Benjamin2037 Benjamin2037 mentioned this issue Sep 15, 2023
Merged
18 tasks
@jayl1e jayl1e mentioned this issue Sep 21, 2023
Closed
11 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 17, 2023
@ti-chi-bot
lightning: always get latest PD leader when access PD after initializ…
49e9daf 
…ed (#46726) (#46758)

close #43436, close #46688
guoshouyan pushed a commit to guoshouyan/tidb that referenced this issue Mar 5, 2024
@ti-chi-bot @mittalrishabh
lightning: always get latest PD leader when access PD after initializ…
360a34b 
…ed (pingcap#46726) (pingcap#46757)

close pingcap#43436, close pingcap#46688
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 affects-7.1 component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

lightning: always get latest PD leader when access PD after initialized lance6716/tidb
2 participants
@lance6716 @jebter