Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server, version: check and wait if cluster is incompatible #2695

Closed
wants to merge 7 commits into from

Conversation

overvenus
Copy link
Member

What problem does this PR solve?

Check and wait if cluster incompatible before starting CDC.

Check List

Tests

  • Unit test

Related changes

  • Need to cherry-pick to the release branch

Release note

Check and wait if TiDB cluster is incompatible

Signed-off-by: Neil Shen <overvenus@gmail.com>
@overvenus overvenus added type/bugfix This PR fixes a bug. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. labels Aug 31, 2021
@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Aug 31, 2021
@ti-chi-bot ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 31, 2021
Comment on lines 39 to 40
// TODO bump 5.2.0-alpha once PD releases.
minPDVersion *semver.Version = semver.New("5.1.0-alpha")
MinPDVersion *semver.Version = semver.New("5.1.0-alpha")
Copy link
Contributor

@amyangfei amyangfei Aug 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this version be updated to 5.2.0-alpha, ditto for MinTiKVVersion

cdc/server.go Outdated
Comment on lines 370 to 371
err = version.CheckClusterVersion(
ctx, pdClient, pdEndpoint, security, errorTiKVIncompatible)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems the parameter errorTiKVIncompatible can be removed from function CheckClusterVersion

@@ -161,7 +161,7 @@ func (f factoryImpl) PdClient() (pd.Client, error) {

// TODO: we need to check all pd endpoint and make sure they belong to the same cluster.
// See also: https://github.com/pingcap/ticdc/pull/2341#discussion_r673021305.
err = version.CheckClusterVersion(ctx, pdClient, pdEndpoints[0], credential, true)
err = version.CheckClusterVersion(ctx, pdClient, pdEndpoints[0], credential)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to check all PD endpoints to avoid failure when the first PD endpoint is unreachable.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#2713 implements your suggestion.

@overvenus overvenus removed needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. labels Sep 2, 2021
// TODO bump 5.2.0-alpha once PD releases.
minPDVersion *semver.Version = semver.New("5.1.0-alpha")
// MinPDVersion is the version of the minimal compatible PD.
MinPDVersion *semver.Version = semver.New("5.2.0-alpha")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove all the version-related global variables? I think we can check whether the major/minor version matched TiKV/PD/TiCDC.

Comment on lines +168 to 172
// Check cluster version and wait if it's incompatible.
// We start status server first to not block tiup cluster upgrading.
checkAndWaitClusterVersion(ctx, s.pdClient, s.pdEndpoints, conf.Security)

return s.run(ctx)
Copy link
Contributor

@amyangfei amyangfei Sep 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently /status API is available after status server starts, however #2691 is considering to change this behavior, which means the /status API will be available after capture info is persisted to etcd.

Comment on lines +360 to +362
if err == nil {
break
}
Copy link
Contributor

@amyangfei amyangfei Sep 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two scenarios have logic conflicts here.

  • More than 3 PDs in upstream, one of them is down, TiCDC can start up, which is the HA feature, in this scenario TiCDC only checks one PD.
  • Upstream upgrade, should TiCDC wait all PDs upgrade to the required version, then TiCDC should check all PDs.

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 3, 2021
@ti-chi-bot
Copy link
Member

@overvenus: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@asddongmen
Copy link
Contributor

Does this pr need to push?

@overvenus overvenus closed this Jan 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. type/bugfix This PR fixes a bug.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants