Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support deploy and manage dm 2.0 #587

Merged
merged 34 commits into from
Jul 29, 2020
Merged

Support deploy and manage dm 2.0 #587

merged 34 commits into from
Jul 29, 2020

Conversation

july2993
Copy link
Contributor

@july2993 july2993 commented Jul 8, 2020

ref #557

close #461

This PR refactor some code and make tiup-dm support all commands like tiup-cluster(except for check and import)

  • code components/cluster/command/* are move into pkg/cluster/deploy/ and change it to deploy a general cluster instead of tidb cluster, both tiup-cluster and tiup-dm use it to avoid many duplicate codes, and maybe later for some tiup-** to use it too.
  • move pkg/dm/ into components/dm
  • many code in pkg/cluster/* are change to support work with a general cluster, I prefer to move code tiup-cluser specified code into component/cluster later.
  • note tiup-dm does not support monitor yet in this pr.

all the existing command are supported:

Deploy a DM cluster for production

Usage:
  tiup-dm [flags]
  tiup-dm [command]

Available Commands:
  deploy      Deploy a DM cluster for production
  start       Start a DM cluster
  stop        Stop a DM cluster
  restart     Restart a DM cluster
  list        List all clusters
  destroy     Destroy a specified DM cluster
  audit       Show audit log of cluster operation
  exec        Run shell command on host in the dm cluster
  edit-config Edit DM cluster config
  display     Display information of a DM cluster
  reload      Reload a DM cluster's config and restart if needed
  upgrade     Upgrade a specified DM cluster
  patch       Replace the remote package with a specified package and restart the service
  scale-out   Scale out a DM cluster
  scale-in    Scale in a DM cluster
  help        Help about any command

Flags:
  -h, --help               help for tiup-dm
      --ssh-timeout int    Timeout in seconds to connect host via SSH, ignored for operations that don't need an SSH connection. (default 5)
  -v, --version            version for tiup-dm
      --wait-timeout int   Timeout in seconds to wait for an operation to complete, ignored for operations that don't fit. (default 60)
  -y, --yes                Skip all confirmations and assumes 'yes'

Use "tiup-dm help [command]" for more information about a command.

a cluster may look like:

root@control:/tiup-cluster# tdm display test
dm Cluster: test
dm Version: nightly
ID                 Role       Host          Ports      OS/Arch       Status     Data Dir                   Deploy Dir
--                 ----       ----          -----      -------       ------     --------                   ----------
172.19.0.101:8261  dm-master  172.19.0.101  8261/8291  linux/x86_64  Healthy    data                       deploy/dm-master-8261
172.19.0.101:8361  dm-master  172.19.0.101  8361/8292  linux/x86_64  Unhealthy  data                       deploy/dm-master-8361
172.19.0.102:8261  dm-master  172.19.0.102  8261/8291  linux/x86_64  Healthy    data                       deploy/dm-master-8261
172.19.0.103:8261  dm-master  172.19.0.103  8261/8291  linux/x86_64  Healthy    /home/tidb/my_master_data  deploy/dm-master-8261
172.19.0.104:8261  dm-master  172.19.0.104  8261/8291  linux/x86_64  Healthy|L  data                       deploy/dm-master-8261
172.19.0.105:8261  dm-master  172.19.0.105  8261/8291  linux/x86_64  Healthy    data                       deploy/dm-master-8261
172.19.0.101:8280  dm-portal  172.19.0.101  8280       linux/x86_64  Up         data                       deploy/dm-portal-8280
172.19.0.102:8280  dm-portal  172.19.0.102  8280       linux/x86_64  Up         data                       deploy/dm-portal-8280
172.19.0.101:8262  dm-worker  172.19.0.101  8262       linux/x86_64  Free       data                       deploy/dm-worker-8262
172.19.0.101:8263  dm-worker  172.19.0.101  8263       linux/x86_64  Free       data                       deploy/dm-worker-8263
172.19.0.102:8262  dm-worker  172.19.0.102  8262       linux/x86_64  Free       data                       deploy/dm-worker-8262
172.19.0.103:8262  dm-worker  172.19.0.103  8262       linux/x86_64  Free       data                       deploy/dm-worker-8262
172.19.0.104:8262  dm-worker  172.19.0.104  8262       linux/x86_64  Free       data                       deploy/dm-worker-8262

tests:

  • manual run every command (except patch) will add integration test in follow up pr.

@codecov-commenter
Copy link

codecov-commenter commented Jul 22, 2020

Codecov Report

Merging #587 into master will decrease coverage by 0.38%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #587      +/-   ##
==========================================
- Coverage   47.79%   47.40%   -0.39%     
==========================================
  Files         224      227       +3     
  Lines       16818    17121     +303     
==========================================
+ Hits         8038     8117      +79     
- Misses       7653     7892     +239     
+ Partials     1127     1112      -15     
Flag Coverage Δ
#coverage 47.40% <ø> (-0.39%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...github.com/pingcap/tiup/pkg/cluster/task/action.go 0.00% <0.00%> (-46.35%) ⬇️
...pingcap/tiup/components/cluster/command/display.go 31.16% <0.00%> (-22.21%) ⬇️
...om/pingcap/tiup/pkg/cluster/operation/operation.go 59.09% <0.00%> (-13.64%) ⬇️
go/src/github.com/pingcap/tiup/pkg/logger/audit.go 52.63% <0.00%> (-12.37%) ⬇️
...b.com/pingcap/tiup/pkg/cluster/operation/action.go 38.25% <0.00%> (-5.45%) ⬇️
...ithub.com/pingcap/tiup/pkg/cluster/task/builder.go 81.78% <0.00%> (-5.37%) ⬇️
...ithub.com/pingcap/tiup/pkg/cluster/spec/profile.go 62.96% <0.00%> (-3.71%) ⬇️
.../pingcap/tiup/components/cluster/command/deploy.go 73.91% <0.00%> (-3.31%) ⬇️
...c/github.com/pingcap/tiup/pkg/cluster/spec/tidb.go 52.12% <0.00%> (-2.93%) ⬇️
...ub.com/pingcap/tiup/pkg/cliutil/prepare/prepare.go 47.61% <0.00%> (-1.84%) ⬇️
... and 42 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 651808e...c19332e. Read the comment docs.

@lonng lonng linked an issue Jul 23, 2020 that may be closed by this pull request
21 tasks
@july2993 july2993 marked this pull request as ready for review July 23, 2020 11:51
@lucklove lucklove requested review from lonng, AstroProfundis and lucklove and removed request for lonng and AstroProfundis July 23, 2020 12:13
@july2993
Copy link
Contributor Author

@lonng @AstroProfundis PTAL

pkg/cluster/manager.go Outdated Show resolved Hide resolved
pkg/cluster/manager.go Outdated Show resolved Hide resolved
pkg/cluster/manager.go Outdated Show resolved Hide resolved
Copy link
Contributor

@lonng lonng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome works.
Rest LGTM

components/dm/command/root.go Show resolved Hide resolved
components/dm/spec/topology_dm.go Show resolved Hide resolved
pkg/cliutil/prepare/prepare.go Show resolved Hide resolved
pkg/cluster/manager.go Show resolved Hide resolved
pkg/cluster/spec/util.go Outdated Show resolved Hide resolved
Copy link
Member

@lucklove lucklove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 29, 2020
@july2993
Copy link
Contributor Author

@lonng @AstroProfundis PTAL

@ti-srebot ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Jul 29, 2020
@ti-srebot ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Jul 29, 2020
@july2993 july2993 merged commit 5f45fb7 into pingcap:master Jul 29, 2020
@july2993 july2993 deleted the dm3 branch July 29, 2020 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status/LGT2 Indicates that a PR has LGTM 2. type/new-feature Categorizes pr as related to a new feature.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

support deploy and manage dm 2.0 Refactor & decouple cluster/dms components
6 participants