New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Aliyun RDS #541

Merged
merged 8 commits into from Apr 16, 2018

Conversation

Projects
None yet
6 participants
@zhangxiaojian
Copy link
Contributor

zhangxiaojian commented Jan 30, 2018

Related issue: #470

Description

This PR makes gh-ost to support Aliyun RDS.

Aliyun RDS is the largest cloud database in China, and the third of the world. For the reason of security and architecture , we need to hide some mysql variables so gh-ost will get ilegal values. This PR add --aliyun-rds flag to avoid getting them.

Usage

For now, users can't connect to the slave node of rds, so gh-ost may working directly on master with flags:

  • --allow-on-master
  • --assume-rbr
  • --assume-master-host
  • --aliyun-rds

Thanks for gh-ost team developed an excellent tools for online DDL, and hope more people can use it on cloud database.

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.
zj118228
Aliyun RDS hide some mysql variables for secrity so gh-ost will get
ilegal values. Add --aliyun-rds flag to avoid getting them.
@dikang123

This comment has been minimized.

Copy link

dikang123 commented Feb 1, 2018

Great,hope this PR could be merged into the trunk ASAP!:)

@shlomi-noach

This comment has been minimized.

Copy link
Collaborator

shlomi-noach commented Feb 19, 2018

I'm very sorry for the delay. I'll be looking into this in more depth shortly.

@shlomi-noach

This comment has been minimized.

Copy link
Collaborator

shlomi-noach commented Mar 7, 2018

Sorry for the delay, I'll be reviewing this shortly.

@zhangxiaojian

This comment has been minimized.

Copy link
Contributor Author

zhangxiaojian commented Mar 27, 2018

@shlomi-noach Any progress ?

@shlomi-noach
Copy link
Collaborator

shlomi-noach left a comment

Thank you for the PR!

Few notes:

  • unused variable version
  • unused global variable Context

The entire block:

	if impliedKey, err := mysql.GetInstanceKey(this.db, this.migrationContext.AliyunRDS); err != nil {
		return err
	} else {
		if this.migrationContext.AliyunRDS != true {
			this.connectionConfig.ImpliedKey = impliedKey
		}
	}

confuses me greatly. Can we not simplify it into:

	if !this.migrationContext.AliyunRDS {
		if impliedKey, err := mysql.GetInstanceKey(this.db); err != nil {
			return err
		} else {
			this.connectionConfig.ImpliedKey = impliedKey
		}
	}

and so also get rid of the change to GetInstanceKey()?

instanceKey = &InstanceKey{}
err = db.QueryRow(`select @@global.hostname, @@global.port`).Scan(&instanceKey.Hostname, &instanceKey.Port)
if isAliyunRDS != true {

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 27, 2018

Collaborator

Please change to if !isAliyunRDS {

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 27, 2018

Collaborator

Or also see my suggestion to replace the entire block.

return err
} else {
this.connectionConfig.ImpliedKey = impliedKey
if this.migrationContext.AliyunRDS != true {

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 27, 2018

Collaborator

Please change to if !isAliyunRDS {

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 27, 2018

Collaborator

Or also see my suggestion to replace the entire block.

@@ -214,6 +215,8 @@ type ContextConfig struct {
}
}

var Context *MigrationContext

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 27, 2018

Collaborator

I'd like to get rid of this global variable.

This comment has been minimized.

@zhangxiaojian

zhangxiaojian Mar 28, 2018

Author Contributor

The global variable is used in base/utils.go ValidateConnection. I need to get the aliyunRDS flag to avoid get the port from database. If don't use the global variable, add a param ( migrationContext may be ) to the function? or any other suggestions ?

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 28, 2018

Collaborator

Indeed it is used there. Sorry, I've missed this.

I still wish to get rid of the global variable. It will not play well with testing and embedding of gh-ost as a library. Please add a migrationContext *MigrationContext argument to base/utils.go ValidateConnection() function and use it.

This comment has been minimized.

@zhangxiaojian

zhangxiaojian Mar 29, 2018

Author Contributor

Branch is updated, thanks for the review.

var port, extraPort int
var version string
if err := db.QueryRow(query).Scan(&port, &version); err != nil {
if err := db.QueryRow(versionQuery).Scan(&version); err != nil {

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 27, 2018

Collaborator

I don't see where version is used?

This comment has been minimized.

@zhangxiaojian

zhangxiaojian Mar 28, 2018

Author Contributor

The version is used in the function Inspector.validateConnection like before, this patch just prevent to get port from database.

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 28, 2018

Collaborator

Thank you, you are right.

@@ -44,6 +44,8 @@ func acceptSignals(migrationContext *base.MigrationContext) {
// main is the application's entry point. It will either spawn a CLI or HTTP interfaces.
func main() {
migrationContext := base.NewMigrationContext()
base.Context = migrationContext

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 27, 2018

Collaborator

please remove the base.Context global variable.

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 27, 2018

Collaborator

I actually don't see where it is used.

return err
} else {
this.connectionConfig.ImpliedKey = impliedKey
if this.migrationContext.AliyunRDS != true {

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 27, 2018

Collaborator

Please change to if !isAliyunRDS {

This comment has been minimized.

@shlomi-noach

shlomi-noach Mar 27, 2018

Collaborator

Or also see my suggestion to replace the entire block.

This comment has been minimized.

@zhangxiaojian

zhangxiaojian Mar 28, 2018

Author Contributor

Thanks for your suggestion ! replace the entire block makes the code cleaner.

zj118228
@shlomi-noach
Copy link
Collaborator

shlomi-noach left a comment

Looking good. I'll send to testing.

return "", err
}
extraPortQuery := `select @@global.extra_port`
if err := db.QueryRow(extraPortQuery).Scan(&extraPort); err != nil {
// swallow this error. not all servers support extra_port
}
// AliyunRDS set users port to "NULL", replace it by gh-ost param

This comment has been minimized.

@everpcpc

everpcpc Mar 29, 2018

Maybe this can be changed to a more general option like assume port (int) or validate port (bool)? Then it is not necessary passing migrationContext here.
Besides, mysql running in docker with NAT would also report different port. This kind of option would be useful for similar cases.

This comment has been minimized.

@zhangxiaojian

zhangxiaojian Apr 1, 2018

Author Contributor

That's a good point, --aliyun-rds is prevent getting port and hostname now, maybe --assume-check or something, @shlomi-noach we may need a product manager to deicide this. : )

This comment has been minimized.

@shlomi-noach

shlomi-noach Apr 8, 2018

Collaborator

--assume-port sounds good.
Please feel free to choose whether you want to change this in this PR or in a different PR, and let me know.

This comment has been minimized.

@zhangxiaojian

zhangxiaojian Apr 8, 2018

Author Contributor

I would like to keep --aliyun-rds flags because we can do more things in the future. For example, the slave node is invisible for users and gh-ost cann't request binlog from one of them, maybe we can open a path for gh-ost. --assume-port also make sense, but in a different PR I think.

This comment has been minimized.

@shlomi-noach

shlomi-noach Apr 8, 2018

Collaborator

Makes sense to me to keep --aliyun-rds.

shlomi-noach added some commits Apr 12, 2018

@shlomi-noach shlomi-noach merged commit f914b93 into github:master Apr 16, 2018

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

@zhangxiaojian zhangxiaojian deleted the zhangxiaojian:support-aliyun-rds branch Jun 5, 2018

@gaotuan

This comment has been minimized.

Copy link

gaotuan commented Jun 7, 2018

./gh-ost
--aliyun-rds
--ok-to-drop-table
--initially-drop-ghost-table
--initially-drop-socket-file
--host="xx.mysql.singapore.rds.aliyuncs.com"
--port=3306
--user="xx"
--password="xx"
--database="xx"
--table="xxx"
--verbose
--alter="MODIFY COLUMN device_info text,add COLUMN country_code varchar(32) COMMENT '国家编码';"
--panic-flag-file=/tmp/ghost.panic.flag
--allow-on-master
--throttle-flag-file /tmp/1.log
–-assume-rbr=true –-assume-master-host=xx.xx:3306 --execute

2018-06-07 16:17:53 ERROR Error 1885: OPERATION need to be executed set by ADMIN.
2018-06-07 16:17:53 ERROR Error 1885: OPERATION need to be executed set by ADMIN.
2018-06-07 16:17:53 INFO Tearing down inspector
2018-06-07 16:17:53 FATAL Error 1885: OPERATION need to be executed set by ADMIN.

aliyun rds 读写账号 不可以使用gh-ots 吗?

@wps353

This comment has been minimized.

Copy link

wps353 commented Jun 21, 2018

@zhangxiaojian
If aliyun RDS database name including special character,such as '_'.
this tool still not working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment