gh-ost: Failed to register slave; too long 'report-password'

[jbuck]

Hi there, I am trying out gh-ost on a clone of our production database in AWS RDS MySQL 5.6. Going from the documentation I am running the following command:

gh-ost \
--max-load=Threads_running=25 \
--assume-rbr \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--user="snipped" \
--password="snipped" \
--host=snipped \
--allow-on-master \
--database="fxa" \
--table="devices" \
--verbose \
--alter="ALTER TABLE devices ADD COLUMN nameUtf8 VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin" \
--allow-master-master \
--cut-over=default \
--exact-rowcount \
--concurrent-rowcount \
--default-retries=120 \
--panic-flag-file=/tmp/ghost.panic.flag \
--postpone-cut-over-flag-file=/tmp/ghost.postpone.fla

and I get the following logs:

2017-08-31 17:48:01 INFO starting gh-ost 1.0.40
2017-08-31 17:48:01 INFO Migrating `fxa`.`devices`
2017-08-31 17:48:01 INFO connection validated on snipped:3306
2017-08-31 17:48:01 INFO User has REPLICATION CLIENT, REPLICATION SLAVE privileges, and has ALL privileges on `fxa`.*
2017-08-31 17:48:01 INFO binary logs validated on snipped:3306
2017-08-31 17:48:01 INFO Inspector initiated on ip-10-3-2-198:3306, version 5.6.34-log
2017-08-31 17:48:01 INFO Table found. Engine=InnoDB
2017-08-31 17:48:01 INFO Estimated number of rows via EXPLAIN: 44337157
2017-08-31 17:48:01 INFO Recursively searching for replication master
2017-08-31 17:48:01 INFO Master found to be ip-10-3-2-198:3306
2017-08-31 17:48:01 INFO log_slave_updates validated on snipped:3306
2017-08-31 17:48:01 INFO connection validated on snipped:3306
2017/08/31 17:48:01 binlogsyncer.go:75: [info] create BinlogSyncer with config &{99999 mysql snipped 3306 snipped snipped  false false <nil>}
2017-08-31 17:48:01 INFO Connecting binlog streamer at mysql-bin-changelog.364229:120
2017/08/31 17:48:01 binlogsyncer.go:241: [info] begin to sync binlog from position (mysql-bin-changelog.364229, 120)
2017/08/31 17:48:01 binlogsyncer.go:134: [info] register slave for master server snipped:3306
2017-08-31 17:48:01 FATAL ERROR 1105 (HY000): Failed to register slave; too long 'report-password'

There's not a whole lot on google about this MySQL error, so if anyone has seen/fixed this error I'd love some help figuring it out :)

[cjoudrey]

@jbuck I ran into this issue when using the siddontang/go-mysql library earlier today. This is the same library gh-ost appears to be using.

After searching for a while, I could not find any information about Failed to register slave; too long 'report-password' besides this issue. 😆

tl;dr I managed to get rid of the issue by reducing the length of the password of the user. The maximum length appears to be 32.

---

At first I thought there was a bug in siddontang/go-mysql but then I tracked the error in the MySQL source code and found this:

The error occurs in register_slave:

https://github.com/mysql/mysql-server/blob/71f48ab393bce80a59e5a2e498cd1f46f6b43f9a/sql/rpl_master.cc#L127-L152

Here's what the SLAVE_INFO struct looks like:

https://github.com/mysql/mysql-server/blob/71f48ab393bce80a59e5a2e498cd1f46f6b43f9a/sql/rpl_master.h#L36-L45

The interesting part and the cause of the error message:

char password[MAX_PASSWORD_LENGTH+1];

and

#define MAX_PASSWORD_LENGTH	32

https://github.com/mysql/mysql-server/blob/71f48ab393bce80a59e5a2e498cd1f46f6b43f9a/sql/sql_const.h#L112

I also stumbled upon this in the MySQL documentation. I'm not sure if it's relevant but seems like it:

If you are using MySQL Replication, be aware that, currently, a password used by a replication slave as part of a CHANGE MASTER TO statement is effectively limited to 32 characters in length; if the password is longer, any excess characters are truncated. This is not due to any limit imposed by the MySQL Server generally, but rather is an issue specific to MySQL Replication. (For more information, see Bug #43439.)

[cjoudrey]

@shlomi-noach re: above, if this is indeed the problem, do you think it would be a good idea to offer a friendlier error message ingh-ost when FATAL ERROR 1105 (HY000): Failed to register slave; too long 'report-password' occurs?

[shlomi-noach]

@jbuck thank you for the analysis!

[jbuck]

@cjoudrey @shlomi-noach thank you for digging into this and fixing it!

[shlomi-noach]

LOL I switched the names, @cjoudrey thank you for the analysis and thank you @jbuck for the bug report!

[jbuck]

Thanks for adding a more clear error message! I reduced the password to 32 characters and the migration is chugging along nicely now. 👍