PT-1508 Adding --read-only-interval flag, and read-only check on wake-up

[shlomi-noach]

First, please let me know if this should go into 3.0 branch.

This PR adds better handling for read-only. Assuming --check-read-only is set:

• The new optional --read-only-interval flag will make for a different sleep time in between read-only checks (and for as long as the server is read-only).
  If unspecified, resort to --interval.
  Why? We run with --interval=0.1, and wish to run pt-heartbeat individually on all our servers. It's pointless to have replicas being hammered every 0.1s with a select @@read_only query. We wish to have a different interval, like every 1s, as long as the server is read-only.
• When waking up, re-check read-only.
  This is useful in the event a server that was writable, and had heartbeat injected, turns read-only (as in the case of a planned master promotion). In which case, we again wish to introduce the read-only-interval sleep time.

Thank you.

[CLAassistant]

All committers have signed the CLA.

[shlomi-noach]

Oh, and apparently my editor stripped quite a few trailing spaces. Please let me know if you wish to retain those spaces.

[shlomi-noach]

For your convenience, adding ?w=1 to the URL diffs without spaces: https://github.com/percona/percona-toolkit/pull/302/files?w=1

[shlomi-noach]

I did sign the Contributor License Agreement. And @tomkrouper merely issued a merge from master.

[rnovikovP]

@shlomi-noach , unfortunately, cla-assistant.io can't get your sign ( can you re-click on the button above from cla-assistant. Thanks

[shlomi-noach]

@rnovikovP I think I may have used a different email address. I will try again.

[shlomi-noach]

OK, I'm good! As per @tomkrouper 's merge commit, maybe I'll squash it or something.

[tomkrouper]

I’ve signed the cla.

[shlomi-noach]

Also added e1b8e2f: --fail-successive-errors

=item --fail-successive-errors

type: int

If specified, pt-heartbeat will fail after given number of successive DBI errors
(failure to connect to server or issue a query).

This is useful when MySQL goes away. pt-heartbeat seems unable to reconnect to the MySQL server if the MySQL server is restarted, for example. --fail-successive-errors allows us to bail out after n sequential failures. A single success resets the counter.

This commit fixes https://bugs.launchpad.net/percona-toolkit/+bug/1629164
JIRA: https://jira.percona.com/browse/PT-1386

[shlomi-noach]

Friendly ping. Any thoughts on the above?

[percona-csalguero]

Hi,

Thank you very much for the improvements.
I've created: https://jira.percona.com/browse/PT-1508 to track this PR.
This must be merged in 3.0 branch. Could you change the PR please? It shouldn't be any problem. I already merged it in my local and ran the tests and everything is OK.

Regards

[shlomi-noach]

I changed the base to 3.0. There's a conflict with README that I seem to be unable to solve.

FYI whitespace is a big issue with pt code. It is inconsistent.

[shlomi-noach]

Fixed README conflict. Trailing whitespace.

[shlomi-noach]

Woot! Thank you