flaky onlineddl tests: reduce -online_ddl_check_interval

[shlomi-noach]

Description

Solves #7834

Not sure how this happened, will later look in git blame and history, but in the tests suite, vtctld should run with a very short -online_ddl_check_interval. For some reason the flag didn't appear in any of the online ddl tests. It must have changes somehow over time, looking into. anyway, this should resolve the current CI failures seen mostly in onlineddl_ghost but also in other onlineddl_* tests.

Related Issue(s)

• Flaky Test: ghost online ddl #7834

Checklist

• Should this PR be backported?
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

Impacted Areas in Vitess

Components that this PR will affect:

• Query Serving
• VReplication
• Cluster Management
• Build/CI
• VTAdmin

[shlomi-noach]

Hmmm actually the problem is more specific to onlineddl_ghost and the fact vttablet (not vtctld) was missing some important flags:

https://github.com/vitessio/vitess/pull/7847/files#diff-2bb5e78bd4cd6cdb8f66836e671f20da41eb0fd6664f550e395cc71657e86b76R155-R159

[shlomi-noach]

Successful run: https://github.com/vitessio/vitess/pull/7847/checks?check_run_id=2331731945

[rohit-nayak-ps]

lgtm

[shlomi-noach]

ugh, now failing: https://github.com/vitessio/vitess/pull/7847/checks?check_run_id=2331837611 . I'll keep looking into.

[shlomi-noach]

The nature of this error is unclear: Error querying information_schema.tables for schema vt_ks: dial tcp 10.1.0.5:16009: i/o timeout, and I feel like it's similar to other flaky errors we're seeing with network issues.

[shlomi-noach]

onlineddl_ghost continues to frequently fail (seldomly passes). I can't figure it out.

Now, it fails on a nonsensical issue: the very first gh-ost test fails not because gh-ost has errors. It fails because the test, 20sec after submittion, is still in ready state (you'd expect it to be complete, or failed or even running by that time!).

This does not reproduce locally, ever. All onlineddl_* tests consistently pass locally.

Also, once in a while, another onlineddl_* test joins the party and fails, but with a different type of error:

Error querying information_schema.tables for schema vt_ks: dial tcp 10.1.0.5:16009: i/o timeout

This feels like resource exhaustion. I have no other explanation to this behavior.

[rohit-nayak-ps]

rebased and removed the os tune up commit since we have a different fix for CI now

[rohit-nayak-ps]

@shlomi-noach: tests are passing. I see you moved it to Draft: is it ready to be merged?

[shlomi-noach]

I moved it to Draft because tests went crazy the other day and I wasn't sure anymore whether this PR helps at all. I'm gonna merge master and if tests pass again, I'll merge the PR.

[shlomi-noach]

Some update on where onlineddl_ghost test fails: it gets as far as running gh-ost in dry-run mode and then goes blank.

In gh-ost migration there's three distinct gh-ost executions:

• gh-ost --version, to vlaidate the binary itself is good. This passes.
• gh-ost --alter=... in dry-run mode. This never completes.
• gh-ost --alter=... --execute, This is never reached because the previous run fails.

I invested time to verify/deby that the problem is with gh-ost hooks. Those call tablet server via HTTP API. So I thought this correlates to problems we saw with local_example. It isn't the issue. I completely disabled hooks and the behavior is unchanged.
So we know gh-ost begins to run, but never completes. Why? What makes it block? No information yet, but that's where I'll look into next.

Reiterating that all these tests pass just fine on multiple environemtns, just no in GitHub CI. BTW, the test passes in #7850. So I'm still looking for a platform problem. Sockets/files/other resources.

[shlomi-noach]

catting the gh-ost logs show that the problem is, after all, the hooks. Specifically, gh-ost runs hooks that report back to vttablet via curl, ie accessing vttablet's HTTP API:

2021-04-19 06:21:05 INFO starting gh-ost 33516f4955c3df5c06e975854a0c3e00f6b52112
2021-04-19 06:21:05 INFO Migrating `vt_ks`.`vt_onlineddl_test_03`
2021-04-19 06:21:05 INFO executing gh-ost-on-startup hook: /tmp/online-ddl-69529787_a0d7_11eb_b97f_000d3a02b579-192278704/gh-ost-on-startup

The gh-ost-on-startup hook runs this (template):

#!/bin/bash

curl --max-time 3 -s 'http://localhost:%d/schema-migration/report-status?uuid=%s&status=%s&dryrun='"$GH_OST_DRY_RUN"'&progress='"$GH_OST_PROGRESS"'&eta='"$GH_OST_ETA_SECONDS"

Running even more experiments to double validate this.

[shlomi-noach]

If I remove hooks, then the next thing gh-ost gets blocked on is opening a connection to the underlying database.

It's all about ports.

[deepthi]

• Was it intentional to checkin a new gh-ost binary?
• When we merge this, we should squash-merge since so many of the commits were experimental.

[deepthi]

Do we still need the port range fix? Or is the /etc/hosts fix sufficient?

[shlomi-noach]

This had immediate positive effect last week on local_example tests. I'll experiment without it.

[shlomi-noach]

Removed, and tests are green!

[shlomi-noach]

Was it intentional to checkin a new gh-ost binary?

No! It was experimental. Reverting.

When we merge this, we should squash-merge since so many of the commits were experimental.

Cool

[shlomi-noach]

Removing the port range patch, 8f4444c, seems to look good. Tests are still passing without the patch.

[shlomi-noach]

Suqash commit: 2179d0c