Online DDL: read and publish gh-ost FATAL message where possible

[shlomi-noach]

Description

When running gh-ost migrations, and gh-ost fails, gh-ost produces a detailed error log. But the vttablet that executes gh-ost only tells something like failed running command: ... ... error=exit status 1.

When the user consequently runs SHOW VITESS_MIGRATIONS LIKE '...' they see aforementioned generic error in the message column:

     migration_uuid: 5ca3b482_be0f_11eb_9cd1_0a43f95f28a3
           keyspace: commerce
              shard: 0
       mysql_schema: vt_commerce
        mysql_table: nopk
migration_statement: alter table nopk add primary key (id)
           strategy: gh-ost
            options: -skip-topo
    added_timestamp: 2021-05-26 10:44:35
requested_timestamp: 2021-05-26 10:44:35
    ready_timestamp: 2021-05-26 10:44:36
  started_timestamp: 2021-05-26 10:44:36
 liveness_timestamp: NULL
completed_timestamp: 2021-05-26 10:44:36
  cleanup_timestamp: NULL
   migration_status: failed
           log_path: ip-172-31-45-90:/tmp/online-ddl-5ca3b482_be0f_11eb_9cd1_0a43f95f28a3-409940722
          artifacts: _5ca3b482_be0f_11eb_9cd1_0a43f95f28a3_20210526104436_gho,_5ca3b482_be0f_11eb_9cd1_0a43f95f28a3_20210526104436_ghc,_5ca3b482_be0f_11eb_9cd1_0a43f95f28a3_20210526104436_del,
            retries: 0
             tablet: zone1-0000000100
     tablet_failure: 0
           progress: 0
  migration_context: vtgate:74e5e7d6-be0b-11eb-9cd1-0a43f95f28a3
         ddl_action: alter
            message: failed running command: ... ... ; error=exit status 1

This PR provides more useful information in in message column.

When gh-ost fails, it typically fails with a FATAL log message. This PR extract that FATAL log message, if possible. If successful, it replaces the generic error with the more informative message from the log. Examples:

*************************** 17. row ***************************
                 id: 17
     migration_uuid: 89ac1fd5_be13_11eb_8862_0a43f95f28a3
           keyspace: commerce
              shard: 0
       mysql_schema: vt_commerce
        mysql_table: corder
migration_statement: alter table corder drop column no_such_column
           strategy: gh-ost
            options: -skip-topo
...
         ddl_action: alter
            message: FATAL Error 1091: Can't DROP 'no_such_column'; check that column/key exists
        eta_seconds: -9223372036854775808

                 id: 18
     migration_uuid: 08bb121d_be14_11eb_8ce7_0a43f95f28a3
           keyspace: commerce
              shard: 0
       mysql_schema: vt_commerce
        mysql_table: nopk
migration_statement: alter table nopk add primary key (id)
           strategy: gh-ost
            options: -skip-topo
...
         ddl_action: alter
            message: FATAL No PRIMARY nor UNIQUE key found in table! Bailing out
        eta_seconds: -9223372036854775808

Related Issue(s)

• Partially addressing Online DDL: allow users to see migration log (in case of migration failure) #8178
• tracking: Online DDL: tracking issue #6926

Checklist

• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

[deepthi]

The change looks good.
Can one of the existing tests be enhanced to expect an error message?

[shlomi-noach]

Can one of the existing tests be enhanced to expect an error message?

You're absolutely right, yes, I'll add a test

[shlomi-noach]

Test added.

[shlomi-noach]

Re-request for review 🙏

[vmg]

I see you're into that ParseAndBind lifestyle now. 👍👍

[shlomi-noach]

I see you're into that ParseAndBind lifestyle now. +1+1

Heh, I actually created ParseAndBind...!