brie: add error info column and history backup/restore info in sql #22699

lichunzhu · 2021-02-03T09:48:43Z

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary: When we use BRIE to run a backup job, if our mysql connection is broken due to some reason, we can't infer whether this BRIE job is successful or not.

What is changed and how it works?

What's Changed: Save history brie jobs in the task queue and add an error info column.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
Run a BR job and stop the network.
Current:

mysql> show backups;
+----------------------+--------+-------------------+---------------------+---------------------+---------------------+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Destination          | State  | Progress          | Queue_time          | Execution_time      | Finish_time         | Connection | Message                                                                                                                                                                                                                                         |
+----------------------+--------+-------------------+---------------------+---------------------+---------------------+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| s3://chunzhu/backup/ | Backup | 83.33333333333333 | 2021-02-03 17:46:52 | 2021-02-03 17:46:52 | 2021-02-03 17:48:31 |          3 | [executor:8124]Backup failed: msg:"Io(Custom { kind: Other, error: \"failed to put object Error during dispatch: error trying to connect: tcp connect error: Connection refused (os error 111)\" })" : [BR:KV:ErrKVUnknown]unknown tikv error |
+----------------------+--------+-------------------+---------------------+---------------------+---------------------+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Previous:
No backup can be found.

Side effects

Performance regression
- Consumes more CPU
- Consumes more MEM
Breaking backward compatibility

Release note

No release note

lichunzhu · 2021-02-04T07:40:29Z

/cc kennytm,3pointer,overvenus,glorv

lichunzhu · 2021-02-22T11:24:05Z

/unlabel sig/execution

lichunzhu · 2021-02-22T11:30:45Z

/remove-sig execution

overvenus · 2021-02-22T12:35:05Z

executor/brie.go

+			if len(item.info.errStr) > 0 {
+				e.result.AppendString(7, item.info.errStr)
+			} else {
+				e.result.AppendNull(7)
+			}


Could you add a test to cover them?

lichunzhu · 2021-02-23T03:08:14Z

/remove-sig execution

lichunzhu · 2021-02-23T03:55:43Z

/remove-sig execution

…howBRIEInfo

…into refineShowBRIEInfo

lichunzhu · 2021-03-01T07:37:36Z

/run-integration-br-test

overvenus · 2021-03-02T09:23:45Z

/run-integration-br-test

overvenus

Rest LGTM

executor/brie.go

overvenus · 2021-03-02T12:18:18Z

util/chunk/row.go

+		} else {
+			switch ft[colIdx].EvalType() {


Why replace continue with else?

If colIdx is the last column of this row, we shouldn't put the comma(, ) after nil nor NULL.

overvenus

LGTM

ti-chi-bot · 2021-03-04T08:41:22Z

@overvenus: Please use /LGTM instead of LGTM when you want to approve the pull request by comment.
If you use the GitHub review feature, please approve the PR directly, the comment will not take effect in the GitHub review feature.
If you have any qustions please refer to lgtm command help or lgtm plugin design.

If you have approved this PR, please ignore this reply. This reply is being used as a temporary reply during the migration of the new bot and will be removed on April 1.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-06-21T10:13:33Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

kennytm
winoros

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

glorv · 2021-06-28T03:02:23Z

/run-check_dev_2

glorv · 2021-07-16T02:51:57Z

/run-check_dev_2

kennytm · 2021-07-18T04:51:28Z

/run-check_dev_2

https://ci.pingcap.net/blue/rest/organizations/jenkins/pipelines/tidb_ghpr_check_2/runs/17857/nodes/64/log/?start=0

stuck on TestCoprocessorOOMAction

kennytm · 2021-07-29T07:56:45Z

/run-check_dev_2

kennytm · 2021-07-30T02:43:01Z

/merge

ti-chi-bot · 2021-07-30T02:43:05Z

This pull request has been accepted and is ready to merge.

Commit hash: c936e17

ti-chi-bot · 2021-07-30T04:07:15Z

@lichunzhu: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

lichunzhu added 2 commits February 3, 2021 17:36

add error info column and history backup/restore info in sql

c86ab18

switch line

66994b8

lichunzhu requested review from a team as code owners February 3, 2021 09:48

lichunzhu requested review from lzmhhh123 and removed request for a team February 3, 2021 09:48

ti-srebot added the first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. label Feb 3, 2021

github-actions bot added the sig/execution SIG execution label Feb 3, 2021

fix bug

da6112e

ti-srebot requested review from 3pointer, glorv, kennytm and overvenus February 4, 2021 07:40

overvenus added the sig/migrate label Feb 22, 2021

ti-srebot removed the sig/execution SIG execution label Feb 22, 2021

github-actions bot added the sig/execution SIG execution label Feb 22, 2021

overvenus reviewed Feb 22, 2021

View reviewed changes

update message column

b49696e

lichunzhu force-pushed the refineShowBRIEInfo branch from dc3ec31 to b49696e Compare February 22, 2021 12:45

ti-chi-bot removed the sig/execution SIG execution label Feb 23, 2021

github-actions bot added the sig/execution SIG execution label Feb 23, 2021

ti-chi-bot removed the sig/execution SIG execution label Feb 23, 2021

Merge branch 'master' of https://github.com/pingcap/tidb into refineS…

052d1aa

…howBRIEInfo

ti-chi-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 23, 2021

github-actions bot added the sig/execution SIG execution label Feb 23, 2021

lichunzhu added 3 commits March 1, 2021 15:21

fix ut

4a203aa

Merge branch 'refineShowBRIEInfo' of https://github.com/lichunzhu/tidb …

2d14365

…into refineShowBRIEInfo

Merge branch 'master' into refineShowBRIEInfo

abbe4d4

overvenus reviewed Mar 2, 2021

View reviewed changes

overvenus reviewed Mar 4, 2021

View reviewed changes

winoros approved these changes Jun 21, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 21, 2021

kennytm approved these changes Jun 21, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 21, 2021

Merge branch 'master' into refineShowBRIEInfo

2d58b6d

Merge branch 'master' into refineShowBRIEInfo

87c5236

lzmhhh123 removed their request for review June 28, 2021 06:16

Merge branch 'master' into refineShowBRIEInfo

c936e17

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 30, 2021

ti-chi-bot added 2 commits July 30, 2021 10:43

Merge branch 'master' into refineShowBRIEInfo

d3b669f

Merge branch 'master' into refineShowBRIEInfo

9975d9b

ti-chi-bot merged commit ab4a990 into pingcap:master Jul 30, 2021

kennytm mentioned this pull request Aug 9, 2021

log backup result(success/failure) in backupmeta pingcap/br#1420

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

brie: add error info column and history backup/restore info in sql #22699

brie: add error info column and history backup/restore info in sql #22699

lichunzhu commented Feb 3, 2021 •

edited

lichunzhu commented Feb 4, 2021

lichunzhu commented Feb 22, 2021

lichunzhu commented Feb 22, 2021

overvenus Feb 22, 2021

lichunzhu commented Feb 23, 2021

lichunzhu commented Feb 23, 2021

lichunzhu commented Mar 1, 2021

overvenus commented Mar 2, 2021

overvenus left a comment

overvenus Mar 2, 2021

lichunzhu Mar 2, 2021

overvenus left a comment

ti-chi-bot commented Mar 4, 2021

ti-chi-bot commented Jun 21, 2021

glorv commented Jun 28, 2021

glorv commented Jul 16, 2021

kennytm commented Jul 18, 2021

kennytm commented Jul 29, 2021

kennytm commented Jul 30, 2021

ti-chi-bot commented Jul 30, 2021

ti-chi-bot commented Jul 30, 2021

brie: add error info column and history backup/restore info in sql #22699

brie: add error info column and history backup/restore info in sql #22699

Conversation

lichunzhu commented Feb 3, 2021 • edited

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

lichunzhu commented Feb 4, 2021

lichunzhu commented Feb 22, 2021

lichunzhu commented Feb 22, 2021

overvenus Feb 22, 2021

Choose a reason for hiding this comment

lichunzhu commented Feb 23, 2021

lichunzhu commented Feb 23, 2021

lichunzhu commented Mar 1, 2021

overvenus commented Mar 2, 2021

overvenus left a comment

Choose a reason for hiding this comment

overvenus Mar 2, 2021

Choose a reason for hiding this comment

lichunzhu Mar 2, 2021

Choose a reason for hiding this comment

overvenus left a comment

Choose a reason for hiding this comment

ti-chi-bot commented Mar 4, 2021

ti-chi-bot commented Jun 21, 2021

glorv commented Jun 28, 2021

glorv commented Jul 16, 2021

kennytm commented Jul 18, 2021

kennytm commented Jul 29, 2021

kennytm commented Jul 30, 2021

ti-chi-bot commented Jul 30, 2021

ti-chi-bot commented Jul 30, 2021

lichunzhu commented Feb 3, 2021 •

edited