Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server: only owner do bootstrap #10029

Merged
merged 13 commits into from Apr 11, 2019

Conversation

Projects
None yet
7 participants
@crazycs520
Copy link
Contributor

crazycs520 commented Apr 3, 2019

What problem does this PR solve?

implement issue: #10017
When there multiple tidb-server do bootstrap, there will be multiple conflict when create system table.
Below is result when I start 5 tidb-server at the same time:

test > admin show ddl jobs 50;
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+-----------+
| JOB_ID | DB_NAME | TABLE_NAME           | JOB_TYPE      | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | START_TIME                        | STATE     |
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+-----------+
| 68     | mysql   | bind_info            | create table  | public       | 11        | 67       | 0         | 2019-04-03 19:18:27.193 +0800 CST | synced    |
| 66     | mysql   | default_roles        | create table  | public       | 11        | 65       | 0         | 2019-04-03 19:18:27.141 +0800 CST | synced    |
| 64     | mysql   | role_edges           | create table  | public       | 11        | 63       | 0         | 2019-04-03 19:18:27.04 +0800 CST  | synced    |
| 62     | mysql   |                      | create table  | none         | 11        | 61       | 0         | 2019-04-03 19:18:26.991 +0800 CST | cancelled |
| 60     | mysql   |                      | create table  | none         | 11        | 59       | 0         | 2019-04-03 19:18:26.939 +0800 CST | cancelled |
| 58     | mysql   |                      | create table  | none         | 11        | 57       | 0         | 2019-04-03 19:18:26.892 +0800 CST | cancelled |
| 56     | mysql   | stats_feedback       | create table  | public       | 11        | 55       | 0         | 2019-04-03 19:18:26.842 +0800 CST | synced    |
| 54     | mysql   | gc_delete_range_done | create table  | public       | 11        | 53       | 0         | 2019-04-03 19:18:26.842 +0800 CST | synced    |
| 52     | mysql   |                      | create table  | none         | 11        | 51       | 0         | 2019-04-03 19:18:26.789 +0800 CST | cancelled |
| 50     | mysql   | gc_delete_range      | create table  | public       | 11        | 49       | 0         | 2019-04-03 19:18:26.739 +0800 CST | synced    |
| 48     | mysql   | stats_buckets        | create table  | public       | 11        | 47       | 0         | 2019-04-03 19:18:26.689 +0800 CST | synced    |
| 46     | mysql   | stats_histograms     | create table  | public       | 11        | 45       | 0         | 2019-04-03 19:18:26.591 +0800 CST | synced    |
| 44     | mysql   | stats_meta           | create table  | public       | 11        | 43       | 0         | 2019-04-03 19:18:26.539 +0800 CST | synced    |
| 42     | mysql   | help_topic           | create table  | public       | 11        | 41       | 0         | 2019-04-03 19:18:26.491 +0800 CST | synced    |
| 40     | mysql   |                      | create table  | none         | 11        | 39       | 0         | 2019-04-03 19:18:26.391 +0800 CST | cancelled |
| 38     | mysql   |                      | create table  | none         | 11        | 37       | 0         | 2019-04-03 19:18:26.391 +0800 CST | cancelled |
| 36     | mysql   |                      | create table  | none         | 11        | 35       | 0         | 2019-04-03 19:18:26.391 +0800 CST | cancelled |
| 34     | mysql   | tidb                 | create table  | public       | 11        | 33       | 0         | 2019-04-03 19:18:26.339 +0800 CST | synced    |
| 32     | mysql   | GLOBAL_VARIABLES     | create table  | public       | 11        | 31       | 0         | 2019-04-03 19:18:26.339 +0800 CST | synced    |
| 30     | mysql   |                      | create table  | none         | 11        | 29       | 0         | 2019-04-03 19:18:26.291 +0800 CST | cancelled |
| 28     | mysql   | columns_priv         | create table  | public       | 11        | 27       | 0         | 2019-04-03 19:18:26.242 +0800 CST | synced    |
| 26     | mysql   | tables_priv          | create table  | public       | 11        | 25       | 0         | 2019-04-03 19:18:26.19 +0800 CST  | synced    |
| 24     | mysql   | db                   | create table  | public       | 11        | 23       | 0         | 2019-04-03 19:18:26.142 +0800 CST | synced    |
| 22     | mysql   |                      | create table  | none         | 11        | 21       | 0         | 2019-04-03 19:18:25.894 +0800 CST | cancelled |
| 20     | mysql   |                      | create table  | none         | 11        | 19       | 0         | 2019-04-03 19:18:25.894 +0800 CST | cancelled |
| 18     | mysql   |                      | create table  | none         | 11        | 17       | 0         | 2019-04-03 19:18:25.842 +0800 CST | cancelled |
| 16     | mysql   |                      | create table  | none         | 11        | 15       | 0         | 2019-04-03 19:18:25.694 +0800 CST | cancelled |
| 14     | mysql   | user                 | create table  | public       | 11        | 13       | 0         | 2019-04-03 19:18:25.54 +0800 CST  | synced    |
| 12     | mysql   |                      | create schema | public       | 11        | 0        | 0         | 2019-04-03 19:18:25.494 +0800 CST | synced    |
| 10     |         |                      | create schema | none         | 9         | 0        | 0         | 2019-04-03 19:18:25.39 +0800 CST  | cancelled |
| 8      |         |                      | create schema | none         | 3         | 0        | 0         | 2019-04-03 19:18:25.39 +0800 CST  | cancelled |
| 7      |         |                      | create schema | none         | 1         | 0        | 0         | 2019-04-03 19:18:25.39 +0800 CST  | cancelled |
| 6      |         |                      | create schema | none         | 2         | 0        | 0         | 2019-04-03 19:18:25.339 +0800 CST | cancelled |
| 5      | test    |                      | create schema | public       | 4         | 0        | 0         | 2019-04-03 19:18:25.189 +0800 CST | synced    |
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+-----------+

As you can see, there are multiple DDL-job was been canceled.

Actual only 1 tidb-server need to do bootstrap to create system table.

So which one? The DDL-owner

What is changed and how it works?

Add a judge to check whether is owner? If is owner, to do bootstrap to create system table.

In this PR, start 5 tidb-server at the same time, there is no conflict.

test> admin show ddl jobs 50;
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| JOB_ID | DB_NAME | TABLE_NAME           | JOB_TYPE      | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | START_TIME                        | STATE  |
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| 36     | mysql   | bind_info            | create table  | public       | 3         | 35       | 0         | 2019-04-03 19:32:42.293 +0800 CST | synced |
| 34     | mysql   | default_roles        | create table  | public       | 3         | 33       | 0         | 2019-04-03 19:32:42.242 +0800 CST | synced |
| 32     | mysql   | role_edges           | create table  | public       | 3         | 31       | 0         | 2019-04-03 19:32:42.142 +0800 CST | synced |
| 30     | mysql   | stats_feedback       | create table  | public       | 3         | 29       | 0         | 2019-04-03 19:32:42.092 +0800 CST | synced |
| 28     | mysql   | gc_delete_range_done | create table  | public       | 3         | 27       | 0         | 2019-04-03 19:32:42.042 +0800 CST | synced |
| 26     | mysql   | gc_delete_range      | create table  | public       | 3         | 25       | 0         | 2019-04-03 19:32:41.942 +0800 CST | synced |
| 24     | mysql   | stats_buckets        | create table  | public       | 3         | 23       | 0         | 2019-04-03 19:32:41.893 +0800 CST | synced |
| 22     | mysql   | stats_histograms     | create table  | public       | 3         | 21       | 0         | 2019-04-03 19:32:41.843 +0800 CST | synced |
| 20     | mysql   | stats_meta           | create table  | public       | 3         | 19       | 0         | 2019-04-03 19:32:41.742 +0800 CST | synced |
| 18     | mysql   | help_topic           | create table  | public       | 3         | 17       | 0         | 2019-04-03 19:32:41.691 +0800 CST | synced |
| 16     | mysql   | tidb                 | create table  | public       | 3         | 15       | 0         | 2019-04-03 19:32:41.643 +0800 CST | synced |
| 14     | mysql   | GLOBAL_VARIABLES     | create table  | public       | 3         | 13       | 0         | 2019-04-03 19:32:41.542 +0800 CST | synced |
| 12     | mysql   | columns_priv         | create table  | public       | 3         | 11       | 0         | 2019-04-03 19:32:41.493 +0800 CST | synced |
| 10     | mysql   | tables_priv          | create table  | public       | 3         | 9        | 0         | 2019-04-03 19:32:41.443 +0800 CST | synced |
| 8      | mysql   | db                   | create table  | public       | 3         | 7        | 0         | 2019-04-03 19:32:41.343 +0800 CST | synced |
| 6      | mysql   | user                 | create table  | public       | 3         | 5        | 0         | 2019-04-03 19:32:41.294 +0800 CST | synced |
| 4      | mysql   |                      | create schema | public       | 3         | 0        | 0         | 2019-04-03 19:32:41.242 +0800 CST | synced |
| 2      | test    |                      | create schema | public       | 1         | 0        | 0         | 2019-04-03 19:32:41.141 +0800 CST | synced |
+--------+---------+----------------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+

Check List

Tests

  • I don't know how to test? Any advise please let me know?

Code changes

Side effects

Related changes

@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Apr 3, 2019

@codecov

This comment has been minimized.

Copy link

codecov bot commented Apr 3, 2019

Codecov Report

❗️ No coverage uploaded for pull request base (master@e90f3b9). Click here to learn what that means.
The diff coverage is 50%.

@@             Coverage Diff             @@
##             master     #10029   +/-   ##
===========================================
  Coverage          ?   78.0309%           
===========================================
  Files             ?        405           
  Lines             ?      82161           
  Branches          ?          0           
===========================================
  Hits              ?      64111           
  Misses            ?      13342           
  Partials          ?       4708
@xiekeyi98
Copy link
Contributor

xiekeyi98 left a comment

Do we need some logs here?
Such as, logutil.Logger(context.Background()).Info("bootstrap success");

crazycs520 added some commits Apr 8, 2019

@tiancaiamao

This comment has been minimized.

Copy link
Contributor

tiancaiamao commented Apr 8, 2019

LGTM

Show resolved Hide resolved session/bootstrap.go Outdated
@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Apr 9, 2019

@xiekeyi98 After consideration, I think print "bootstrap success" log here is too early, there is a lot of other jobs need to do. Maybe we can add this log in some other place.

crazycs520 added some commits Apr 10, 2019

@winkyao
Copy link
Member

winkyao left a comment

LGTM

@winkyao

This comment has been minimized.

Copy link
Member

winkyao commented Apr 10, 2019

@zimulala PTAL

@zimulala
Copy link
Member

zimulala left a comment

LGTM

@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Apr 10, 2019

/run-all-tests

@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Apr 10, 2019

/run-all-tests

@zimulala

This comment has been minimized.

Copy link
Member

zimulala commented Apr 10, 2019

/run-all-tests

@zimulala zimulala added status/LGT3 and removed status/LGT1 labels Apr 10, 2019

@shenli

This comment has been minimized.

Copy link
Member

shenli commented Apr 10, 2019

The cancelled jobs have only a few info. Is it a bug?

40     | mysql   |                      | create table  | none         | 11        | 39       | 0         | 2019-04-03 19:18:26.391 +0800 CST | cancelled |
@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Apr 11, 2019

Because the table name is got from domin.infoschema by table ID, but the create table job was cancelled, so can't find table by job.tableID.

Maybe we can get the table name from job.Arg.

@crazycs520 crazycs520 merged commit 5a34c4b into pingcap:master Apr 11, 2019

6 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
idc-jenkins-ci-tidb/build Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/check_dev Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/code_coverage Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details

@crazycs520 crazycs520 deleted the crazycs520:bootstrap-in-owner branch Apr 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.