Add binlog_status for http api and `TIDB_SERVERS_INFO` table #13025

zier-one · 2019-10-30T08:35:47Z

What problem does this PR solve?

Users can check if all of tidb servers is enable binlog conveniently.

What is changed and how it works?

add a item named binlog_enabled into ServerInfo, and display it on http api and TIDB_SERVERS_INFO.

Manual test

When binlog is not enable

curl http://127.0.0.1:10080/info/all

{
 "servers_num": 2,
 "owner_id": "29a65ec0-d931-4f9e-a212-338eaeffab96",
 "is_all_server_version_consistent": true,
 "all_servers_info": {
  "29a65ec0-d931-4f9e-a212-338eaeffab96": {
   "version": "5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty",
   "git_hash": "8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e",
   "ddl_id": "29a65ec0-d931-4f9e-a212-338eaeffab96",
   "ip": "",
   "listening_port": 4000,
   "status_port": 10080,
   "lease": "45s",
   "binlog_status": "Off"
  },
  "cd13c9eb-c3ee-4887-af9b-e64f3162d92c": {
   "version": "5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty",
   "git_hash": "8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e",
   "ddl_id": "cd13c9eb-c3ee-4887-af9b-e64f3162d92c",
   "ip": "",
   "listening_port": 4001,
   "status_port": 10081,
   "lease": "45s",
   "binlog_status": "Off"
  }
 }
}

mysql> select * from information_schema.TIDB_SERVERS_INFO;
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
| DDL_ID                               | IP   | PORT | STATUS_PORT | LEASE | VERSION                                       | GIT_HASH                                 | BINLOG_STATUS |
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
| 29a65ec0-d931-4f9e-a212-338eaeffab96 |      | 4000 |       10080 | 45s   | 5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty | 8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e | Off           |
| cd13c9eb-c3ee-4887-af9b-e64f3162d92c |      | 4001 |       10081 | 45s   | 5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty | 8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e | Off           |
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
2 rows in set (0.00 sec)

When binlog is enable

curl http://127.0.0.1:10080/info/all
{
 "servers_num": 2,
 "owner_id": "b45f2a44-3794-459d-b969-4453b5cf4820",
 "is_all_server_version_consistent": true,
 "all_servers_info": {
  "419b3208-d6c0-4466-ae38-8895904c1483": {
   "version": "5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty",
   "git_hash": "8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e",
   "ddl_id": "419b3208-d6c0-4466-ae38-8895904c1483",
   "ip": "",
   "listening_port": 4001,
   "status_port": 10081,
   "lease": "45s",
   "binlog_status": "On"
  },
  "b45f2a44-3794-459d-b969-4453b5cf4820": {
   "version": "5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty",
   "git_hash": "8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e",
   "ddl_id": "b45f2a44-3794-459d-b969-4453b5cf4820",
   "ip": "",
   "listening_port": 4000,
   "status_port": 10080,
   "lease": "45s",
   "binlog_status": "On"
  }
 }
}

mysql> select * from information_schema.TIDB_SERVERS_INFO;
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
| DDL_ID                               | IP   | PORT | STATUS_PORT | LEASE | VERSION                                       | GIT_HASH                                 | BINLOG_STATUS |
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
| 419b3208-d6c0-4466-ae38-8895904c1483 |      | 4001 |       10081 | 45s   | 5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty | 8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e | On            |
| b45f2a44-3794-459d-b969-4453b5cf4820 |      | 4000 |       10080 | 45s   | 5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty | 8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e | On            |
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
2 rows in set (0.00 sec)

When binlog is enable and `ignore-error` is enable, but write binlog failure

curl http://127.0.0.1:10080/info/all

{
 "servers_num": 2,
 "owner_id": "4ebdeaea-70f0-472c-9ca9-8a237aa22b10",
 "is_all_server_version_consistent": true,
 "all_servers_info": {
  "142744c8-69b4-4779-8113-3c69200d6c16": {
   "version": "5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty",
   "git_hash": "8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e",
   "ddl_id": "142744c8-69b4-4779-8113-3c69200d6c16",
   "ip": "",
   "listening_port": 4001,
   "status_port": 10081,
   "lease": "45s",
   "binlog_status": "Skipping"
  },
  "4ebdeaea-70f0-472c-9ca9-8a237aa22b10": {
   "version": "5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty",
   "git_hash": "8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e",
   "ddl_id": "4ebdeaea-70f0-472c-9ca9-8a237aa22b10",
   "ip": "",
   "listening_port": 4000,
   "status_port": 10080,
   "lease": "45s",
   "binlog_status": "Skipping"
  }
 }
}

mysql> select * from information_schema.TIDB_SERVERS_INFO;
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
| DDL_ID                               | IP   | PORT | STATUS_PORT | LEASE | VERSION                                       | GIT_HASH                                 | BINLOG_STATUS |
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
| 4ebdeaea-70f0-472c-9ca9-8a237aa22b10 |      | 4000 |       10080 | 45s   | 5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty | 8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e | Skipping      |
| 142744c8-69b4-4779-8113-3c69200d6c16 |      | 4001 |       10081 | 45s   | 5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty | 8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e | Skipping      |
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
2 rows in set (0.00 sec)

When some of tidb server is old version which isn't support show `binlog_status`

curl http://127.0.0.1:10080/info/all

{
 "servers_num": 2,
 "owner_id": "e5998f02-bf2b-434f-80b9-48d3f6260fa1",
 "all_servers_diff_versions": [
  {
   "version": "5.7.25-TiDB-v4.0.0-alpha-638-g6fd74f3e4",
   "git_hash": "6fd74f3e4c42a61fa9dff1eaae4fcbad625f347a"
  },
  {
   "version": "5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty",
   "git_hash": "8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e"
  }
 ],
 "all_servers_info": {
  "dd480398-f595-4b35-8e33-773f6f15fae9": {
   "version": "5.7.25-TiDB-v4.0.0-alpha-638-g6fd74f3e4",
   "git_hash": "6fd74f3e4c42a61fa9dff1eaae4fcbad625f347a",
   "ddl_id": "dd480398-f595-4b35-8e33-773f6f15fae9",
   "ip": "",
   "listening_port": 4001,
   "status_port": 10081,
   "lease": "45s",
   "binlog_status": "Unknown"
  },
  "e5998f02-bf2b-434f-80b9-48d3f6260fa1": {
   "version": "5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty",
   "git_hash": "8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e",
   "ddl_id": "e5998f02-bf2b-434f-80b9-48d3f6260fa1",
   "ip": "",
   "listening_port": 4000,
   "status_port": 10080,
   "lease": "45s",
   "binlog_status": "On"
  }
 }
}

mysql> select * from information_schema.TIDB_SERVERS_INFO;
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
| DDL_ID                               | IP   | PORT | STATUS_PORT | LEASE | VERSION                                       | GIT_HASH                                 | BINLOG_STATUS |
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
| dd480398-f595-4b35-8e33-773f6f15fae9 |      | 4001 |       10081 | 45s   | 5.7.25-TiDB-v4.0.0-alpha-638-g6fd74f3e4       | 6fd74f3e4c42a61fa9dff1eaae4fcbad625f347a | Unknown       |
| e5998f02-bf2b-434f-80b9-48d3f6260fa1 |      | 4000 |       10080 | 45s   | 5.7.25-TiDB-v4.0.0-alpha-669-g8f2a09a52-dirty | 8f2a09a52fdcaf9d9bfd775d2c6023f363dc121e | On            |
+--------------------------------------+------+------+-------------+-------+-----------------------------------------------+------------------------------------------+---------------+
2 rows in set (0.00 sec)

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Related changes

Need to cherry-pick to the release branch
Need to update the documentation

shenli · 2019-10-30T08:41:50Z

domain/infosync/info.go

+	Port          uint   `json:"listening_port"`
+	StatusPort    uint   `json:"status_port"`
+	Lease         string `json:"lease"`
+	BinlogEnabled bool   `json:"binlog_enabled"`


If the tidb-server is disconneted with Pump and the user chooses to ignore the error, will the tidb-server update the binlog_enabled to false?

fixed PTAL again

codecov · 2019-10-30T09:00:19Z

Codecov Report

Merging #13025 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #13025   +/-   ##
===========================================
  Coverage   80.1291%   80.1291%           
===========================================
  Files           469        469           
  Lines        111188     111188           
===========================================
  Hits          89094      89094           
  Misses        15209      15209           
  Partials       6885       6885

ericsyh · 2019-10-30T10:14:41Z

@leoppro Need to update the infoall api response sample in tidb_http_api.md

tiancaiamao · 2019-10-30T11:00:37Z

/run-all-tests

tiancaiamao · 2019-10-30T11:20:54Z

/run-all-tests

zier-one · 2019-10-31T07:26:35Z

I will fix ci later

sre-bot · 2019-11-06T02:57:57Z

Your auto merge job has been accepted, waiting for 13152

sre-bot · 2019-11-06T03:04:29Z

/run-all-tests

sre-bot · 2019-11-06T03:07:10Z

@leoppro merge failed.

zier-one · 2019-11-06T03:15:03Z

/run-all-tests

zier-one · 2019-11-06T03:18:05Z

/run-all-tests

zier-one · 2019-11-06T03:27:47Z

/merge

sre-bot · 2019-11-06T03:27:50Z

Your auto merge job has been accepted, waiting for 13164

sre-bot · 2019-11-06T03:43:41Z

/run-all-tests

sre-bot · 2019-11-06T03:47:55Z

cherry pick to release-3.0 failed

sre-bot · 2019-11-06T03:48:21Z

cherry pick to release-3.1 failed

sre-bot · 2019-11-06T03:48:54Z

cherry pick to release-2.1 failed

…#13025) # Conflicts: # domain/info.go # infoschema/tables.go # sessionctx/binloginfo/binloginfo.go

…#13025) # Conflicts: # domain/info.go # infoschema/tables.go # sessionctx/binloginfo/binloginfo.go # Conflicts: # docs/tidb_http_api.md

leoppro added 3 commits October 29, 2019 18:54

add binlog status to info http api

caddc21

Merge remote-tracking branch 'upstream/master' into check_binlog

979c53b

show binlog_enabled in system_table

e08e16b

zier-one added the status/WIP label Oct 30, 2019

fmt

189a4a3

shenli reviewed Oct 30, 2019

View reviewed changes

Merge branch 'master' into check_binlog

c469d62

tiancaiamao added the status/all tests passed label Oct 30, 2019

leoppro added 3 commits October 31, 2019 11:59

Merge remote-tracking branch 'upstream/master' into check_binlog

8f2a09a

add binlog_status

e1e8392

Merge branch 'check_binlog' of github.com:leoppro/tidb into check_binlog

98a638a

zier-one changed the title ~~Add binlog_enabled status for http api and TIDB_SERVERS_INFO table~~ Add binlog_status for http api and TIDB_SERVERS_INFO table Oct 31, 2019

leoppro added 3 commits October 31, 2019 15:21

add comments

388e175

Merge branch 'master' into check_binlog

4d9670d

update docs

1001b4a

zier-one removed the status/all tests passed label Oct 31, 2019

leoppro added 4 commits October 31, 2019 16:47

fix ci

047d108

Merge branch 'check_binlog' of github.com:leoppro/tidb into check_binlog

ed895bd

fix ci

04792d6

fix statusListener maybe called more than once

560a1d3

zier-one added component/binlog needs-cherry-pick-3.0 and removed status/WIP labels Oct 31, 2019

Merge branch 'master' into check_binlog

9ce19d5

sre-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 6, 2019

tiancaiamao approved these changes Nov 6, 2019

View reviewed changes

zier-one added the needs-cherry-pick-3.1 label Nov 6, 2019

Merge branch 'master' into check_binlog

9ced4b1

zier-one added the needs-cherry-pick-2.1 label Nov 6, 2019

Merge branch 'master' into check_binlog

ce8496a

Merge branch 'master' into check_binlog

6bb7d0f

sre-bot merged commit 903cd1f into pingcap:master Nov 6, 2019

zier-one deleted the check_binlog branch November 6, 2019 05:17

zier-one pushed a commit to zier-one/tidb that referenced this pull request Nov 6, 2019

Add binlog_status for http api and TIDB_SERVERS_INFO table (pingcap…

d214da3

…#13025) # Conflicts: # domain/info.go # infoschema/tables.go # sessionctx/binloginfo/binloginfo.go

zier-one mentioned this pull request Nov 6, 2019

Add binlog_status for http api and TIDB_SERVERS_INFO table (#13025) #13184

Merged

zier-one pushed a commit to zier-one/tidb that referenced this pull request Nov 6, 2019

Add binlog_status for http api and TIDB_SERVERS_INFO table (pingcap…

b9757e4

…#13025) # Conflicts: # domain/info.go # infoschema/tables.go # sessionctx/binloginfo/binloginfo.go

zier-one mentioned this pull request Nov 6, 2019

Add binlog_status for http api and TIDB_SERVERS_INFO table (#13025) #13187

Merged

zier-one mentioned this pull request Nov 6, 2019

Add binlog_status for http api and TIDB_SERVERS_INFO table (#13025) #13188

Merged

zimulala pushed a commit that referenced this pull request Nov 8, 2019

Add binlog_status for http api and TIDB_SERVERS_INFO table (#13025) (…

75cd818

…#13188)

sre-bot pushed a commit that referenced this pull request Nov 8, 2019

Add binlog_status for http api and TIDB_SERVERS_INFO table (#13025) (…

81d8981

…#13184)

sre-bot pushed a commit that referenced this pull request Nov 8, 2019

Add binlog_status for http api and TIDB_SERVERS_INFO table (#13025) (…

cb2b715

…#13187)

XiaTianliang pushed a commit to XiaTianliang/tidb that referenced this pull request Dec 21, 2019

Add binlog_status for http api and TIDB_SERVERS_INFO table (pingcap…

0be7d33

…#13025)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add binlog_status for http api and `TIDB_SERVERS_INFO` table #13025

Add binlog_status for http api and `TIDB_SERVERS_INFO` table #13025

zier-one commented Oct 30, 2019 •

edited

shenli Oct 30, 2019

zier-one Oct 31, 2019

codecov bot commented Oct 30, 2019 •

edited

ericsyh commented Oct 30, 2019

tiancaiamao commented Oct 30, 2019

tiancaiamao commented Oct 30, 2019

zier-one commented Oct 31, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

zier-one commented Nov 6, 2019

zier-one commented Nov 6, 2019

zier-one commented Nov 6, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

Add binlog_status for http api and TIDB_SERVERS_INFO table #13025

Add binlog_status for http api and TIDB_SERVERS_INFO table #13025

Conversation

zier-one commented Oct 30, 2019 • edited

What problem does this PR solve?

What is changed and how it works?

Manual test

When binlog is not enable

When binlog is enable

When binlog is enable and ignore-error is enable, but write binlog failure

When some of tidb server is old version which isn't support show binlog_status

Check List

shenli Oct 30, 2019

Choose a reason for hiding this comment

zier-one Oct 31, 2019

Choose a reason for hiding this comment

codecov bot commented Oct 30, 2019 • edited

Codecov Report

ericsyh commented Oct 30, 2019

tiancaiamao commented Oct 30, 2019

tiancaiamao commented Oct 30, 2019

zier-one commented Oct 31, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

zier-one commented Nov 6, 2019

zier-one commented Nov 6, 2019

zier-one commented Nov 6, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

sre-bot commented Nov 6, 2019

Add binlog_status for http api and `TIDB_SERVERS_INFO` table #13025

Add binlog_status for http api and `TIDB_SERVERS_INFO` table #13025

zier-one commented Oct 30, 2019 •

edited

When binlog is enable and `ignore-error` is enable, but write binlog failure

When some of tidb server is old version which isn't support show `binlog_status`

codecov bot commented Oct 30, 2019 •

edited