server: support compressed MySQL protocol #36780

dveeden · 2022-08-01T12:20:17Z

What problem does this PR solve?

Issue Number: close #22605

Problem Summary: Support deflate compression in the mysql protocol

This helps with:

Performance: Especially data intensive work over low bandwidth connections
Cost: This can reduce cost when bandwidth is payed for, e.g. in Cloud environments.

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Support for the the compressed version of the MySQL protocol was added

ti-chi-bot · 2022-08-01T12:20:18Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

hawkingrei
lance6716

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

dveeden · 2022-08-01T12:21:18Z

/cc @bb7133 @xhebox

dveeden · 2022-08-01T12:25:58Z

Tested with:

mysql --compress
mysqlsh --compress
MySQL Connector/Python with mysql.connector.connect(compress=True, ...)

All the above with 8.x versions

sre-bot · 2022-08-01T12:41:41Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/04b0a5f9bf70136a12e06daf2f35c8b61889b5fe

dveeden · 2022-08-09T06:52:29Z

Tested with:

MySQL Connector/J 8.0.30 and useCompression=true
MySQL Client 8.0.30 and mysql --compression-algorithms=zlib,zstd --zstd-compression-level=9 ...
MySQL Shell 8.0.29 and mysqlsh --compression-algorithms=zstd --sql mysql://...
MySQL Connector/Python 8.0.21 and mysql.connector.connect(compress=True, ...)

dveeden · 2022-08-09T07:34:52Z

/run-common-test

lance6716 · 2023-02-16T15:09:09Z

/cc @lance6716

dveeden · 2023-02-17T13:33:32Z

With my latest update it now compresses multiple mysql packets into a single compressed packet once Flush() is called.

This works somewhat:

$ mysql -h 127.0.0.1 -u root -P 4000 --ssl-mode=DISABLED --compression-algorithms=zlib 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 405
Server version: 5.7.25-TiDB-v6.7.0-alpha-66-gc67761448-dirty TiDB Server (Apache License 2.0) Community Edition, MySQL 5.7 compatible

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> \s
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
Connection id:    407
Current database: *** NONE ***

--------------
mysql  Ver 8.0.32 for Linux on x86_64 (MySQL Community Server - GPL)

Connection id:		407
Current database:	
Current user:		root@127.0.0.1
SSL:			Not in use
Current pager:		stdout
Using outfile:		''
Using delimiter:	;
Server version:		5.7.25-TiDB-v6.7.0-alpha-66-gc67761448-dirty TiDB Server (Apache License 2.0) Community Edition, MySQL 5.7 compatible
Protocol version:	10
Connection:		127.0.0.1 via TCP/IP
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
Connection id:    409
Current database: *** NONE ***

Server characterset:	utf8mb4
Db     characterset:	utf8mb4
Client characterset:	utf8mb4
Conn.  characterset:	utf8mb4
TCP port:		4000
Protocol:		Compressed
Binary data as:		Hexadecimal
--------------

mysql>

Next up is to figure out where we violate the protocol and fix that.

- https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_compression.html - https://dev.mysql.com/worklog/task/?id=12039 - pingcap#22605

server/packetio.go

lance6716 · 2023-05-09T07:03:43Z

server/packetio.go

-		} else if n != (4 + mysql.MaxPayloadLen) {
-			return errors.Trace(mysql.ErrBadConn)
+		if p.compressionAlgorithm != mysql.CompressionNone {
+			if n, err := p.compressedWriter.Write(data[:4+maxPayloadLen]); err != nil {


seems we only use Write and Flush for both compressedWriter and bufWriter, maybe we can abstract an interface so there's no if-branch

@dveeden I have no strong opinion about it. You can merge this PR or address it in following commit.

server/packetio.go

dveeden · 2023-05-10T09:47:44Z

/test unit-test

dveeden · 2023-05-12T06:36:50Z

/merge

ti-chi-bot · 2023-05-12T06:36:54Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: fb55f65

ti-chi-bot requested review from bb7133 and xhebox August 1, 2022 12:21

dveeden force-pushed the mysql_proto_compressed branch from fadd451 to c668b62 Compare August 9, 2022 05:51

dveeden force-pushed the mysql_proto_compressed branch from d3e862a to 04b0a5f Compare August 11, 2022 22:02

dveeden force-pushed the mysql_proto_compressed branch from 04b0a5f to 6189bb0 Compare February 14, 2023 19:35

xhebox marked this pull request as ready for review February 15, 2023 03:28

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 15, 2023

xhebox marked this pull request as draft February 15, 2023 03:28

ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 15, 2023

ti-chi-bot requested a review from lance6716 February 16, 2023 15:09

dveeden force-pushed the mysql_proto_compressed branch from c677614 to 176d542 Compare April 12, 2023 19:40

dveeden force-pushed the mysql_proto_compressed branch 2 times, most recently from 1825842 to b61b8fb Compare May 2, 2023 08:35

server: support compressed MySQL protocol

6b44c45

- https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_compression.html - https://dev.mysql.com/worklog/task/?id=12039 - pingcap#22605

dveeden force-pushed the mysql_proto_compressed branch from b61b8fb to 6b44c45 Compare May 3, 2023 07:07

Update Bazel

9702391

hawkingrei approved these changes May 3, 2023

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 3, 2023

Fix missing error check, found by nogo/errcheck

a752576

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 3, 2023

dveeden added 2 commits May 4, 2023 08:14

Add tests for compressedWriter

eb29843

Add compressed packet testing to TestPacketIORead

c56f5ef

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 4, 2023

dveeden added 2 commits May 4, 2023 09:12

Add comment about magic nrs

8b74cce

Update BUILD.bazel for server

7b93e3d

lance6716 reviewed May 9, 2023

View reviewed changes

dveeden added 2 commits May 9, 2023 21:59

Small updates based on review

6484a7e

Correctly propagate zstd compression level

16a3079

lance6716 approved these changes May 10, 2023

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 10, 2023

dveeden added 2 commits May 10, 2023 10:40

Split large compressed packets

b218dcf

Fix nogo/errcheck

3c4597b

Update comment about splitting compressed packets

fb55f65

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label May 12, 2023

ti-chi-bot bot merged commit 72cf5df into pingcap:master May 12, 2023

yahonda mentioned this pull request Jun 7, 2023

Support SHOW SESSION STATUS LIKE "Compression"; #44473

Closed

djshow832 mentioned this pull request Sep 7, 2023

Support compressed MySQL protocol pingcap/tiproxy#361

Closed

dveeden mentioned this pull request Oct 3, 2023

server: Change zlib compression level #47387

Merged

13 tasks

you06 mentioned this pull request Aug 4, 2025

conn: add buffer for zlib & zstd compressor #62802

Merged

13 tasks

server: support compressed MySQL protocol #36780

server: support compressed MySQL protocol #36780

Uh oh!

Conversation

dveeden commented Aug 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

ti-chi-bot commented Aug 1, 2022 • edited by ti-chi-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dveeden commented Aug 1, 2022

Uh oh!

dveeden commented Aug 1, 2022

Uh oh!

sre-bot commented Aug 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dveeden commented Aug 9, 2022

Uh oh!

dveeden commented Aug 9, 2022

Uh oh!

lance6716 commented Feb 16, 2023

Uh oh!

dveeden commented Feb 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

lance6716 May 9, 2023

Choose a reason for hiding this comment

Uh oh!

lance6716 May 10, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dveeden commented May 10, 2023

Uh oh!

dveeden commented May 12, 2023

Uh oh!

ti-chi-bot bot commented May 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dveeden commented Aug 1, 2022 •

edited

Loading

ti-chi-bot commented Aug 1, 2022 •

edited by ti-chi-bot bot

Loading

sre-bot commented Aug 1, 2022 •

edited

Loading

dveeden commented Feb 17, 2023 •

edited

Loading