VReplication: Support MySQL Binary Log Transaction Compression #12950

mattlord · 2023-04-21T21:59:51Z

Description

MySQL 8.0 added support for binary log compression via transaction (GTID) compression in 8.0.20. You can read more about this feature here: https://dev.mysql.com/doc/refman/8.0/en/binary-log-transaction-compression.html

This can — at the cost of increased CPU usage — dramatically reduce the amount of data sent over the wire for MySQL replication while also dramatically reducing the overall storage space needed to retain binary logs (for replication, backup and recovery, CDC, etc). For larger installations this was a very desirable feature and while you could technically use it with Vitess (the MySQL replica sets making up each shard could use it fine) there was one very big limitation — VReplication workflows would not work. Given the criticality of VReplication workflows within Vitess, this meant that in practice this MySQL feature was not usable within Vitess clusters.

This PR addresses the issue by adding support for processing the compressed transaction events in VReplication, without any (known) limitations.

Note if you want to test this locally — e.g. with the local examples — create a my.cnf file that enables the option and use that in the EXTRA_MY_CNF env variable before starting the cluster (mysqlctl looks for this). For example:

$ cat ~/.my.cnf
[mysql]
binary-as-hex=false

[mysqld]
binlog_transaction_compression = ON

$ env | grep EXTRA
EXTRA_MY_CNF=/Users/matt/.my.cnf

Related Issue(s)

Fixes: MySQL 8.0 binlog compression does not work with Vitess VReplication #7567

Checklist

"Backport to:" labels have been added if this change should be back-ported
Tests were updated
Did the new or modified tests pass consistently locally and on the CI
Documentation: Document VStreamerCompressedTransactionsDecoded metric website#1453

Signed-off-by: Matt Lord <mattalord@gmail.com>

vitess-bot · 2023-04-21T21:59:54Z

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

Ensure that the Pull Request has a descriptive title.
If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
If a test is added or modified, there should be a documentation on top of the test to explain what the expected behavior is what the test does.

If a new flag is being introduced:

Is it really necessary to add this flag?
Flag names should be clear and intuitive (as far as possible)
Help text should be descriptive.
Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

Each item in Jobs should be named in order to mark it as required.
If the workflow should be required, the maintainer team should be notified.

Bug fixes

There should be at least one unit or end-to-end test.
The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

There should be some code comments as to why things are implemented the way they are.

New/Existing features

Should be documented, either by modifying the existing documentation or creating new documentation.
New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

Protobuf changes should be wire-compatible.
Changes to _vt tables and RPCs need to be backward compatible.
vtctl command output order should be stable and awk-able.
RPC changes should be compatible with vitess-operator
If a flag is removed, then it should also be removed from VTop, if used there.

Signed-off-by: Matt Lord <mattalord@gmail.com>

We needed more random data in order to generate the necessary binlog size when compression is enabled. Signed-off-by: Matt Lord <mattalord@gmail.com>

Signed-off-by: Matt Lord <mattalord@gmail.com>

rohit-nayak-ps

Very nice :-)

Similar to VStreamerEventsStreamed, should we add a VStreamerCompressedEventsStreamed and possibly metrics for bytes before/after compression for visibility? Not sure if it will add value, so just thinking aloud here.

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord · 2023-04-26T14:20:12Z

Very nice :-)

Similar to VStreamerEventsStreamed, should we add a VStreamerCompressedEventsStreamed and possibly metrics for bytes before/after compression for visibility? Not sure if it will add value, so just thinking aloud here.

Good idea. We don't stream them, but we decode them and stream the internal events. So I added a metric for VStreamerCompressedTransactionsDecoded.

I'll open a docs PR to add that here:
https://vitess.io/docs/17.0/reference/vreplication/metrics/

Signed-off-by: Matt Lord <mattalord@gmail.com>

go/mysql/binlog_event.go

go/mysql/binlog_event_compression.go

dbussink · 2023-04-27T19:10:18Z

go/mysql/binlog_event_compression.go

+}
+
+// Create a reader that caches decompressors.
+var zstdDecoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(0))


Is this concurrency safe? Would we ever have multiple decodings happening in parallel?

Ah wait, looks like https://pkg.go.dev/github.com/klauspost/compress/zstd#NewReader answers this as we're using DecodeAll.

Would it be useful at some point to do streaming mode? Since from what I understand a single encrypted entry can contain multiple events, we would already emit events while decompressing so we don't have to decompress everything into memory?

I was using the streaming mode at first and it's pretty slow. It would make sense to potentially switch to that based on the uncompressed payload size.... maybe we switch to streaming if it's > 128MiB or something?

@mattlord Hmm, interesting. How slow? Seems like maybe something we should dig into at some point but it doesn't have to hold up this PR I think.

Not THAT slow, but slow enough that it caused failures in the OnlineDDL VRepl stress tests — which have a bunch of threads executing small transactions against the DB while the workflow is running — as the vstream could not keep up enough to perform the final cutover before hitting the cutover timeout.

@mattlord That reminds me. I think we should update to the latest https://github.com/klauspost/compress as well here, looks like there recently were quite a few ztsd improvements.

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord · 2023-04-27T20:23:30Z

Thanks for the review, @dbussink ! I think that I've now addressed most of your comments.

mattlord · 2023-04-27T20:35:15Z

go/mysql/binlog_event_compression.go

+// At what size should we switch from the in-memory buffer
+// decoding to streaming mode -- which is slower, but does not
+// require everything be done in memory.
+const zstdInMemoryDecompressorMaxSize = 128 << (10 * 2) // 128MB


Maybe we should make this a vttablet flag? ...

shlomi-noach

Just a few comments. I realize you've turned some magic numbers into constants, but while we're here, the current codebase makes false assumptions when parsing the events (it assumes CRC32 is enabled when it might not be).

Is there a potential to add a unit test file in go/mysql to test the new TransactionPayload parsing?

shlomi-noach · 2023-04-28T03:58:27Z

go/mysql/binlog_event_common.go

+	// Offset from 0 where the 4 byte length is stored.
+	binlogEventLenOffset = 9
+	// Byte length of the checksum suffix.
+	binlogChecksumLen = 4


Please rename this as binlogCRC32ChecksumLen; there are potentially, even if not practically, other tpyes of checksums with different lengths.

Please make all of the above public constants; I have a use case to use those values externally (and I have, like you, created constants for those outside Vitess).

shlomi-noach · 2023-04-28T04:01:21Z

go/mysql/binlog_event_common.go

 // dataBytes returns the event bytes without header prefix and without checksum suffix
 func (ev binlogEvent) dataBytes(f BinlogFormat) []byte {
 	data := ev.Bytes()[f.HeaderLength:]
-	data = data[0 : len(data)-4]
+	data = data[0 : len(data)-binlogChecksumLen]


We should really check BinlogFormat to see which checksum is being used, before assuming a checksum is at all used. Sample code:

switch f.ChecksumAlgorithm { case mysql.BinlogChecksumAlgCRC32: .... size is binlogChecksumLen case mysql.BinlogChecksumAlgOff: ... size is 0 default: return vterrors.Errorf(vtrpc.Code_INVALID_ARGUMENT, "unsupported checksum algorithm: %v", format.ChecksumAlgorithm)

I'd rather not expand this PR. I don't disagree with you that we can improve this but it's not directly related and I want to keep this focused.

shlomi-noach · 2023-04-28T04:03:19Z

go/mysql/binlog_event_mariadb.go

-		checksum := data[length-4:]
-		data = data[:length-4]
+		checksum := data[length-binlogChecksumLen:]
+		data = data[:length-binlogChecksumLen]


again, we must not assume CRC32 here.

Again, sure. But it's not related to this PR, I only made the 4 less "magical". 🙂

It's also worth noting that the checksum is either 0 bytes, when there is no checksum, or it's 4 bytes regardless of the algorithm used. In both MySQL and MariaDB. Although in both cases none and CRC32 are the ONLY options supported today:
https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#option_mysqld_binlog-checksum

So we're being forward looking here, which I'm all for. But again, I don't want to expand the scope of this PR.

go/mysql/binlog_event_mysql56.go

shlomi-noach · 2023-04-28T04:08:29Z

go/mysql/binlog_event_compression.go

+var transactionPayloadCompressionTypes = map[uint64]string{
+	transactionPayloadCompressionZstd: "ZSTD",
+	transactionPayloadCompressionNone: "NONE",
+}


I might have a need for these to be public to; but I will iterate in a different PR if needed.

go/mysql/binlog_event_compression.go

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord · 2023-04-28T16:13:03Z

Thank you for the review, @shlomi-noach ! ❤️ I believe that I've addressed all of your comments now (w/o expanding the scope of the PR too much). Please let me know if I missed something though.

Signed-off-by: Matt Lord <mattalord@gmail.com>

vmg

This looks great. I think we can still remove some allocations from the ZSTD decompression code but I don't want to block the PR, since the behavior is correct. I'll run some benchmarks next week!

mattlord · 2023-04-28T16:25:01Z

This looks great. I think we can still remove some allocations from the ZSTD decompression code but I don't want to block the PR, since the behavior is correct. I'll run some benchmarks next week!

@vmg Yeah, I agree. That's why I wanted you, the perf master, to review. 😄 Thank you! ❤️

Signed-off-by: Matt Lord <mattalord@gmail.com>

…ansaction Compression (vitessio#2047) * cherry pick of 12950 * Fix merge conflicts Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> --------- Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com>

mattlord added 3 commits April 21, 2023 09:51

WIP

512f6c4

Signed-off-by: Matt Lord <mattalord@gmail.com>

Correct variable length field processing

c2a9dcf

Signed-off-by: Matt Lord <mattalord@gmail.com>

More tweaks and fixes

00b77e7

Signed-off-by: Matt Lord <mattalord@gmail.com>

vitess-bot bot added NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says labels Apr 21, 2023

github-actions bot added this to the v17.0.0 milestone Apr 21, 2023

mattlord added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: VReplication labels Apr 21, 2023

mattlord added 4 commits April 21, 2023 18:23

Add more comments

f73b680

Signed-off-by: Matt Lord <mattalord@gmail.com>

Minor changes

4e06432

Signed-off-by: Matt Lord <mattalord@gmail.com>

Fix VStreamFlush test

06fc05c

We needed more random data in order to generate the necessary binlog size when compression is enabled. Signed-off-by: Matt Lord <mattalord@gmail.com>

Move info logging to a higher/debug log level.

2b69ba7

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the vrepl_binlog_compression branch from f115ab4 to 2b69ba7 Compare April 24, 2023 14:06

mattlord added 8 commits April 24, 2023 14:32

Deal with additional timing issues in onlineddl tests

ee2673b

Signed-off-by: Matt Lord <mattalord@gmail.com>

Correct 3 byte length encoding handling

e085b43

Signed-off-by: Matt Lord <mattalord@gmail.com>

Enable binlog-transaction-compression for all vrepl MySQL 8.0 workflows

fc8786a

Signed-off-by: Matt Lord <mattalord@gmail.com>

Merge remote-tracking branch 'origin/main' into vrepl_binlog_compression

ffe86e4

Signed-off-by: Matt Lord <mattalord@gmail.com>

Switch from stream usage to in-memory buffer usage with compress/zstd.

a1fcc1f

Signed-off-by: Matt Lord <mattalord@gmail.com>

Revert onlinddl e2e test changes

24cf708

Signed-off-by: Matt Lord <mattalord@gmail.com>

Use uint64 everywhere

376e7f2

Signed-off-by: Matt Lord <mattalord@gmail.com>

Minor changes after self review

5da6339

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord removed NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says labels Apr 25, 2023

mattlord marked this pull request as ready for review April 25, 2023 16:01

mattlord requested review from GuptaManan100, frouioui, rohit-nayak-ps, deepthi and harshit-gangal as code owners April 25, 2023 16:01

Update release notes summary

8f760b7

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the vrepl_binlog_compression branch from a9057b1 to 8f760b7 Compare April 25, 2023 18:31

mattlord mentioned this pull request Apr 25, 2023

VReplication Workflows: support noblob binlog_row_image format for MoveTables and Reshard #12905

Merged

4 tasks

rohit-nayak-ps approved these changes Apr 26, 2023

View reviewed changes

mattlord added 2 commits April 26, 2023 10:14

Add VStreamerCompressedTransactionsDecoded metric

b5f64ba

Signed-off-by: Matt Lord <mattalord@gmail.com>

Merge remote-tracking branch 'origin/main' into vrepl_binlog_compression

6088e64

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord mentioned this pull request Apr 26, 2023

Document VStreamerCompressedTransactionsDecoded metric vitessio/website#1453

Merged

Minor text change

a889eaf

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord requested a review from vmg April 27, 2023 17:25

dbussink reviewed Apr 27, 2023

View reviewed changes

mattlord added 2 commits April 27, 2023 15:43

Remove debug logging

2023780

Signed-off-by: Matt Lord <mattalord@gmail.com>

Switch to streaming mode for large payloads

50191bb

Signed-off-by: Matt Lord <mattalord@gmail.com>

dbussink approved these changes Apr 27, 2023

View reviewed changes

mattlord commented Apr 27, 2023

View reviewed changes

shlomi-noach reviewed Apr 28, 2023

View reviewed changes

mattlord added 2 commits April 28, 2023 11:50

Add unit test

28878df

Signed-off-by: Matt Lord <mattalord@gmail.com>

Address review comments

c88fb0f

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord requested a review from ajm188 as a code owner April 28, 2023 16:12

mattlord requested a review from shlomi-noach April 28, 2023 16:13

Nitty nit

7a4f159

Signed-off-by: Matt Lord <mattalord@gmail.com>

vmg approved these changes Apr 28, 2023

View reviewed changes

Two more tiny nits

9330e98

Signed-off-by: Matt Lord <mattalord@gmail.com>

shlomi-noach approved these changes Apr 30, 2023

View reviewed changes

mattlord merged commit 0655342 into vitessio:main Apr 30, 2023
112 checks passed

mattlord deleted the vrepl_binlog_compression branch April 30, 2023 21:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VReplication: Support MySQL Binary Log Transaction Compression #12950

VReplication: Support MySQL Binary Log Transaction Compression #12950

mattlord commented Apr 21, 2023 •

edited

vitess-bot bot commented Apr 21, 2023

rohit-nayak-ps left a comment

mattlord commented Apr 26, 2023

dbussink Apr 27, 2023

dbussink Apr 27, 2023

mattlord Apr 27, 2023 •

edited

dbussink Apr 27, 2023 •

edited by mattlord

mattlord Apr 27, 2023 •

edited

dbussink Apr 28, 2023

mattlord commented Apr 27, 2023

mattlord Apr 27, 2023

shlomi-noach left a comment

shlomi-noach Apr 28, 2023

shlomi-noach Apr 28, 2023

shlomi-noach Apr 28, 2023

mattlord Apr 28, 2023

shlomi-noach Apr 28, 2023

mattlord Apr 28, 2023

mattlord Apr 28, 2023 •

edited

shlomi-noach Apr 30, 2023

shlomi-noach Apr 28, 2023

mattlord commented Apr 28, 2023

vmg left a comment

mattlord commented Apr 28, 2023

VReplication: Support MySQL Binary Log Transaction Compression #12950

VReplication: Support MySQL Binary Log Transaction Compression #12950

Conversation

mattlord commented Apr 21, 2023 • edited

Description

Related Issue(s)

Checklist

vitess-bot bot commented Apr 21, 2023

Review Checklist

General

If a new flag is being introduced:

If a workflow is added or modified:

Bug fixes

Non-trivial changes

New/Existing features

Backward compatibility

rohit-nayak-ps left a comment

Choose a reason for hiding this comment

mattlord commented Apr 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattlord Apr 27, 2023 • edited

Choose a reason for hiding this comment

dbussink Apr 27, 2023 • edited by mattlord

Choose a reason for hiding this comment

mattlord Apr 27, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattlord commented Apr 27, 2023

Choose a reason for hiding this comment

shlomi-noach left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattlord Apr 28, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattlord commented Apr 28, 2023

vmg left a comment

Choose a reason for hiding this comment

mattlord commented Apr 28, 2023

mattlord commented Apr 21, 2023 •

edited

mattlord Apr 27, 2023 •

edited

dbussink Apr 27, 2023 •

edited by mattlord

mattlord Apr 27, 2023 •

edited

mattlord Apr 28, 2023 •

edited