Binlog parser with high cpu usage

[hjweddie]

I have a binlog analyzer process for MySQL 8.0 with zstd compressed binlog. My analyzer suffered from much heavy cpu usage. After I got pprof graph, I can find that zstd.NewReader created much gc load.
My process had many network IO bound jobs with GOMAXPROCS setted to 198. I think zstd.WithDecoderConcurrency(0) may lead to call zstd.newBlockDec and zstd.newFrameDec many times (198) for every transaction_payload_event with heavy gc load.

code

My question is could we add functions for user to offers value for zstd.WithDecoderConcurrency (for TransactionPayloadEvent or BinlogSyncer ?) ? Or do you have any better solution/advice ?

[lance6716]

In your CPU profile I think runtime GC costs a lot. Maybe there are many objects to be checked when GC. I recommend you modify your local code with zstd.WithDecoderConcurrency and check if it work first.

[hjweddie]

I try to modify local code, from zstd.NewReader(nil, zstd.WithDecoderConcurrency(0)) to zstd.NewReader(nil, zstd.WithDecoderConcurrency(runtime.NumCPU())), with runtime.NumCPU() == 4.
After updated after 10:35, my process cpu usage decrease 50+%，while binlog throughput does not decrease

[lance6716]

👍 Can you open a PR to add a configuration item in BinlogSyncerConfig?

[hjweddie]

@lance6716 PR

[dveeden]

I think we call zstd.NewReader() for every TransactionPayloadEvent payload. Resetting the decoder instead creating a new one might help with performance and resource usage.

[hjweddie]

Sorry for previous pr, I missed calling set function for concurrentcy configuration
I create a new PR to fix it

PR

[hjweddie]

I think we call zstd.NewReader() for every TransactionPayloadEvent payload. Resetting the decoder instead creating a new one might help with performance and resource usage.

I agree with you. I think we can add a *zstd.Decoder in BinlogParser, and pass it to TransactionPayloadEvent