Skip to content

Pipe: RPC compression for data sync connectors (SNAPPY, GZIP, LZ4, ZSTD and LZMA2)#12519

Merged
SteveYurongSu merged 15 commits intoapache:masterfrom
DanielWang2035:pipe-rpc-compress
May 15, 2024
Merged

Pipe: RPC compression for data sync connectors (SNAPPY, GZIP, LZ4, ZSTD and LZMA2)#12519
SteveYurongSu merged 15 commits intoapache:masterfrom
DanielWang2035:pipe-rpc-compress

Conversation

@DanielWang2035
Copy link
Collaborator

@DanielWang2035 DanielWang2035 commented May 13, 2024

SNAPPY, GZIP, LZ4, ZSTD and LZMA2 compression are supported, other compression/decompression algorithm can be added later.

TODO:

  • support air-gap.

This PR has:

  • been self-reviewed.

Usage:

CREATE PIPE TEST1
WITH ("sink.compressor"="SNAPPY");

CREATE PIPE TEST1
WITH ("sink.compressor"="SNAPPY, SNAPPY, GZIP");

CREATE PIPE TEST1
WITH ("sink.compressor"="SNAPPY, SNAPPY, SNAPPY");

@SteveYurongSu SteveYurongSu self-requested a review May 13, 2024 11:45
@SteveYurongSu SteveYurongSu self-assigned this May 13, 2024
# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/thrift/async/handler/PipeTransferTabletBatchEventHandler.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/thrift/sync/IoTDBDataRegionSyncConnector.java
@SteveYurongSu SteveYurongSu changed the title Pipe: RPC compression Pipe: RPC compression for data sync connectors (SNAPPY and LZMA2) May 15, 2024
Copy link
Member

@SteveYurongSu SteveYurongSu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@DanielWang2035 DanielWang2035 changed the title Pipe: RPC compression for data sync connectors (SNAPPY and LZMA2) Pipe: RPC compression for data sync connectors (SNAPPY, GZIP, LZ4, ZSTD and LZMA2) May 15, 2024
Copy link
Member

@SteveYurongSu SteveYurongSu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done!

@SteveYurongSu SteveYurongSu merged commit e3aa4d8 into apache:master May 15, 2024
SzyWilliam pushed a commit to SzyWilliam/iotdb that referenced this pull request Nov 26, 2024
…TD and LZMA2) (apache#12519)

Co-authored-by: Steve Yurong Su <rong@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants