Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dumpling: compress supports snappy/zstd #38910

Merged
merged 28 commits into from Nov 14, 2022
Merged

dumpling: compress supports snappy/zstd #38910

Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
88a8841
dumpling: compress supports snappy/zstd
lyzx2001 Nov 4, 2022
d3443f1
solve conflicts
lyzx2001 Nov 4, 2022
4fbb17c
dumpling: compress supports snappy/zstd
lyzx2001 Nov 4, 2022
8d15c12
dumpling: compress supports snappy/zstd
lyzx2001 Nov 4, 2022
cff6ee6
dumpling: compress supports snappy/zstd
lyzx2001 Nov 4, 2022
f5e11ac
dumpling: compress supports snappy/zstd
lyzx2001 Nov 4, 2022
cfa6563
dumpling: compress supports snappy/zstd
lyzx2001 Nov 4, 2022
5256527
dumpling: compress supports snappy/zstd
lyzx2001 Nov 4, 2022
628f704
modify go.mod and go.sum
lyzx2001 Nov 4, 2022
fe1e21d
modify install.sh
lyzx2001 Nov 4, 2022
2626711
modify install.sh
lyzx2001 Nov 4, 2022
12c54b8
modify install.sh
lyzx2001 Nov 4, 2022
7a81d30
modify install.sh
lyzx2001 Nov 4, 2022
8c158cc
modify install.sh
lyzx2001 Nov 4, 2022
fc5281c
modify install.sh
lyzx2001 Nov 4, 2022
b067c88
modify install.sh
lyzx2001 Nov 4, 2022
48fa4db
modify install.sh
lyzx2001 Nov 9, 2022
808277c
dumpling: compress supports snappy/zstd
lyzx2001 Nov 10, 2022
d2b7560
Merge branch 'master' into dumpling-compress
lyzx2001 Nov 10, 2022
a6accde
improve backward compatibility of 'SHOW SLAVE HOSTS' command
lyzx2001 Nov 10, 2022
40e0584
dumpling: compress supports snappy/zstd
lyzx2001 Nov 10, 2022
5c7072e
dumpling: compress supports snappy/zstd
lyzx2001 Nov 10, 2022
f303c95
dumpling: compress supports snappy/zstd
lyzx2001 Nov 10, 2022
bd2de30
Merge branch 'master' into dumpling-compress
lyzx2001 Nov 11, 2022
556bb36
dumpling: compress supports snappy/zstd
lyzx2001 Nov 11, 2022
bdc2814
dumpling: compress supports snappy/zstd
lyzx2001 Nov 11, 2022
5ba6b50
Merge branch 'master' into dumpling-compress
lichunzhu Nov 14, 2022
f43b8af
Merge branch 'master' into dumpling-compress
ti-chi-bot Nov 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 5 additions & 1 deletion dumpling/export/config.go
View file
Open in desktop
Expand Up @@ -291,7 +291,7 @@ func (*Config) DefineFlags(flags *pflag.FlagSet) {
_ = flags.MarkHidden(flagReadTimeout)
flags.Bool(flagTransactionalConsistency, true, "Only support transactional consistency")
_ = flags.MarkHidden(flagTransactionalConsistency)
flags.StringP(flagCompress, "c", "", "Compress output file type, support 'gzip', 'no-compression' now")
flags.StringP(flagCompress, "c", "", "Compress output file type, support 'gzip', 'snappy', 'zstd', 'no-compression' now")
}

// ParseFromFlags parses dumpling's export.Config from flags
Expand Down Expand Up @@ -596,6 +596,10 @@ func ParseCompressType(compressType string) (storage.CompressType, error) {
return storage.NoCompression, nil
case "gzip", "gz":
return storage.Gzip, nil
case "snappy":
return storage.Snappy, nil
case "zstd", "zst":
return storage.Zstd, nil
default:
return storage.NoCompression, errors.Errorf("unknown compress type %s", compressType)
}
Expand Down
4 changes: 4 additions & 0 deletions dumpling/export/writer_util.go
View file
Open in desktop
Expand Up @@ -591,6 +591,10 @@ func compressFileSuffix(compressType storage.CompressType) string {
return ""
case storage.Gzip:
return ".gz"
case storage.Snappy:
return ".snappy"
case storage.Zstd:
return ".zst"
default:
return ""
}
Expand Down
8 changes: 8 additions & 0 deletions dumpling/install.sh
View file
Open in desktop
Expand Up @@ -15,3 +15,11 @@ chmod a+x bin/minio

wget https://dl.minio.io/client/mc/release/linux-amd64/mc -O bin/mc
chmod a+x bin/mc

go get github.com/ma6174/snappy
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
go get github.com/ma6174/snappy

only need go install?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error encountered in CI integration test

https://github.com/pingcap/tidb/actions/runs/3444134903/jobs/5746413416#step:6:7138

i think maybe should add back go get github.com/ma6174/snappy?

go install github.com/ma6174/snappy

wget https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-1.5.2.tar.gz
tar xvfz zstd-1.5.2.tar.gz
cd zstd-1.5.2
make
65 changes: 51 additions & 14 deletions dumpling/tests/s3/run.sh
View file
Open in desktop
Expand Up @@ -80,21 +80,58 @@ echo "1st records count is ${cnt}"
# run dumpling with compress option
export DUMPLING_OUTPUT_DIR=s3://mybucket/dump-compress

run_dumpling --s3.endpoint="http://$S3_ENDPOINT/" --compress "gzip"
for compressType in "gzip" "snappy" "zst"
do
echo "start ${compressType} test"
run_dumpling --s3.endpoint="http://$S3_ENDPOINT/" --compress $compressType

mkdir -p "${HOST_DIR}/compress"
mkdir -p "${HOST_DIR}/compress"

bin/mc cp minio/mybucket/dump-compress/s3-schema-create.sql.gz "${HOST_DIR}/compress/s3-schema-create.sql.gz"
bin/mc cp minio/mybucket/dump-compress/s3.t-schema.sql.gz "${HOST_DIR}/compress/s3.t-schema.sql.gz"
bin/mc cp minio/mybucket/dump-compress/s3.t.000000000.sql.gz "${HOST_DIR}/compress/s3.t.000000000.sql.gz"

gzip "${HOST_DIR}/compress/s3-schema-create.sql.gz" -d
diff "${HOST_DIR}/local/s3-schema-create.sql" "${HOST_DIR}/compress/s3-schema-create.sql"

gzip "${HOST_DIR}/compress/s3.t-schema.sql.gz" -d
diff "${HOST_DIR}/local/s3.t-schema.sql" "${HOST_DIR}/compress/s3.t-schema.sql"

gzip "${HOST_DIR}/compress/s3.t.000000000.sql.gz" -d
diff "${HOST_DIR}/local/s3.t.000000000.sql" "${HOST_DIR}/compress/s3.t.000000000.sql"
if [ "$compressType" = "gzip" ]; then
compressFormat="gz"
else
compressFormat=$compressType
fi
bin/mc cp minio/mybucket/dump-compress/s3-schema-create.sql.${compressFormat} "${HOST_DIR}/compress/s3-schema-create.sql.${compressFormat}"
bin/mc cp minio/mybucket/dump-compress/s3.t-schema.sql.${compressFormat} "${HOST_DIR}/compress/s3.t-schema.sql.${compressFormat}"
bin/mc cp minio/mybucket/dump-compress/s3.t.000000000.sql.${compressFormat} "${HOST_DIR}/compress/s3.t.000000000.sql.${compressFormat}"

case $compressType in
"gzip")
gzip "${HOST_DIR}/compress/s3-schema-create.sql.gz" -d
Copy link
Contributor

@lance6716 lance6716 Nov 11, 2022 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems we can assign gzip, snappy, zst to $binary, and also assign $suffix, then use

    ${binary} "${HOST_DIR}/compress/s3-schema-create.sql.${suffix}" -d
    diff "${HOST_DIR}/local/s3-schema-create.sql" "${HOST_DIR}/compress/s3-schema-create.sql"

    ${binary} "${HOST_DIR}/compress/s3.t-schema.sql.${suffix}" -d
    diff "${HOST_DIR}/local/s3.t-schema.sql" "${HOST_DIR}/compress/s3.t-schema.sql"

    ${binary} "${HOST_DIR}/compress/s3.t.000000000.sql.${suffix}" -d
    diff "${HOST_DIR}/local/s3.t.000000000.sql" "${HOST_DIR}/compress/s3.t.000000000.sql"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

modified

diff "${HOST_DIR}/local/s3-schema-create.sql" "${HOST_DIR}/compress/s3-schema-create.sql"

gzip "${HOST_DIR}/compress/s3.t-schema.sql.gz" -d
diff "${HOST_DIR}/local/s3.t-schema.sql" "${HOST_DIR}/compress/s3.t-schema.sql"

gzip "${HOST_DIR}/compress/s3.t.000000000.sql.gz" -d
diff "${HOST_DIR}/local/s3.t.000000000.sql" "${HOST_DIR}/compress/s3.t.000000000.sql"
;;
"snappy")
snappy -d "${HOST_DIR}/compress/s3-schema-create.sql.snappy"
diff "${HOST_DIR}/local/s3-schema-create.sql" "${HOST_DIR}/compress/s3-schema-create.sql"

snappy -d "${HOST_DIR}/compress/s3.t-schema.sql.snappy"
diff "${HOST_DIR}/local/s3.t-schema.sql" "${HOST_DIR}/compress/s3.t-schema.sql"

snappy -d "${HOST_DIR}/compress/s3.t.000000000.sql.snappy"
diff "${HOST_DIR}/local/s3.t.000000000.sql" "${HOST_DIR}/compress/s3.t.000000000.sql"
;;
"zst")
zstd "${HOST_DIR}/compress/s3-schema-create.sql.zst" -d
diff "${HOST_DIR}/local/s3-schema-create.sql" "${HOST_DIR}/compress/s3-schema-create.sql"

zstd "${HOST_DIR}/compress/s3.t-schema.sql.zst" -d
diff "${HOST_DIR}/local/s3.t-schema.sql" "${HOST_DIR}/compress/s3.t-schema.sql"

zstd "${HOST_DIR}/compress/s3.t.000000000.sql.zst" -d
diff "${HOST_DIR}/local/s3.t.000000000.sql" "${HOST_DIR}/compress/s3.t.000000000.sql"
;;
esac

rm "${HOST_DIR}/compress/s3-schema-create.sql"
rm "${HOST_DIR}/compress/s3.t-schema.sql"
rm "${HOST_DIR}/compress/s3.t.000000000.sql"
done

run_sql "drop database if exists \`$DB_NAME\`;"