Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use multi-threaded compression and support more archive formats #79

Merged
merged 10 commits into from Sep 20, 2022

Conversation

felipecruz91
Copy link
Collaborator

@felipecruz91 felipecruz91 commented Sep 16, 2022

This PR:

  • Adds support for more archive formats: zstd (.tar.zst) and bzip2 (.tar.bz2).
  • The default archive format when exporting has changed from gzip (.tar.gz) to zstd (.tar.zst). Also, the compression implementation for gzip has changed to pigz (parallel implementation of gzip) which is faster.

I’ve run some benchmarks in GHA (2-cpu VM) with a volume that contains 4GB of Postgres data:

goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz

Regarding time:

name                                                                          time/op
ExportVolume/compression_docker.io/felipecruz/postgres_pgdata_4gb_.tar.gz-2    122s ± 0%
ExportVolume/compression_docker.io/felipecruz/postgres_pgdata_4gb_.tar.zst-2  46.9s ± 0%
ExportVolume/compression_docker.io/felipecruz/postgres_pgdata_4gb_.tar.bz2-2   429s ± 0%

Regarding compressed size:

2f91f352f0ba381893b9e15ea87db0e28a88aa6e28070c07892681d7a0d6ba6b.tar.gz - Size: 1690867939 bytes (1.69 GB)
2f91f352f0ba381893b9e15ea87db0e28a88aa6e28070c07892681d7a0d6ba6b.tar.zst - Size: 1593686093 bytes (1.59 GB)
2f91f352f0ba381893b9e15ea87db0e28a88aa6e28070c07892681d7a0d6ba6b.tar.bz2 - Size: 1328059635 bytes (1.32 GB)

@github-actions
Copy link

github-actions bot commented Sep 16, 2022

Docker image tag(s) pushed:

docker/volumes-backup-extension:pr-79

Labels added to images:

org.opencontainers.image.title=volumes-backup-extension
org.opencontainers.image.description=Back up, clone, restore, and share Docker volumes effortlessly.
org.opencontainers.image.url=https://github.com/docker/volumes-backup-extension
org.opencontainers.image.source=https://github.com/docker/volumes-backup-extension
org.opencontainers.image.version=pr-79
org.opencontainers.image.created=2022-09-19T14:32:40.581Z
org.opencontainers.image.revision=46d0f42f8399b9669bc82450009f27db8b10b0b0
org.opencontainers.image.licenses=Apache-2.0

@atomist
Copy link
Contributor

atomist bot commented Sep 16, 2022

docker/volumes-backup-extension

docker/volumes-backup-extension:pr-79

Vulnerabilities
Comparison

👏 No new critical or high vulnerabilities compared with target branch main


docker/volumes-backup-extension

docker/volumes-backup-extension:pr-79

Vulnerabilities
Comparison

👏 No new critical or high vulnerabilities compared with target branch main


More details are available in the vulnerability report

@atomist
Copy link
Contributor

atomist bot commented Sep 16, 2022

Dockerfile

docker/volumes-backup-extension:pr-79

Vulnerabilities
Comparison

👏 No new critical or high vulnerabilities compared with target branch main


More details are available in the vulnerability report

@felipecruz91 felipecruz91 force-pushed the feature/support-multithreaded-compressions branch 2 times, most recently from 8b268fb to fea6c8f Compare September 18, 2022 14:28
@felipecruz91 felipecruz91 force-pushed the feature/support-multithreaded-compressions branch from fea6c8f to f6a1e51 Compare September 18, 2022 14:43
@felipecruz91 felipecruz91 changed the title Use zstd as compression algorithm Use multi-threaded compression and support more archive formats Sep 19, 2022
@felipecruz91 felipecruz91 marked this pull request as ready for review September 19, 2022 10:02
@gtardif
Copy link
Contributor

gtardif commented Sep 19, 2022

On M1:

name                                                                           time/op
ExportVolume/compression_docker.io/felipecruz/postgres_pgdata_4gb_.tar.gz-10   203s ± 0%
ExportVolume/compression_docker.io/felipecruz/postgres_pgdata_4gb_.tar.zst-10  139s ± 0%
ExportVolume/compression_docker.io/felipecruz/postgres_pgdata_4gb_.tar.bz2-10  925s ± 0%

Comparing clicking "export" on the same 4gb volume, in the extension UI:

  • Published version : 10mins57
  • This PR version: 3mins43

Copy link
Contributor

@gtardif gtardif left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: maybe set the default export name to be .tar.zst by default, as the performance gain is huge. LGTM

@felipecruz91 felipecruz91 force-pushed the feature/support-multithreaded-compressions branch from ec76dfe to f28c7b0 Compare September 19, 2022 14:30
@felipecruz91 felipecruz91 merged commit accf16b into main Sep 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants