Feature request: Use zstandard compression instead of zip for gitea dump?

[symgryph]

gitea dump is cool, but would be way cooler if we could use zstandard for compression instead of zip.....

[dsseng]

I can try myself on this. Found a good CGO-free implementation: https://pkg.go.dev/github.com/klauspost/compress/zstd . Does anyone mind depending on this? Another option is using system archiver on *nix.

[clavinet]

I just made a backup of a large instance and would welcome such a feature.

Zstandard would already be faster, but when using it multithreaded (zstdmt), it could go even faster.
Single-threaded zip/gzip only wastes time these days when we have faster methods readily available.

[tepozoa]

(24 Jul 2022) The package already being used, https://github.com/mholt/archiver/v3, (now) supports Zstd. I had a look to dump.go and I think it might just now be a matter of making sure the import is up to date with the latest stable version and adding the tar.zstd extension to the outputTypeEnum?

https://pkg.go.dev/github.com/mholt/archiver/v3#Zstd

[wULLSnpAXbWZGYDYyhWTKKspEQoaYxXyhoisqHf]

is it just me or are the resulting archives twice the size of .zip?

I'm a fan of zstd, I am aware of what it can do but if this Gitea setting is using zstd's default compression level (-3 out of 1-19), that might explain the archive size.

perhaps exposing some knobs in the future could cater to users wishing to tweak at least the compression levels of zstd...

[symgryph]

You can actually specify the level for z standard to use via a environment variable. Look in the reference manuals. I was able to do this with the tar version via a environment variable. Sincerely Thomas Munn

…

________________________________ From: wULLSnpAXbWZGYDYyhWTKKspEQoaYxXyhoisqHf ***@***.***> Sent: Wednesday, August 10, 2022 7:24:07 AM To: go-gitea/gitea ***@***.***> Cc: Thomas ***@***.***>; Author ***@***.***> Subject: Re: [go-gitea/gitea] Feature request: Use zstandard compression instead of zip for gitea dump? (#14290) is it just me or are the resulting archives twice the size of .zip? I'm a fan of zstd, I am aware of what it can do but if this Gitea setting is using zstd's default compression level (-3 out of 1-19), that might explain the archive size. perhaps exposing some knobs in the future could cater to users wishing to tweak at least the compression levels of zstd... — Reply to this email directly, view it on GitHub<#14290 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABCSE3RUPF4NBHRKIZ225ELVYOGNPANCNFSM4V4DKZAQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[wULLSnpAXbWZGYDYyhWTKKspEQoaYxXyhoisqHf]

You can actually specify the level for z standard to use via a environment variable. Look in the reference manuals. I was able to do this with the tar version via a environment variable. Sincerely Thomas Munn
…
________________________________ From: wULLSnpAXbWZGYDYyhWTKKspEQoaYxXyhoisqHf @.> Sent: Wednesday, August 10, 2022 7:24:07 AM To: go-gitea/gitea @.> Cc: Thomas @.>; Author @.> Subject: Re: [go-gitea/gitea] Feature request: Use zstandard compression instead of zip for gitea dump? (#14290) is it just me or are the resulting archives twice the size of .zip? I'm a fan of zstd, I am aware of what it can do but if this Gitea setting is using zstd's default compression level (-3 out of 1-19), that might explain the archive size. perhaps exposing some knobs in the future could cater to users wishing to tweak at least the compression levels of zstd... — Reply to this email directly, view it on GitHub<#14290 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABCSE3RUPF4NBHRKIZ225ELVYOGNPANCNFSM4V4DKZAQ. You are receiving this because you authored the thread.Message ID: @.***>

thanks for the reply, did you mean somehow using ZSTD_CLEVEL in the process? could you by any chance provide an example command (e.g. gitea dump -c <config> --type <whattype-you-mentioned-tar>?
I know you can do tar <opts> --zstd or even tar <opts> -I"zstd <zstdopts>", but I don't see how that could be used with gitea directly.

or did you mean that you were able to make gitea create a tar and then zstd-compress it?

apologies, it just wasn't clear to me from your answer. @symgryph

[symgryph]

Fish format set -gx ZSTD_CLEVEL 19 run your command. The set command is fish shell specific, as I ain't doin no bash! Sincerely Thomas Munn

…

________________________________ From: wULLSnpAXbWZGYDYyhWTKKspEQoaYxXyhoisqHf ***@***.***> Sent: Tuesday, October 11, 2022 1:26:36 PM To: go-gitea/gitea ***@***.***> Cc: Thomas ***@***.***>; Mention ***@***.***> Subject: Re: [go-gitea/gitea] Feature request: Use zstandard compression instead of zip for gitea dump? (#14290) You can actually specify the level for z standard to use via a environment variable. Look in the reference manuals. I was able to do this with the tar version via a environment variable. Sincerely Thomas Munn …

________________________________ From: wULLSnpAXbWZGYDYyhWTKKspEQoaYxXyhoisqHf @.> Sent: Wednesday, August 10, 2022 7:24:07 AM To: go-gitea/gitea @.> Cc: Thomas @.>; Author @.> Subject: Re: [go-gitea/gitea] Feature request: Use zstandard compression instead of zip for gitea dump? (#14290<#14290>) is it just me or are the resulting archives twice the size of .zip? I'm a fan of zstd, I am aware of what it can do but if this Gitea setting is using zstd's default compression level (-3 out of 1-19), that might explain the archive size. perhaps exposing some knobs in the future could cater to users wishing to tweak at least the compression levels of zstd... — Reply to this email directly, view it on GitHub<#14290 (comment)<#14290 (comment)>>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABCSE3RUPF4NBHRKIZ225ELVYOGNPANCNFSM4V4DKZAQ. You are receiving this because you authored the thread.Message ID: @.***> thanks for the reply, did you mean somehow using ZSTD_CLEVEL in the process? could you by any chance provide an example command (e.g. gitea dump -c <config> --type <whattype-you-mentioned-tar>? I know you can do tar <opts> --zstd or even tar <opts> -I"zstd <zstdopts>", but I don't see how that could be used with gitea directly. or did you mean that you were able to make gitea create a tar and then zstd-compress it? apologies, it just wasn't clear to me from your answer. @symgryph<https://github.com/symgryph> — Reply to this email directly, view it on GitHub<#14290 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABCSE3XMQWDSYSSPMLK67FTWCWPMZANCNFSM4V4DKZAQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[wULLSnpAXbWZGYDYyhWTKKspEQoaYxXyhoisqHf]

Fish format set -gx ZSTD_CLEVEL 19
run your command. The set command is fish shell specific, as I ain't doin no bash!

Sincerely

Thomas Munn

should be sth like export in bash, or running ENVVAR <command> directly.. will try it out, thanks.