Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release tarballs should be compressed with xz #9267

Closed
thiagomacieira opened this issue Jun 22, 2018 · 7 comments · Fixed by #13700
Assignees

Comments

@thiagomacieira
Copy link

@thiagomacieira thiagomacieira commented Jun 22, 2018

Mono should release its sources in a .tar.xz format. Quick benchmark for the 5.12.0 release file:

Compression Format Tarball size Decompression time
BZip2 204 MB 31.04s
Xz 132 MB 9.89s
@akoeplinger

This comment has been minimized.

Copy link
Member

@akoeplinger akoeplinger commented Jun 22, 2018

@directhex what do you think about this?

@directhex

This comment has been minimized.

Copy link
Contributor

@directhex directhex commented Jun 22, 2018

It's a 1-line change to configure.ac, plus however many hours to fix assumptions made in our relevant Jenkins jobs

@ghost

This comment has been minimized.

Copy link

@ghost ghost commented Jun 24, 2018

curl, openssh, git etc. provide both .tar.gz and .tar.xz (maybe because some systems don't support latter), could mono provide both as well?

@directhex

This comment has been minimized.

Copy link
Contributor

@directhex directhex commented Jul 6, 2018

I'm not terribly keen on supporting both.

Are there any distributions which can't handle .tar.xz? We only care about tarballs on Linux & friends, so if there's a distribution adjacent to our support matrix which can't handle .xz then I could be persuaded to do both - otherwise I'd prefer to just switch to .xz wholesale.

Any Debian derivative with dpkg 1.15.5 (Nov 2009) is fine with it. Not sure on RPM

@thiagomacieira

This comment has been minimized.

Copy link
Author

@thiagomacieira thiagomacieira commented Jul 6, 2018

There's a lot of software that has switched to only .xz (and even .7z for Windows), ditching the .gz and especially ditching the .bz2 files. Unlike the Lempel-Ziv family of decompressors, bunzip2 takes a lot of CPU time to work. It's not only more bandwidth consumed for the project, it's also time for the user. It's so bad that decompressing the 5.12.0 release tarball on-the-fly while downloading stalls the download (averaged 5 MB/s while a pure download saturated my 100 Mbit/s office connection).

If you want to hedge your bets, add .xz and keep the current compression (bzip2) for another year, then check your download stats.

And if you want to be bold, try zstd.

Tarball Size Decompression time
141 MB 3.047s

Just a pure xxxcat of the compressed tarball:

Decompressor xxxxcat time
bunzip2 35.86 s
xzcat 10.86 s
zstdcat 1.23 s
@akoeplinger

This comment has been minimized.

Copy link
Member

@akoeplinger akoeplinger commented Jul 6, 2018

I've seen build scripts that do stuff like wget https://download.mono-project.com/sources/mono/mono-$(VERSION).tar.bz2 so I'd like to avoid breaking them if we can avoid it.

@directhex is there a reason why we can't easily do both?

@directhex

This comment has been minimized.

Copy link
Contributor

@directhex directhex commented Jul 8, 2018

The algo used is a consequence of this configure.ac setting: https://github.com/mono/mono/blob/master/configure.ac#L16

I don't think multiple are supported, so we'd be hand-rolling an "uncompress and recompress" addition, which sorta sucks

directhex added a commit to directhex/mono that referenced this issue Mar 27, 2019
monojenkins added a commit to monojenkins/mono that referenced this issue Mar 28, 2019
@marek-safar marek-safar added the task label Apr 2, 2019
@marek-safar marek-safar added this to Backlog in Short Term Projects via automation Apr 2, 2019
@marek-safar marek-safar moved this from Backlog to Completed Tasks in this Sprint in Short Term Projects Apr 2, 2019
@marek-safar marek-safar moved this from Completed Tasks in this Sprint to Done in Short Term Projects Apr 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
4 participants
You can’t perform that action at this time.