Consider providing lzma tarballs?

[QuLogic]

I did a really simple test of unzipping and then re-compressing the directory with a gzipped tarball (tar zc) and an LZMA-compressed tarball (tar Jc):

-rw-r--r--. 1.2M Feb 23 03:18 110m_cultural.tar.gz
-rw-r--r--. 334K Feb 23 03:18 110m_cultural.tar.xz
-rw-r--r--. 1.2M Oct 27 13:58 110m_cultural.zip
-rw-r--r--. 4.0M Feb 23 03:18 110m_physical.tar.gz
-rw-r--r--. 1.6M Feb 23 03:18 110m_physical.tar.xz
-rw-r--r--. 4.0M Feb 21 19:28 110m_physical.zip
-rw-r--r--. 7.4M Feb 23 03:18 50m_cultural.tar.gz
-rw-r--r--. 2.5M Feb 23 03:18 50m_cultural.tar.xz
-rw-r--r--. 7.5M Oct 27 13:58 50m_cultural.zip
-rw-r--r--. 7.3M Feb 23 03:18 50m_physical.tar.gz
-rw-r--r--. 3.7M Feb 23 03:18 50m_physical.tar.xz
-rw-r--r--. 7.3M Oct 27 13:58 50m_physical.zip
-rw-r--r--. 202M Feb 23 03:19 10m_cultural.tar.gz
-rw-r--r--. 127M Feb 23 03:22 10m_cultural.tar.xz
-rw-r--r--. 202M Oct 27 13:58 10m_cultural.zip
-rw-r--r--.  54M Feb 23 03:18 10m_physical.tar.gz
-rw-r--r--.  35M Feb 23 03:19 10m_physical.tar.xz
-rw-r--r--.  54M Oct 27 13:58 10m_physical.zip

The gzip file doesn't really provide any advantage, but the xz provides a significant savings, shaving off between 70% (for the small files) and 40% (for the larger files).

It is a bit slower though (/tmp is tmpfs and in memory):

$ time unzip -q 10m_cultural.zip -d /tmp/asdfg
real	0m3.075s
user	0m2.931s
sys	0m0.130s

$ time tar xf 10m_cultural.tar.xz -C /tmp/asdfgh
real	0m9.101s
user	0m9.012s
sys	0m0.624s

[pnorman]

The gzip file doesn't really provide any advantage

Does this change with -9 compression? Because these are compressed once and decompressed many times, it makes sense to optimize for size, not compression time.

[nvkelso]

If there's a better ZIP setting I'm all ears. Otherwise I think these files are all relatively small so don't see a big reason to add tar/gzip (we can't remove ZIP).

…

On Fri, May 18, 2018 at 11:41 AM, Paul Norman ***@***.***> wrote: The gzip file doesn't really provide any advantage Does this change with -9 compression? Because these are compressed once and decompressed many times, it makes sense to optimize for size, not compression time. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#248 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0EO_1RFZ6ADWx5DEIeo00EE738DDYdks5tzxXUgaJpZM4SQi7k> .

[QuLogic]

The gzip file doesn't really provide any advantage

Does this change with -9 compression? Because these are compressed once and decompressed many times, it makes sense to optimize for size, not compression time.

No, -9 compression saves like 0.5% over default gzip.

Otherwise I think these files are all relatively small so don't see a big reason to add tar/gzip

Sure, pipes are bigger now, but I don't know that saving 100M is not insignificant.

[nvkelso]

Which files are you regularly downloading from the site that are that large?

…

On May 19, 2018, at 13:35, Elliott Sales de Andrade ***@***.***> wrote: The gzip file doesn't really provide any advantage Does this change with -9 compression? Because these are compressed once and decompressed many times, it makes sense to optimize for size, not compression time. No, -9 compression saves like 0.5% over default gzip. Otherwise I think these files are all relatively small so don't see a big reason to add tar/gzip Sure, pipes are bigger now, but I don't know that saving 100M is not insignificant. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[QuLogic]

Well, not really anything on the regular. But for example, Cartopy has automatic downloaders for these files. It does default to 110m and has a cache though. So I'm speaking more as to the aggregate savings on the server side rather than the client (although I would appreciate the savings myself).