server: gzip encoding of tiles

[mojodna]

When requesting tiles with an Accept-Encoding: gzip header, I would expect Tegola to respond with gzipped tiles (and a Content-Encoding: gzip header) to reduce the amount of data transferred.

However, they don't appear to be gzipped:

$ curl -H "Accept-Encoding: gzip" -O http://localhost:8080/maps/borders/11/328/792.pbf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9785    0  9785    0     0  10526      0 --:--:-- --:--:-- --:--:-- 10521
$ file 792.pbf
792.pbf: data

Is the intent for Tegola to run behind a reverse proxy that handles gzipping?

[ARolek]

@mojodna the pbf files are already zig zag encoded so I don't know how much of a win gzipping will add, but we can easily add support for handling the Accept-Encoding header appropriately. Have you tried gzipping a pbf tile from tegola by chance?

[mojodna]

Gzipping makes a pretty big difference. This tile went from 512K to 125K:

5_4_11.pbf.gz

[ARolek]

Well that's pretty significant. I will work on getting it implemented.

[pnorman]

I wrote this up as a new issue, but found this issue already about it.

When MVTs were designed, gzipping was intended. This allows use of gzip's property that gzip(file 1 + file 2) = gzip(file 1) + gzip(file 2), and the ability to concatanate multiple layers with MVTs.

Mapbox's implementations of consumers and servers assumed that browsers would always request compressed data, server-side software would always process compressed data, and servers would always send compressed data. This assumption was so strong their servers sent compressed data to clients which explicitly said only identity compression was acceptable.

Tegola should

1. Compress data before saving it to the cache. The compression level should be adjustable.

2. Send the compressed data directly from cache to clients that send an Accept-Encoding accepting gzip, *, or no Accept-Encoding header. Content-Encoding: gzip should also be set.

3. For Accept-Encoding headers which have gzip;q=0 or *;q=0 without gzip appearing elsewhere or an Accept-Encoding header that is empty, either decompress on the fly to send an identity encoded response, with no Accept-Encoding; or return a 406 error. If on-demand decompression is allowed, there should be a way to disable this in configuration or an environment variable for those who don't want to support it.

I'm not sure what this means for setting the Vary header.

---

Looking at the tiles I'm generating, this would reduce the size from 36MB to 7.6MB of what I get with tegola cache seed --map bolder --min-zoom 0 --max-zoom 14 --bounds "-123.41,49.03,-122.47,49.83". It takes about 4m to generate the tiles, and under 1.2s to compress them with gzip -k -3 -S .3.gz -r. The on-disk savings are less significant because of minimum block sizes, but the data transferred to the