compress/flate: fails to decompress zlib format compressed data

[vitalyisaev2]

First I apologize in advance in case if I just misunderstood some basics of Go language. I work with a REST API that compresses HTTP responses with deflate algorithm. So I wasn't able to decompress JSON object from HTTP body with Go standard library tools. Please check the test file (actually it is a dumped HTTP body) and code below:

Input data

File with deflate encoded JSON object

$ cat test
x��PAN�0��
;��NtJ�F�ӮdU�|"oVR�C%�f������Z.�^Hs�dW뮑�'��DH�S��SFVC�����r)��G,�����<���z}�x_�g�+��2���sl�r/�Oy>��J3�\�G�9���N���#[5M�^v/�2Ҕ���|��h��[�~7�_�崛<D*����/���i

Let's make sure that this file can be decompressed and even contains valid JSON:

$ zlib-flate -uncompress < test
{"timestamp":{"tv_sec":1428488670,"tv_usec":197041},"string_timestamp":"2015-04-08 10:24:30.197041","monitor_status":"enabled","commands":{"REVERSE_LOOKUP":{"cache":{"outside":{"successes":0,"failures":0,"size":0,"time":0},"internal":{"successes":0,"failures":0,"size":0,"time":0}},"disk":{"outside":{"successes":0,"failures":0,"size":0,"time":0},"internal":{"successes":13366,"failures":0,"size":0,"time":501808}},"total":{"storage":{"successes":0,"failures":0},"proxy":{"successes":13366,"failures":0}}},"clients":{}}}
$ zlib-flate -uncompress < test | python -m json.tool
{
    "commands": {
        "REVERSE_LOOKUP": {
            "cache": {
               ....

Source code

I am going to decompress it with Go:

package main

import (
    "bytes"
    "compress/flate"
    "fmt"
    "io/ioutil"
)

func main() {
    fname := "./test"
    content, err := ioutil.ReadFile(fname)
    if err != nil {
        panic(err)
    }
    fmt.Println("File content:\n", content)

    enflated, err := ioutil.ReadAll(flate.NewReader(bytes.NewReader(content)))
    if err != nil {
        panic(err)
    }
    fmt.Println("Enflated:\n", enflated)
}

Error

$ go run uncompress.go 
File content:
 [120 156 181 80 65 78 195 48 16 252 10 242 57 69 118 226 166 38 247 156 64 42 42 130 107 100 156 165 88 196 118 149 93 35 160 234 223 89 183 61 112 42 226 192 109 118 118 102 103 180 123 65 62 0 146 13 59 209 237 5 189 15 8 78 116 74 215 70 27 211 174 100 85 184 124 34 111 86 82 171 67 37 144 102 31 183 195 15 167 168 165 90 46 164 94 72 115 165 100 87 235 174 145 215 39 189 168 68 72 209 83 154 7 22 83 70 86 67 180 207 19 140 188 114 41 4 27 71 44 225 155 254 169 223 60 244 195 221 122 125 251 120 95 24 103 221 43 20 144 50 161 31 143 16 179 115 128 8 108 225 114 47 214 79 121 62 15 232 191 224 8 74 51 6 92 213 71 130 57 218 233 175 78 182 142 30 223 254 35 91 53 77 219 94 118 47 165 50 210 148 18 148 232 124 128 31 104 183 151 91 176 126 55 167 143 207 95 3 15 229 180 155 60 68 42 159 231 241 27 47 165 167 25]
panic: flate: corrupt input before offset 5

goroutine 1 [running]:
runtime.panic(0x4a7180, 0x5)
    /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
main.main()
    /home/isaev/side-projects/elliptics-manager/uncompress.go:20 +0x2a3
exit status 2

Version

I use Ubuntu 14.10, Go 1.2.1:

$ dpkg -l | grep golang
ii  golang                                               2:1.2.1-2ubuntu1                         all          Go programming language compiler - metapackage
ii  golang-doc                                           2:1.2.1-2ubuntu1                         all          Go programming language compiler - documentation
ii  golang-go                                            2:1.2.1-2ubuntu1                         amd64        Go programming language compiler
ii  golang-go-linux-amd64                                2:1.2.1-2ubuntu1                         amd64        Go standard library compiled for linux_amd64
ii  golang-src                                           2:1.2.1-2ubuntu1                         amd64        Go programming language compiler - source files

Any help will be appreciated.

[Thomasdezeeuw]

Did you try it with go 1.4.2? It might be fixed already.

[vitalyisaev2]

@Thomasdezeeuw , I've just tried it, looks like the same problem:

$ docker run -it --rm -v $(pwd):/root/test -w /root/test golang:1.4.2 bash
root@86a293d14e46:~/test# go version
go version go1.4.2 linux/amd64
root@86a293d14e46:~/test# go run uncompress.go 
File content:
 [120 156 181 80 65 78 195 48 16 252 10 242 57 69 118 226 166 38 247 156 64 42 42 130 107 100 156 165 88 196 118 149 93 35 160 234 223 89 183 61 112 42 226 192 109 118 118 102 103 180 123 65 62 0 146 13 59 209 237 5 189 15 8 78 116 74 215 70 27 211 174 100 85 184 124 34 111 86 82 171 67 37 144 102 31 183 195 15 167 168 165 90 46 164 94 72 115 165 100 87 235 174 145 215 39 189 168 68 72 209 83 154 7 22 83 70 86 67 180 207 19 140 188 114 41 4 27 71 44 225 155 254 169 223 60 244 195 221 122 125 251 120 95 24 103 221 43 20 144 50 161 31 143 16 179 115 128 8 108 225 114 47 214 79 121 62 15 232 191 224 8 74 51 6 92 213 71 130 57 218 233 175 78 182 142 30 223 254 35 91 53 77 219 94 118 47 165 50 210 148 18 148 232 124 128 31 104 183 151 91 176 126 55 167 143 207 95 3 15 229 180 155 60 68 42 159 231 241 27 47 165 167 25]
panic: flate: corrupt input before offset 5

goroutine 1 [running]:
main.main()
    /root/test/uncompress.go:20 +0x379

goroutine 2 [runnable]:
runtime.forcegchelper()
    /usr/src/go/src/runtime/proc.go:90
runtime.goexit()
    /usr/src/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 3 [runnable]:
runtime.bgsweep()
    /usr/src/go/src/runtime/mgc0.go:82
runtime.goexit()
    /usr/src/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 4 [runnable]:
runtime.runfinq()
    /usr/src/go/src/runtime/malloc.go:712
runtime.goexit()
    /usr/src/go/src/runtime/asm_amd64.s:2232 +0x1
exit status 2

[mikioh]

You need to use http://tools.ietf.org/html/rfc1950 instead of http://tools.ietf.org/html/rfc1951.
http://godoc.org/compress/zlib implements RFC 1950, and http://godoc.org/compress/flate implements RFC 1951.