Content-Length issue when using compression

[swill]

I have the following code:

f, err := os.Create(fmt.Sprintf("downloads/%s", obj_path))
if err != nil {
    return err
}
defer f.Close()

h, err := conn.ObjectGet(*bucket, obj_path, f, true, nil)
if err != nil {
    fmt.Printf("\nERROR: Problem downloading %s\n", obj_path)
    fmt.Println(h)
    fmt.Println(err)
    return err
}
fmt.Printf(" downloaded: %s\n", obj_path)
fmt.Println(h)

This code works just fine when we have compression turned off on the NetScaler (which is between the client and the swift proxy).

Example Output:

 downloaded: 100.txt
map[X-Timestamp:1434554271.52524 Content-Type:text/plain; charset=utf-8 X-Trans-Id:txc301ab2320a64791b978e-0055818fab Date:Wed, 17 Jun 2015 15:18:03 GMT Content-Length:102400 Accept-Ranges:bytes Last-Modified:Wed, 17 Jun 2015 15:17:52 GMT Etag:4c6426ac7ef186464ecbb0d81cbfcb1e]

This code fails when we turn on compression on the NetScaler.

Example Output:

ERROR: Problem downloading 100.txt
map[Content-Type:text/plain; charset=utf-8 X-Trans-Id:txca5e91027dce4491820f5-0055819411 Last-Modified:Wed, 17 Jun 2015 15:36:45 GMT X-Timestamp:1434555404.78653 Etag:4c6426ac7ef186464ecbb0d81cbfcb1e Date:Wed, 17 Jun 2015 15:36:49 GMT Cache-Control:private Cteonnt-Length:102400 Accept-Ranges:bytes]
Bad Header 'Content-Length': '': strconv.ParseInt: parsing "": invalid syntax

Notice that when the code works, we have Content-Length:102400

Notice that when the code fails, we have Cteonnt-Length:102400 and Content-Length is not set.

I have tested other libraries in Java and Python and they do not fail in this same case. In doing some research, this seems to be a standard way to specify if the request was compressed or not. The Content-Length header is returned jumbled if the request was compressed to signify that it is reporting the compressed length.

Do you have any ideas how to address this? Right now I can upload files, but I can't download any files when compression is turned on.

[ncw]

I think a patch like this will fix the issue

--- a/swift.go
+++ b/swift.go
@@ -1422,8 +1422,10 @@ func (c *Connection) ObjectOpen(container string, objectName string, checkHash b
        file.body = io.TeeReader(resp.Body, file.hash)
    }
    // Read Content-Length
-   file.length, err = getInt64FromHeader(resp, "Content-Length")
-   file.lengthOk = (err == nil)
+   if resp.Header.Get("Content-Length") != "" {
+       file.length, err = getInt64FromHeader(resp, "Content-Length")
+       file.lengthOk = (err == nil)
+   }
    return
 }

Reading through the code, having length not set will mean you can't seek from the end of a file and that is about it.

The Length method on the file will fetch the length using a HEAD request if required.

I pushed a branch fix-44 with that change in for you to try.

[swill]

I just tested the fix-44 branch and it works perfectly now...

Here is the result with compression turned on in the NetScaler.

 downloaded: 100.txt
map[Cteonnt-Length:102400 Last-Modified:Wed, 17 Jun 2015 17:03:48 GMT X-Timestamp:1434560627.86336 Date:Wed, 17 Jun 2015 17:03:48 GMT Accept-Ranges:bytes Cache-Control:private Etag:4c6426ac7ef186464ecbb0d81cbfcb1e X-Trans-Id:tx3025cdfdbf5d4d6092916-005581a874 Content-Type:text/plain; charset=utf-8]

Thank you for the quick turn around on this. Excellent support... 👍

[ncw]

Glad it worked and thanks for testing . I'll merge to master shortly...

[ncw]

This is merged to master now - thanks for the report

-- Nick