net/http reading response.body doesn't show the data

[hanshasselberg]

Please answer these questions before submitting your issue. Thanks!

1. What version of Go are you using (go version)?

go version go1.6.2 darwin/amd64

1. What operating system and processor architecture are you using (go env)?

OSX amd64

1. What did you do?

I am using net/http to request a URL. This URL is streaming data back and I can see the data when I use curl. I can also see the data, when I read from the raw tcp connection provdided through net/http/httputil. But I do not see the data when I read response.Body. This is however not always the case, when I try other URLs I can read the data. There seems to something special about mine. These are the response headers:

< HTTP/1.1 200 OK
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: application/json;odata.metadata=minimal;IEEE754Compatible=false;charset=utf-8
< Server: Microsoft-IIS/8.0
< X-BackEndHttpStatus: 200
< Set-Cookie: fubarcookie=a5771fb58997980808080980039a32af2; expires=Tue, 25-Jul-2017 09:58:21 GMT; path=/; HttpOnly
< OData-Version: 4.0
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
< Date: Mon, 25 Jul 2016 09:58:21 GMT

1. What did you expect to see?

I expect to be able to read the data from response.Body.

1. What did you see instead?

No data when reading from response.Body.

[bradfitz]

Sorry, there's not enough information here to help you.

Please provide a URL and/or your code. Ideally both.

[hanshasselberg]

One more bit of information is that when the server closes the connection, I can read the data from request.Body.

One needs to be authenticated in order to be able to access this URL. This is the code however:

package main

import (
    "bufio"
    "fmt"
    "net/http"
)

func main() {
    resp, err := http.Get("https://golang.org")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer resp.Body.Close()
    scanner := bufio.NewScanner(resp.Body)
    scanner.Split(bufio.ScanBytes)
    for scanner.Scan() {
        fmt.Print(scanner.Text())
    }
}

I will try to provide you with a way to access the real thing.

[bradfitz]

Yeah, that code sample doesn't help much. I assume golang.org works fine for you also.

Can you provide a packet capture?

[hanshasselberg]

I created a test account. The access token will expire, but I will try to keep it up to date here.

package main

import (
    "bufio"
    "fmt"
    "net/http"
    "strings"
)

func main() {
    body := `{"SubscriptionIds":["MjhDNjk1QjctNkM3RS00RDBFLTgxRUEtMENGOERCNDhCM0EwXzAwMDM3RkZFLTAwQjgtNEJCMi0wMDAwLTAwMDAwMDAwMDAwMA=="]}`
    req, err := http.NewRequest("POST", "https://outlook.office.com/api/beta/me/getnotifications", strings.NewReader(body))
    if err != nil {
        fmt.Println(err)
        return
    }
    req.Header.Add("Content-Type", "application/json")
    req.Header.Add("Authorization", "Bearer EwAQA+l3BAAUo4xeBIbHjhBxWOFekj4Xy2fhaGQAAabCK+xXoTpX5oPXMILudgQrR7jXzKh+ESBfqE2D/ihCZxXazqWUhcpObIfMZtBxRmef0xWPzYgJhrT9DICVP+1yossFV93G3vTux2yiT9j2RAWm3b+QImS3B4puD08FRxc++D/vCqchHRr8b6uIH6MY5MS7/Rsjb7R3Yr4PJ8wdf39M83JniFrxQw96kD8daIsrVfQN4Le4ejwhpvEdkcCI5HAwJrV8ZH2b2yvu8oM2sFCZxfM6aMWgkqUlO2QfgT0DjqKR11xDG+PiTNYjSpnA1/isPLkmeYCB3aZa6tTLt4pOfyoeBbEm3RbJMoHlNErUBAGccY8lK6qfyQ0pxGADZgAACPectY4BZKX04AEnOHxbHH0G6gExfoOqWBDmmDYnnuS4ETf5eTkAIaPYoA2xjwjrGPi0CRmFgtwqBRt//T3wMnNB4DvtePscTwmo0brSKO/Nw49Z7Eeel0fLkaDqsCy/KxInDa/kBIyjuW8uKNVNtWQE8vIFEysKhNKh/BmKm0a09DonN+mdk2cNLeUvLfl/JZWBeYUA6ryKOjqT5zj78ajDM278MXMlzW1tkG0BBPZWbDWgy7i+SMFDYSLpczdl3S4Sx3qe32B7IUwOQzXKyi+/0tLoFLELsAC1k2D+DZB2Y5etx7BBiuRTuTc721gkSPyQ0R+ms9x2EW7OPnOU3axQznbxHMMd/lp264dLp5QrTusrg0X/wArUtHHa8hoib9p7S5BwOiK9cYxaYnE1F4E+5bMK9bnJGDKNh/Khe5uN51O1/4zNIv+n1q+rRoAwwe34AyIT6GvGD2pED5MA093cjL+Szt5huizOfg5KXg6bwBEfusjo8CQZR8o1vProcru+25/4ElwcVTUQTxel4NTYpSWi+PoRqiWbDa/S+MIdrMpR+cEhj8CA+oRCvxYL/3LnbEw46OJ1IBv1dL1G3wlXrmfwj0a/iSDLjavt/ouMCLQ6HYorUy03Iq/gKnhPSBoxniFQ24YBXrYTAg==")

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer resp.Body.Close()
    scanner := bufio.NewScanner(resp.Body)
    scanner.Split(bufio.ScanBytes)
    for scanner.Scan() {
        fmt.Print(scanner.Text())
    }
}

This is the curl equivalent:

curl -N -H "Content-Type: application/json" -H "X-AnchorMailbox: golangissue16488@outlook.com" -H "Authorization: Bearer EwAQA+l3BAAUo4xeBIbHjhBxWOFekj4Xy2fhaGQAAabCK+xXoTpX5oPXMILudgQrR7jXzKh+ESBfqE2D/ihCZxXazqWUhcpObIfMZtBxRmef0xWPzYgJhrT9DICVP+1yossFV93G3vTux2yiT9j2RAWm3b+QImS3B4puD08FRxc++D/vCqchHRr8b6uIH6MY5MS7/Rsjb7R3Yr4PJ8wdf39M83JniFrxQw96kD8daIsrVfQN4Le4ejwhpvEdkcCI5HAwJrV8ZH2b2yvu8oM2sFCZxfM6aMWgkqUlO2QfgT0DjqKR11xDG+PiTNYjSpnA1/isPLkmeYCB3aZa6tTLt4pOfyoeBbEm3RbJMoHlNErUBAGccY8lK6qfyQ0pxGADZgAACPectY4BZKX04AEnOHxbHH0G6gExfoOqWBDmmDYnnuS4ETf5eTkAIaPYoA2xjwjrGPi0CRmFgtwqBRt//T3wMnNB4DvtePscTwmo0brSKO/Nw49Z7Eeel0fLkaDqsCy/KxInDa/kBIyjuW8uKNVNtWQE8vIFEysKhNKh/BmKm0a09DonN+mdk2cNLeUvLfl/JZWBeYUA6ryKOjqT5zj78ajDM278MXMlzW1tkG0BBPZWbDWgy7i+SMFDYSLpczdl3S4Sx3qe32B7IUwOQzXKyi+/0tLoFLELsAC1k2D+DZB2Y5etx7BBiuRTuTc721gkSPyQ0R+ms9x2EW7OPnOU3axQznbxHMMd/lp264dLp5QrTusrg0X/wArUtHHa8hoib9p7S5BwOiK9cYxaYnE1F4E+5bMK9bnJGDKNh/Khe5uN51O1/4zNIv+n1q+rRoAwwe34AyIT6GvGD2pED5MA093cjL+Szt5huizOfg5KXg6bwBEfusjo8CQZR8o1vProcru+25/4ElwcVTUQTxel4NTYpSWi+PoRqiWbDa/S+MIdrMpR+cEhj8CA+oRCvxYL/3LnbEw46OJ1IBv1dL1G3wlXrmfwj0a/iSDLjavt/ouMCLQ6HYorUy03Iq/gKnhPSBoxniFQ24YBXrYTAg==" -d '{"SubscriptionIds":["MjhDNjk1QjctNkM3RS00RDBFLTgxRUEtMENGOERCNDhCM0EwXzAwMDM3RkZFLTAwQjgtNEJCMi0wMDAwLTAwMDAwMDAwMDAwMA=="]}' "https://outlook.office.com/api/beta/me/getnotifications"

[bradfitz]

This is a bug in Outlook's streaming response when they gzip it. They're not flushing their flate blocks.

Your curl command-line isn't equivalent because it's not doing "Accept-Encoding: gzip".

That endpoints sends a keep-alive JSON value every 15 seconds. It works fine if you do:

        c := &http.Client{
                Transport: &http.Transport{
                        DisableCompression: true,
                },
        }
        resp, err := c.Do(req)

But the default Go behavior is to request gzipped responses, and Outlook is responding with a gzipped response, but their heartbeats every 15 seconds don't include any flate flush (Z_SYNC_FLUSH) so Go can't decode any flate blocks and can't yield any data. Or at least can't yield any data for a very long time (15 seconds * small keep-alive chunks will eventually fill a block).

I recommend you disable compression on Go's side and/or file a bug with Microsoft.

[hanshasselberg]

Thank you very much @bradfitz!

[bradfitz]

There's a Microsoft HTTP guy here at the workshop. I'll get him the details.

[hanshasselberg]

I work at Microsoft so I can take care of it too!

[bradfitz]

That works too. :)

Please file a bug with Go again if you find out my quick analysis was wrong. If we have a bug in our flate reading code, I'm sure @dsnet would love to fix it. :)