EOF errors from reqwest .next()

[clux]

Seeing a lot more of:

ReqwestError: request or response body error: error reading a body from connection: unexpected EOF during chunk size line: ReqwestError(reqwest::Error { kind: Body, source: hyper::Error(Body, Custom { kind: UnexpectedEof, error: "unexpected EOF during chunk size line" }) })

I don't remember them happening this frequently. Anyone else seeing this with reflectors?

[clux]

Wondering if this is related to #144 and us doing something wrong in the request_events loop by not stopping correctly. The EOF error seems to originate from hyper in https://github.com/hyperium/hyper/blob/6b47c69f4a0c67ef416a492cc129ae1955c659f6/src/proto/h1/decode.rs#L101-L127, and it's just coming straight out of the response.next().await as an Err.

Unless there's an underlying bug in reqwest, I'd imagine this is would only happen if we tried to carry on too much in the unfold.

[clux]

I can see it happening when the watch times out completely (300s no events):

[2020-02-25T13:05:22Z DEBUG kube::client] start stream 200 https://cluster.invalid:3026/apis/some_group/v1/namespaces/dev/some_resource?&watch=true&resourceVersion=339349192&timeoutSeconds=300
[2020-02-25T13:05:22Z DEBUG kube::client] headers: {"audit-id": "cb12d329-9b49-441c-a8f0-c6d1c542b8a8", "cache-control": "no-cache, no-store, must-revalidate", "content-type": "application/json", "date": "Tue, 25 Feb 2020 13:05:22 GMT", "expires": "0", "pragma": "no-cache", "transfer-encoding": "chunked"}
[2020-02-25T13:05:22Z DEBUG kube::client] Await chunk
[2020-02-25T13:10:22Z ERROR kube::client] err poll: reqwest::Error { kind: Body, source: hyper::Error(Body, Custom { kind: UnexpectedEof, error: "unexpected EOF during chunk size line" }) } - None
[2020-02-25T13:10:22Z WARN  kube::api::reflector] Poll error on RawApi { resource: "some_resource", group: "some_group", namespace: Some("ns"), version: "v1", prefix: "apis" }: ReqwestError: request or response body error: error reading a body from connection: unexpected EOF during chunk size line: ReqwestError(reqwest::Error { kind: Body, source: hyper::Error(Body, Custom { kind: UnexpectedEof, error: "unexpected EOF during chunk size line" }) })

[clux]

Can be fixed by setting a hard timeout on the client, and catching the e.is_timeout() inside the chunk error. But possible there's a better way. Maybe we should be more opportunistic about error types, but that would probably conflict with #144

[clux]

This is primarily due to kubernetes/kubernetes#6513

We should have our timeout slightly less than 5 minutes whereas now we have 300s and that's just too much. The issue appears to go away if we hard timeout the client (in the client builder) to 295s, and disallow listparams.timeout >= 295.

We should probably still ignore eof errors in the unfold so will try to add that as well.
ETA of fix release: late tonight.

Should also include reqwest bump for #144

[clux]

Fixed in #145 (kube 0.26.0)