net/http: do not log error in http.Server for TCP probes

[sylr]

What version of Go are you using ?

1.10.x

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using?

N/A

What did you do?

Kubernetes apiserver is based on http.Server with a TLS config. If you put a Loadbalancer in front of it which implements basic TCP probing for the load balancing you end up being flooded by the following message:

I0810 08:34:34.483565 1 logs.go:49] http: TLS handshake error from 168.63.129.16:64551: EOF

Which originates from:

go/src/net/http/server.go

Lines 1762 to 1765 in d3c3aaa

	if err := tlsConn.Handshake(); err != nil {
	c.server.logf("http: TLS handshake error from %s: %v", c.rwc.RemoteAddr(), err)
	return
	}

What did you expect to see?

I was hoping we could avoid logging an error for opened and closed tcp sockets.

[bcmills]

CC @bradfitz

[bradfitz]

Well, you control https://golang.org/pkg/net/http/#Server.ErrorLog so you can filter those out yourself if you'd like. That's at least a fix you could do today, without waiting for a decision and a new Go release and for Kubernetes to pick up said new Go release.

I could imagine other people would prefer to see it logged for debugging connection issues, though. It's not obvious that the correct thing is to not log on such errors.

[sylr]

@bradfitz Yeah that could be a solution but that would turn off the logging for all errors right ?

Would it be considerable to only log an error if something has been written on the socket ?

[agnivade]

You can still pass your own io.Writer implementation and handle the error that you want.

[bradfitz]

@bradfitz Yeah that could be a solution but that would turn off the logging for all errors right ?

No, when I said "filter" I meant you could write code to decide which log prints you don't care about.

Part of the log.Logger contract (at https://golang.org/pkg/log/#Logger) is:

Each logging operation makes a single call to the Writer's Write method.

So it's super easy to filter with a custom io.Writer, as @agnivade said. You don't need to implement any buffering.

[sylr]

I've been looking at the code of the kubernetes apiserver and I can't find any ErrorLog being set :

https://github.com/kubernetes/kubernetes/blob/201c11f147c85b029665915bee3a62eea19d6d57/staging/src/k8s.io/apiserver/pkg/server/serve.go#L48-L61

[sylr]

All right I think I found it, they overcharged the default logger via https://github.com/kubernetes/kubernetes/blob/201c11f147c85b029665915bee3a62eea19d6d57/staging/src/k8s.io/apiserver/pkg/util/logs/logs.go#L54-L59

I don't think they'll accept that I filter out this log in their Write() implementation ...

[lavalamp]

I'm definitely not excited about running a regexp on every single log line.

I haven't done a survey, but I think open/close a TCP port is not an unheard of way to monitor something.

On the other hand, if you're not monitoring your service that way, then you probably absolutely want to see this log message somehow, it looks like a DoS attempt.

I think what I'd really like in either case way is a metric to monitor.

I can imagine setting up an optional channel for the http server to send non-fatal errors down instead of logging, that's a lot of machinery but should satisfy all the use cases.