TLS Handshake Error

[alastairs]

Do you want to request a feature or report a bug?

Report a bug.

What did you do?

We are running traefik in Docker on Kubernetes, and K8s is deployed on Azure Container Service. We see the following error appearing in the console output approximately once a minute:

2017-07-05T12:38:59.832673311Z 2017/07/05 12:38:59 server.go:2753: http: TLS handshake error from 10.240.0.6:58147: EOF 

Having turned on debug logging, I see it is always accompanied by this log message:

2017-07-05T12:38:59.718337351Z time="2017-07-05T12:38:59Z" level=debug msg="Skipping event from kubernetes &Endpoints{ObjectMeta:ObjectMeta{Name:kube-scheduler,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,UID:673d8e35-5c66-11e7-b220-000d3a216355,ResourceVersion:10786823,Generation:0,CreationTimestamp:2017-06-29 01:00:54 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"k8s-master-16396601-0","leaseDurationSeconds":15,"acquireTime":"2017-06-29T01:00:54Z","renewTime":"2017-07-05T12:38:59Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}"  

As per this thread, this appears to be a new instance of #1025. Although the line number has changed, it remains well outside the range of possible lines in server.go. Additionally the IP addresses lie outside of the cluster subnet.

What did you expect to see?

traefik ticking along nicely, some informational messages, no TLS handshake messages.

What did you see instead?

These error messages. They're particularly annoying as they don't appear to be log messages, and so can't be turned down, turned off, or filtered out.

Output of traefik version: (What version of Traefik are you using?)

Version:      v1.3.2
Codename:     raclette
Go version:   go1.8.3
Built:        2017-06-29_04:52:35PM
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

We are running traefik:1.3 from Docker on Kubernetes 1.5.3, running on Azure Container Service.

docker images traefik:1.3 output:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
traefik             1.3                 917039496236        5 days ago          45MB

kubectl version output:json

Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:34:56Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes Deployment for traefik is (including traefik args at the bottom):

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-controller
  labels:
    service: traefik
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
        service: traefik
    spec:
      terminationGracePeriodSeconds: 60
      volumes:
        - name: config
          configMap:
            name: traefik-conf
        - name: acme
          hostPath:
            path: /acme
      containers:
      - image: traefik:1.3
        name: traefik-ingress-lb
        resources:
          limits:
            cpu: 200m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi
        volumeMounts:
        - mountPath: "/config"
          name: "config"
        - mountPath: "/acme"
          name: "acme"
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        - containerPort: 8080
          hostPort: 8080
        args:
        - --configfile=/config/traefik.toml
        - --web
        - --kubernetes
        - --kubernetes.namespaces={{K8S_NAMESPACE}}
        - --loglevel=INFO

defaultEntryPoints = ["http","https"]
[file]
watch=true
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
  [entryPoints.https.tls]
[web]
address = ":8080"
ReadOnly = true
  [web.statistics]
  RecentErrors = 10
[acme]
email = "{{REDACTED}}"
storageFile = "/acme/acme-ssl.json"
entryPoint = "https"
# caServer = "https://acme-staging.api.letsencrypt.org/directory"
OnHostRule = true

If applicable, please paste the log output in debug mode (--debug switch)

I can provide a full log file privately; please DM @alastairs on Slack if you would like it.

[timoreimann]

The log line is apparently from net/http's server.go. If it helps, someone could hack the file temporarily to add more logging / dump a stack trace in order to identify the root cause.

[michaelchiche]

I am currently having the same issue in a docker swarm cluster in Digital Ocean

[kvncrw]

Seeing the same issue using Docker EE 17.06.1-ee-2 running on Ubuntu 16.04.03/vmware, I've also seen this on Swarm CE 17.06 running on Ubuntu 16.04/vmware. It doesn't appear to cause any critical issues, just log noise.

[emilevauge]

@alastairs @michaelchiche Do you see any issue related to this log ? As @KCrawley said, I think it's just some log noise.

[alastairs]

Do you see any issue related to this log

Yeah, we ship our logs to Elastic Search, and the large amount of noise puts a ton of extra stress on our cluster that we'd rather avoid.

[MichaelErmer]

Also seeing this with v1.4.0 / roquefort, only the line number changed to 2848.

We are seeing several hundred of these messages per second at (~10% of the current throughput), the ip is the one of the user requesting data.

[johnhamelink]

<redacted>

[ldez]

@johnhamelink The symptoms you are experiencing differ from those of the original bug submitter.

Please ask your question on:

• Stack Overflow using the "traefik" tag
• the Traefik community Slack channel

[johnhamelink]

@ldez Yes you are right, I have removed my previous comment as it would probably cause more confusion than anything else!