This repository has been archived by the owner. It is now read-only.

Why does "km controller-manager" think it is an invalid event? #694

Closed
NanXiao opened this Issue Dec 11, 2015 · 7 comments

Comments

Projects
None yet
2 participants
@NanXiao

NanXiao commented Dec 11, 2015

I follow Getting started with Kubernetes on Mesos and try to build k8s on Mesos. After executing the km scheduler ... command:

  $ km scheduler \
  --address=${KUBERNETES_MASTER_IP} \
  --mesos-master=${MESOS_MASTER} \
  --etcd-servers=http://${KUBERNETES_MASTER_IP}:4001 \
  --mesos-user=root \
  --api-servers=${KUBERNETES_MASTER_IP}:8888 \
  --cluster-dns=10.10.10.10 \
  --cluster-domain=cluster.local \
  --v=2 >scheduler.log 2>&1 &

I find there is error log in controller.log:

I1211 07:24:25.090789    1841 nodecontroller.go:363] NodeController observed a new Node: api.Node{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"ubuntu", GenerateName:"", Namespace:"", SelfLink:"/api/v1/nodes/ubuntu", UID:"1c84a626-a002-11e5-beea-24be0518a746", ResourceVersion:"21", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63585433463, nsec:0, loc:(*time.Location)(0x3529ee0)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"kubernetes.io/hostname":"ubuntu"}, Annotations:map[string]string(nil)}, Spec:api.NodeSpec{PodCIDR:"", ExternalID:"ubuntu", ProviderID:"", Unschedulable:false}, Status:api.NodeStatus{Capacity:api.ResourceList(nil), Phase:"Pending", Conditions:[]api.NodeCondition(nil), Addresses:[]api.NodeAddress(nil), DaemonEndpoints:api.NodeDaemonEndpoints{KubeletEndpoint:api.DaemonEndpoint{Port:0}}, NodeInfo:api.NodeSystemInfo{MachineID:"", SystemUUID:"", BootID:"", KernelVersion:"", OsImage:"", ContainerRuntimeVersion:"", KubeletVersion:"", KubeProxyVersion:""}}}
I1211 07:24:25.090973    1841 nodecontroller.go:511] Recording Registered Node ubuntu in NodeController event message for node ubuntu
I1211 07:24:25.091184    1841 event.go:206] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"ubuntu", UID:"ubuntu", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node ubuntu event: Registered Node ubuntu in NodeController
E1211 07:24:25.141989    1841 event.go:188] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"ubuntu.141edac3ed15ee27", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"ubuntu", UID:"ubuntu", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"RegisteredNode", Message:"Node ubuntu event: Registered Node ubuntu in NodeController", Source:api.EventSource{Component:"controllermanager", Host:""}, FirstTimestamp:unversioned.Time{Time:time.Time{sec:63585433465, nsec:90993703, loc:(*time.Location)(0x3529ee0)}}, LastTimestamp:unversioned.Time{Time:time.Time{sec:63585433465, nsec:90993703, loc:(*time.Location)(0x3529ee0)}}, Count:1, Type:"Normal"}': 'Event "ubuntu.141edac3ed15ee27" is invalid: involvedObject.namespace: invalid value '', Details: namespace is not required for node' (will not retry!)

Why does km controller-manager complain "'Event "ubuntu.141edac3ed15ee27" is invalid: involvedObject.namespace: invalid value '', Details: namespace is not required for node'"?

Thanks in advance!

Best Regards
Nan Xiao

@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts Dec 11, 2015

If this is master, node events are broken: kubernetes/kubernetes#18460

sttts commented Dec 11, 2015

If this is master, node events are broken: kubernetes/kubernetes#18460

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 12, 2015

@sttts So merge kubernetes/kubernetes#18462 can fix this issue, right? Thanks!

NanXiao commented Dec 12, 2015

@sttts So merge kubernetes/kubernetes#18462 can fix this issue, right? Thanks!

@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts Dec 12, 2015

@NanXiao yes, it fixed it for me.

sttts commented Dec 12, 2015

@NanXiao yes, it fixed it for me.

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 12, 2015

@sttts Yes, this patch can work! But there is error when try to schedule ngnix pod, and below is scheduler.log:

    ......
I1212 07:18:13.735919    7193 algorithm.go:77] Try to schedule pod nginx
I1212 07:18:13.736126    7193 fcfs.go:95] failed to find a fit for pod: default/nginx
I1212 07:18:13.736151    7193 controller.go:88] Failed to schedule: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:nginx GenerateName: Namespace:default SelfLink:/api/v1/namespaces/default/pods/nginx UID:6a80c457-a0ca-11e5-a7a0-24be0518a746 ResourceVersion:1160 Generation:0 CreationTimestamp:2015-12-12 07:18:13 -0500 EST DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[]} Spec:{Volumes:[] Containers:[{Name:nginx Image:nginx Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:80 Protocol:TCP HostIP:}] Env:[] Resources:{Limits:map[cpu:{Amount:0.250 Format:DecimalSI} memory:{Amount:67108864 Format:BinarySI}] Requests:map[memory:{Amount:67108864 Format:BinarySI} cpu:{Amount:0.250 Format:DecimalSI}]} VolumeMounts:[] LivenessProbe:<nil> ReadinessProbe:<nil> Lifecycle:<nil> TerminationMessagePath:/dev/termination-log ImagePullPolicy:Always SecurityContext:<nil> Stdin:false StdinOnce:false TTY:false}] RestartPolicy:Always TerminationGracePeriodSeconds:0xc820673d50 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[] ServiceAccountName: NodeName: SecurityContext:0xc8205cfa80 ImagePullSecrets:[]} Status:{Phase:Pending Conditions:[] Message: Reason: HostIP: PodIP: StartTime:<nil> ContainerStatuses:[]}}
I1212 07:18:13.736536    7193 errorhandler.go:59] Error scheduling nginx: No suitable offers for pod/task; retrying
I1212 07:18:13.836747    7193 queuer.go:164] attempting to yield a pod

Could you give some comments? Thanks in advance!

NanXiao commented Dec 12, 2015

@sttts Yes, this patch can work! But there is error when try to schedule ngnix pod, and below is scheduler.log:

    ......
I1212 07:18:13.735919    7193 algorithm.go:77] Try to schedule pod nginx
I1212 07:18:13.736126    7193 fcfs.go:95] failed to find a fit for pod: default/nginx
I1212 07:18:13.736151    7193 controller.go:88] Failed to schedule: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:nginx GenerateName: Namespace:default SelfLink:/api/v1/namespaces/default/pods/nginx UID:6a80c457-a0ca-11e5-a7a0-24be0518a746 ResourceVersion:1160 Generation:0 CreationTimestamp:2015-12-12 07:18:13 -0500 EST DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[]} Spec:{Volumes:[] Containers:[{Name:nginx Image:nginx Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:80 Protocol:TCP HostIP:}] Env:[] Resources:{Limits:map[cpu:{Amount:0.250 Format:DecimalSI} memory:{Amount:67108864 Format:BinarySI}] Requests:map[memory:{Amount:67108864 Format:BinarySI} cpu:{Amount:0.250 Format:DecimalSI}]} VolumeMounts:[] LivenessProbe:<nil> ReadinessProbe:<nil> Lifecycle:<nil> TerminationMessagePath:/dev/termination-log ImagePullPolicy:Always SecurityContext:<nil> Stdin:false StdinOnce:false TTY:false}] RestartPolicy:Always TerminationGracePeriodSeconds:0xc820673d50 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[] ServiceAccountName: NodeName: SecurityContext:0xc8205cfa80 ImagePullSecrets:[]} Status:{Phase:Pending Conditions:[] Message: Reason: HostIP: PodIP: StartTime:<nil> ContainerStatuses:[]}}
I1212 07:18:13.736536    7193 errorhandler.go:59] Error scheduling nginx: No suitable offers for pod/task; retrying
I1212 07:18:13.836747    7193 queuer.go:164] attempting to yield a pod

Could you give some comments? Thanks in advance!

@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts Dec 12, 2015

Not an actual error. We should change the log message for that. There is just no offer (yet) matching that pod. Either waiting a second fixes this or one has to dig into why the offers do not match the pod. There can be many reasons like missing resources which are required by a pod.

sttts commented Dec 12, 2015

Not an actual error. We should change the log message for that. There is just no offer (yet) matching that pod. Either waiting a second fixes this or one has to dig into why the offers do not match the pod. There can be many reasons like missing resources which are required by a pod.

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 14, 2015

@sttts : The following is kubectl get node output:

$ kubectl get node
NAME      LABELS                          STATUS     AGE
ubuntu    kubernetes.io/hostname=ubuntu   NotReady   4h

The node isn't ready, so could you give some clues? Thanks in advance!

NanXiao commented Dec 14, 2015

@sttts : The following is kubectl get node output:

$ kubectl get node
NAME      LABELS                          STATUS     AGE
ubuntu    kubernetes.io/hostname=ubuntu   NotReady   4h

The node isn't ready, so could you give some clues? Thanks in advance!

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 17, 2015

@sttts The root cause has been found, it is the Mesos slave script should be executed in root privilege:

./bin/mesos-slave.sh --master=127.0.0.1:5050

Thanks very much for your support!

BTW, I think Getting started with Kubernetes on Mesos is a little out of date, do you have plan to update it?

Best Regards
Nan Xiao

NanXiao commented Dec 17, 2015

@sttts The root cause has been found, it is the Mesos slave script should be executed in root privilege:

./bin/mesos-slave.sh --master=127.0.0.1:5050

Thanks very much for your support!

BTW, I think Getting started with Kubernetes on Mesos is a little out of date, do you have plan to update it?

Best Regards
Nan Xiao

@NanXiao NanXiao closed this Dec 18, 2015

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.