Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Endpoints without service get removed on controller leader acquire #78933

Closed
CallMeFoxie opened this issue Jun 12, 2019 · 2 comments

Comments

Projects
None yet
3 participants
@CallMeFoxie
Copy link
Contributor

commented Jun 12, 2019

Hi
What happened:
We run a several "fake" services in our cluster -- just endpoints, WITHOUT the Service resource available. However when leader election for controller-manager happens the Endpoints resource is deleted.

What you expected to happen:
Keep the Endpoints resource.

How to reproduce it (as minimally and precisely as possible):

  1. Create v1.Endpoints resource with several endpoints WITHOUT creating a Service alongside it
  2. Change elected leader for controller-manager
  3. Observe the Endpoints resource gone

Anything else we need to know?:
Happens on 1.13.5, 1.13.7, 1.14.1 and 1.14.3 (other versions not tested)

sample v1.Endpoints resource:

apiVersion: v1
kind: Endpoints
metadata:
  name: kupitest
  namespace: ftxt-hint
subsets:
- addresses:
  - ip: 10.57.10.152
    nodeName: foo1.com
  - ip: 10.57.10.153
    nodeName: foo2.com
  ports:
  - name: hint
    port: 4234
    protocol: TCP

Environment:

  • Kubernetes version (use kubectl version): 1.13.x, 1.14.x
  • Cloud provider or hardware configuration: baremetal
  • OS (e.g: cat /etc/os-release): Debian stretch & buster
  • Kernel (e.g. uname -a): unrelated
  • Install tools: custom
  • Network plugin and version (if this is a network-related bug): unrelated
@CallMeFoxie

This comment has been minimized.

Copy link
Contributor Author

commented Jun 12, 2019

/sig apps
/sig cluster-lifecycle

@liggitt

This comment has been minimized.

Copy link
Member

commented Jun 12, 2019

This is working as designed. If you want to define endpoints persistently, create a service of the same name with an empty selector. See https://kubernetes.io/docs/concepts/services-networking/service/#services-without-selectors

// checkLeftoverEndpoints lists all currently existing endpoints and adds their
// service to the queue. This will detect endpoints that exist with no
// corresponding service; these endpoints need to be deleted. We only need to
// do this once on startup, because in steady-state these are detected (but
// some stragglers could have been left behind if the endpoint controller
// reboots).
func (e *EndpointController) checkLeftoverEndpoints() {
list, err := e.endpointsLister.List(labels.Everything())
if err != nil {
utilruntime.HandleError(fmt.Errorf("Unable to list endpoints (%v); orphaned endpoints will not be cleaned up. (They're pretty harmless, but you can restart this component if you want another attempt made.)", err))
return
}
for _, ep := range list {
if _, ok := ep.Annotations[resourcelock.LeaderElectionRecordAnnotationKey]; ok {
// when there are multiple controller-manager instances,
// we observe that it will delete leader-election endpoints after 5min
// and cause re-election
// so skip the delete here
// as leader-election only have endpoints without service
continue
}
key, err := controller.KeyFunc(ep)
if err != nil {
utilruntime.HandleError(fmt.Errorf("Unable to get key for endpoint %#v", ep))
continue
}
e.queue.Add(key)
}
}

@liggitt liggitt closed this Jun 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.