Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Endpoints without service get removed on controller leader acquire #78933

CallMeFoxie opened this issue Jun 12, 2019 · 2 comments


None yet
3 participants
Copy link

commented Jun 12, 2019

What happened:
We run a several "fake" services in our cluster -- just endpoints, WITHOUT the Service resource available. However when leader election for controller-manager happens the Endpoints resource is deleted.

What you expected to happen:
Keep the Endpoints resource.

How to reproduce it (as minimally and precisely as possible):

  1. Create v1.Endpoints resource with several endpoints WITHOUT creating a Service alongside it
  2. Change elected leader for controller-manager
  3. Observe the Endpoints resource gone

Anything else we need to know?:
Happens on 1.13.5, 1.13.7, 1.14.1 and 1.14.3 (other versions not tested)

sample v1.Endpoints resource:

apiVersion: v1
kind: Endpoints
  name: kupitest
  namespace: ftxt-hint
- addresses:
  - ip:
  - ip:
  - name: hint
    port: 4234
    protocol: TCP


  • Kubernetes version (use kubectl version): 1.13.x, 1.14.x
  • Cloud provider or hardware configuration: baremetal
  • OS (e.g: cat /etc/os-release): Debian stretch & buster
  • Kernel (e.g. uname -a): unrelated
  • Install tools: custom
  • Network plugin and version (if this is a network-related bug): unrelated

This comment has been minimized.

Copy link
Contributor Author

commented Jun 12, 2019

/sig apps
/sig cluster-lifecycle


This comment has been minimized.

Copy link

commented Jun 12, 2019

This is working as designed. If you want to define endpoints persistently, create a service of the same name with an empty selector. See

// checkLeftoverEndpoints lists all currently existing endpoints and adds their
// service to the queue. This will detect endpoints that exist with no
// corresponding service; these endpoints need to be deleted. We only need to
// do this once on startup, because in steady-state these are detected (but
// some stragglers could have been left behind if the endpoint controller
// reboots).
func (e *EndpointController) checkLeftoverEndpoints() {
list, err := e.endpointsLister.List(labels.Everything())
if err != nil {
utilruntime.HandleError(fmt.Errorf("Unable to list endpoints (%v); orphaned endpoints will not be cleaned up. (They're pretty harmless, but you can restart this component if you want another attempt made.)", err))
for _, ep := range list {
if _, ok := ep.Annotations[resourcelock.LeaderElectionRecordAnnotationKey]; ok {
// when there are multiple controller-manager instances,
// we observe that it will delete leader-election endpoints after 5min
// and cause re-election
// so skip the delete here
// as leader-election only have endpoints without service
key, err := controller.KeyFunc(ep)
if err != nil {
utilruntime.HandleError(fmt.Errorf("Unable to get key for endpoint %#v", ep))

@liggitt liggitt closed this Jun 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.