Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Two controller-manager leader exist #89466

Open
fengzixu opened this issue Mar 25, 2020 · 4 comments
Open

Two controller-manager leader exist #89466

fengzixu opened this issue Mar 25, 2020 · 4 comments

Comments

@fengzixu
Copy link
Contributor

@fengzixu fengzixu commented Mar 25, 2020

What happened:
In the cluster, two controller-manager work. Both of them acquire the leader lease successfully.

What you expected to happen:

Obviously, only one controller-manager

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):1.15.3
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:

3 master nodes(etcd also was deployed on those hosts, 3 etcd servers)
3 worker nodes

@fengzixu

This comment has been minimized.

Copy link
Contributor Author

@fengzixu fengzixu commented Mar 25, 2020

master1

 {"log":"I0324 03:40:10.801139       1 leaderelection.go:235] attempting to acquire leader lease  kube-system/kube-controller-manager...\n","stream":"stderr","time":"2020-03-24T03:40:10.801173872Z"}
76607 {"log":"I0324 03:40:42.273548       1 leaderelection.go:245] successfully acquired lease kube-system/kube-controller-manager\n","stream":"stderr","time":"2020-03-24T03:40:42.274833354Z"}

master3

{"log":"I0324 03:40:10.963945       1 leaderelection.go:235] attempting to acquire leader lease  kube-system/kube-controller-manager...\n","stream":"stderr","time":"2020-03-24T03:40:10.964040404Z"}
{"log":"I0324 03:40:26.196409       1 leaderelection.go:245] successfully acquired lease kube-system/kube-controller-manager\n","stream":"stderr","time":"2020-03-24T03:40:26.197261473Z"}

And I checked the endpoint of kube-controller-manager,holderIdentity shows the master1 is the leader.

[]# k -n kube-system get ep kube-controller-manager -o yaml
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"c001-dev_d7343974-329d-42f9-91ff-95b2d5ecd7d3","leaseDurationSeconds":15,"acquireTime":"2020-03-24T03:40:42Z","renewTime":"2020-03-25T08:28:33Z","leaderTransitions":3}'
  creationTimestamp: "2020-03-14T03:30:38Z"
  name: kube-controller-manager
  namespace: kube-system
  resourceVersion: "627833"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
  uid: 227af113-2618-492c-8af2-fa88fca65d1a
@fengzixu

This comment has been minimized.

Copy link
Contributor Author

@fengzixu fengzixu commented Mar 25, 2020

/sig api-machinery

@fedebongio

This comment has been minimized.

Copy link
Contributor

@fedebongio fedebongio commented Mar 26, 2020

@cheftako

This comment has been minimized.

Copy link
Member

@cheftako cheftako commented Mar 26, 2020

Hi @fengzixu, I'm seeing a few odd things and would like to clarify.
You lease duration is set to 15 seconds but renew time appears to be a day later. Did you do your kubectl get the following day?

Can you give us the following flag values from both (all 3) kubernetes controller managers?
--leader-elect
--leader-elect-resource-lock
--leader-elect-resource-name
--leader-elect-resource-namespace

Can we get the kubeconfig file for each kubernetes controller managers?

Can we get the following flags from the relevant etcd instances?
--initial-cluster
–listen-peer-urls
–initial-advertise-peer-urls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.