Endpoints API Object could not support large number of endpoints #73324
Assignees
Labels
kind/feature
Categorizes issue or PR as related to a new feature.
sig/network
Categorizes an issue or PR as relevant to SIG Network.
Comments
freehan
added
sig/network
|
Hi @freehan , is another PR needed to close this? |
thockin
added
the
triage/unresolved
|
given this is not a regression, it doesn't seem release blocking |
|
/milestone clear Please come talk to us in #sig-release if you feel this was done in error |
thockin
removed
the
triage/unresolved
|
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
the
lifecycle/stale
|
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
lifecycle/rotten
|
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
|
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
This is being tackled in https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/0752-endpointslices |
k8s-ci-robot
removed
the
lifecycle/rotten
jhrozek
added a commit
to jhrozek/security-profiles-operator
that referenced
this issue
Mar 11, 2021
This PR introduced a new API secpolnodestatus that aims to address the following problems: - it was impossible to reflect per-node status such as a profile failed to install on a single node or a node does not support the selected security profile - simply adding per-node attributes like a map might not scale, given very high number of nodes, the object might become too big for etcd's 1MB limit (see also kubernetes/kubernetes#73324) - because the per-profile status is written to by several sources (each pod in a DaemonSet), the status might appear as "flapping" as different pods reach different states at their own pace. The secpolnodestatus is created, managed and deleted together with finalizers through an API in a new module. Instead of updating the global state directly, the DS pods now call the API update method that updates the node status and if needed, also the global status. When a policy is deleted, the object is marked as terminating, when the policy payload is removed, the node status object along with its finalizer is deleted. Finally, when all finalizers are gone, so is the global policy object.
jhrozek
added a commit
to jhrozek/security-profiles-operator
that referenced
this issue
Mar 12, 2021
This PR introduced a new API secpolnodestatus that aims to address the following problems: - it was impossible to reflect per-node status such as a profile failed to install on a single node or a node does not support the selected security profile - simply adding per-node attributes like a map might not scale, given very high number of nodes, the object might become too big for etcd's 1MB limit (see also kubernetes/kubernetes#73324) - because the per-profile status is written to by several sources (each pod in a DaemonSet), the status might appear as "flapping" as different pods reach different states at their own pace. The secpolnodestatus is created, managed and deleted together with finalizers through an API in a new module. Instead of updating the global state directly, the DS pods now call the API update method that updates the node status and if needed, also the global status. When a policy is deleted, the object is marked as terminating, when the policy payload is removed, the node status object along with its finalizer is deleted. Finally, when all finalizers are gone, so is the global policy object.
jhrozek
added a commit
to jhrozek/security-profiles-operator
that referenced
this issue
Mar 12, 2021
This PR introduced a new API secpolnodestatus that aims to address the following problems: - it was impossible to reflect per-node status such as a profile failed to install on a single node or a node does not support the selected security profile - simply adding per-node attributes like a map might not scale, given very high number of nodes, the object might become too big for etcd's 1MB limit (see also kubernetes/kubernetes#73324) - because the per-profile status is written to by several sources (each pod in a DaemonSet), the status might appear as "flapping" as different pods reach different states at their own pace. The secpolnodestatus is created, managed and deleted together with finalizers through an API in a new module. Instead of updating the global state directly, the DS pods now call the API update method that updates the node status and if needed, also the global status. When a policy is deleted, the object is marked as terminating, when the policy payload is removed, the node status object along with its finalizer is deleted. Finally, when all finalizers are gone, so is the global policy object.
jhrozek
added a commit
to jhrozek/security-profiles-operator
that referenced
this issue
Mar 15, 2021
This PR introduced a new API secpolnodestatus that aims to address the following problems: - it was impossible to reflect per-node status such as a profile failed to install on a single node or a node does not support the selected security profile - simply adding per-node attributes like a map might not scale, given very high number of nodes, the object might become too big for etcd's 1MB limit (see also kubernetes/kubernetes#73324) - because the per-profile status is written to by several sources (each pod in a DaemonSet), the status might appear as "flapping" as different pods reach different states at their own pace. The secpolnodestatus is created, managed and deleted together with finalizers through an API in a new module. Instead of updating the global state directly, the DS pods now call the API update method that updates the node status and if needed, also the global status. When a policy is deleted, the object is marked as terminating, when the policy payload is removed, the node status object along with its finalizer is deleted. Finally, when all finalizers are gone, so is the global policy object.
jhrozek
added a commit
to jhrozek/security-profiles-operator
that referenced
this issue
Mar 15, 2021
This PR introduced a new API secpolnodestatus that aims to address the following problems: - it was impossible to reflect per-node status such as a profile failed to install on a single node or a node does not support the selected security profile - simply adding per-node attributes like a map might not scale, given very high number of nodes, the object might become too big for etcd's 1MB limit (see also kubernetes/kubernetes#73324) - because the per-profile status is written to by several sources (each pod in a DaemonSet), the status might appear as "flapping" as different pods reach different states at their own pace. The secpolnodestatus is created, managed and deleted together with finalizers through an API in a new module. Instead of updating the global state directly, the DS pods now call the API update method that updates the node status and if needed, also the global status. When a policy is deleted, the object is marked as terminating, when the policy payload is removed, the node status object along with its finalizer is deleted. Finally, when all finalizers are gone, so is the global policy object.
jhrozek
added a commit
to jhrozek/security-profiles-operator
that referenced
this issue
Mar 17, 2021
This PR introduced a new API secpolnodestatus that aims to address the following problems: - it was impossible to reflect per-node status such as a profile failed to install on a single node or a node does not support the selected security profile - simply adding per-node attributes like a map might not scale, given very high number of nodes, the object might become too big for etcd's 1MB limit (see also kubernetes/kubernetes#73324) - because the per-profile status is written to by several sources (each pod in a DaemonSet), the status might appear as "flapping" as different pods reach different states at their own pace. The secpolnodestatus is created, managed and deleted together with finalizers through an API in a new module. Instead of updating the global state directly, the DS pods now call the API update method that updates the node status and if needed, also the global status. When a policy is deleted, the object is marked as terminating, when the policy payload is removed, the node status object along with its finalizer is deleted. Finally, when all finalizers are gone, so is the global policy object.
jhrozek
added a commit
to jhrozek/security-profiles-operator
that referenced
this issue
Mar 17, 2021
This PR introduced a new API secpolnodestatus that aims to address the following problems: - it was impossible to reflect per-node status such as a profile failed to install on a single node or a node does not support the selected security profile - simply adding per-node attributes like a map might not scale, given very high number of nodes, the object might become too big for etcd's 1MB limit (see also kubernetes/kubernetes#73324) - because the per-profile status is written to by several sources (each pod in a DaemonSet), the status might appear as "flapping" as different pods reach different states at their own pace. The secpolnodestatus is created, managed and deleted together with finalizers through an API in a new module. Instead of updating the global state directly, the DS pods now call the API update method that updates the node status and if needed, also the global status. When a policy is deleted, the object is marked as terminating, when the policy payload is removed, the node status object along with its finalizer is deleted. Finally, when all finalizers are gone, so is the global policy object.
k8s-ci-robot
pushed a commit
to kubernetes-sigs/security-profiles-operator
that referenced
this issue
Mar 18, 2021
This PR introduced a new API secpolnodestatus that aims to address the following problems: - it was impossible to reflect per-node status such as a profile failed to install on a single node or a node does not support the selected security profile - simply adding per-node attributes like a map might not scale, given very high number of nodes, the object might become too big for etcd's 1MB limit (see also kubernetes/kubernetes#73324) - because the per-profile status is written to by several sources (each pod in a DaemonSet), the status might appear as "flapping" as different pods reach different states at their own pace. The secpolnodestatus is created, managed and deleted together with finalizers through an API in a new module. Instead of updating the global state directly, the DS pods now call the API update method that updates the node status and if needed, also the global status. When a policy is deleted, the object is marked as terminating, when the policy payload is removed, the node status object along with its finalizer is deleted. Finally, when all finalizers are gone, so is the global policy object.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened:
When a service select many backend pods (e.g. 10k), the corresponding Endpoints object becomes big (>1MB in proto).
On Kube-proxy, the API watcher will start dropping the endpoints object and will not program iptables for the service. #57073
On master, if the object is too big. It can result in write failure to etcd.
What you expected to happen:
It should just work.
Proposed Short Term Fix:
Environment: OSS K8s
kubectl version): 1.11The text was updated successfully, but these errors were encountered: