Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log spam (and higher cpu usage) due to postgres-operator #1458

Closed
thoro opened this issue Mar 30, 2023 · 4 comments
Closed

Log spam (and higher cpu usage) due to postgres-operator #1458

thoro opened this issue Mar 30, 2023 · 4 comments
Labels

Comments

@thoro
Copy link
Contributor

thoro commented Mar 30, 2023

What happened?
zalando/postgres-operator users the annotations of the main db service service as a keep alive token. This causes an endpoints update for kube-router.
It will spam the output with lines like:

I0330 09:11:58.326271       1 network_services_controller.go:1106] Skipping service name:live-bitbucket-database namespace:atlassian as there is no cluster IP

etc. every second

What did you expect to happen?

Do not consider updates to annotations relevant, since they are not used in building the endpoint information anyways. This means they can be safely ignored

How can we reproduce the behavior you experienced?
Steps to reproduce the behavior:

  1. Create a service
  2. Update the services annotations
  3. Kube-router will recheck all the services on all nodes

** System Information (please complete the following information):**

  • Kube-Router Version (kube-router --version): [e.g. 1.0.1] 1.5.3
  • Kube-Router Parameters: [e.g. --run-router --run-service-proxy --enable-overlay --overlay-type=full etc.]
  •      - "--run-router=false"
         - "--run-firewall=false"
         - "--enable-overlay=false"
         - "--run-service-proxy=true"
         - "--ipvs-sync-period=1m"
         - "--masquerade-all=true"
         - "--enable-pod-egress=false"
         - "--enable-cni=false"
         - "--metrics-port=8081"
         - "--metrics-path=/metrics"
         - "-v=2"
    
- Kubernetes Version (`kubectl version`) : [e.g. 1.18.3] 1.24.12
- Cloud Type: [e.g. AWS, GCP, Azure, on premise] on premise
- Kubernetes Deployment Type: [e.g. EKS, GKE, Kops, Kubeadm, etc.] manual
- Kube-Router Deployment Type: [e.g. DaemonSet, System Service] DaemonSet
- Cluster Size: [e.g. 200 Nodes] 10
@thoro thoro added the bug label Mar 30, 2023
thoro added a commit to thoro/kube-router that referenced this issue Mar 30, 2023
@thoro
Copy link
Contributor Author

thoro commented Mar 31, 2023

See #1459 for possible fix

@aauren
Copy link
Collaborator

aauren commented Apr 3, 2023

Seems reasonable. I commented on the PR that you submitted. Thanks for looking into this and submitting a fix!

thoro added a commit to thoro/kube-router that referenced this issue Apr 3, 2023
thoro added a commit to thoro/kube-router that referenced this issue Apr 3, 2023
@aauren
Copy link
Collaborator

aauren commented Apr 4, 2023

Fixed via #1461

@aauren aauren closed this as completed Apr 4, 2023
@aauren
Copy link
Collaborator

aauren commented Apr 4, 2023

I'll try to do a new RC release either this week or next to incorporate the change. Thanks again for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants