Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When istiod is unavailable portforwarding requests scale with namespaces #5692

Closed
nrfox opened this issue Jan 4, 2023 · 3 comments · Fixed by #6891
Closed

When istiod is unavailable portforwarding requests scale with namespaces #5692

nrfox opened this issue Jan 4, 2023 · 3 comments · Fixed by #6891
Labels
backlog Triaged Issue added to backlog enhancement This is the preferred way to describe new end-to-end features. scalability

Comments

@nrfox
Copy link
Contributor

nrfox commented Jan 4, 2023

This is a followup to #5669. That issue solves for number of requests scaling with pods. This issue is to cover requests scaling with number of selected namespaces.

Steps to reproduce:

  1. Turn on debug logging for Kiali
  2. Scale istiod to 0
  3. Navigate to the overview page
  4. Observe number of attempts to connect to istiod in logs increase with the number of namespaces. There are better ways to observe the connections but observing Kiali's connection logs is the easiest way.
@nrfox nrfox added the bug Something isn't working label Jan 4, 2023
@jshaughn jshaughn added the backlog Triaged Issue added to backlog label Jan 4, 2023
@nrfox
Copy link
Contributor Author

nrfox commented Jan 20, 2023

I'm fairly sure we can remove the refresh handler in the cache altogether if we move to polling istiod. We already recreate the cache any time there is a write operation so we don't need to worry about returning stale data after a write. I think it's perfectly acceptable to return slightly stale data from the debug endpoints if the registry changes in-between polling periods. proxy status is already using polling rather than fetching the data on demand.

@jshaughn
Copy link
Collaborator

I agree with @nrfox .

@jshaughn jshaughn added scalability enhancement This is the preferred way to describe new end-to-end features. and removed bug Something isn't working labels May 30, 2023
@nrfox nrfox self-assigned this Jul 28, 2023
@nrfox
Copy link
Contributor Author

nrfox commented Jul 31, 2023

I've poked at this a little bit and because the istio config is being returned directly from the registry rather than the kube api, when you do an update through the kube api you'll sometimes get back stale data since the registry may not be updated by the time the cache is refreshed. Ideally we'd return the istio config from the kube api rather than the registry so that we can be sure that the previous write has been read. That's a larger refactor though than I was hoping for.

@nrfox nrfox removed their assignment Jul 31, 2023
jshaughn pushed a commit that referenced this issue Dec 7, 2023
Part of Multi-primary support
- #6432
- #5692
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backlog Triaged Issue added to backlog enhancement This is the preferred way to describe new end-to-end features. scalability
Projects
Development

Successfully merging a pull request may close this issue.

2 participants