New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix rate limiting latency for list endpoint #65

Merged
merged 1 commit into from Dec 3, 2018

Conversation

Projects
None yet
3 participants
@alexellis
Member

alexellis commented Dec 3, 2018

Fix rate limiting latency for list endpoint as shown in #64

Signed-off-by: Alex Ellis (VMware) alexellis2@gmail.com

Description

  • the list endpoint of the REST API was making too many API calls
    in a short period of time (1 per existing function in cluster)
    so we hit rate limiting and the Kubernetes API server started to
    rate limit us causing a huge delay to the end-user and timeouts.

Motivation and Context

  • I have raised an issue to propose this change (required)

This issue was found when @bartsmykla tried out OpenFaas Cloud with
the Operator which makes use of the REST API. The UI would have
had the same issue causing a poor and confusing experience for
our users.

I logged the issue and explored a fix in #64

How Has This Been Tested?

An initial fix would have used List instead of Get to reduce API
calls, but this fix follows an idea from @stefanprodan to make use of
a shared Informer which already has a cache of the state of
Deployments in the cluster.

Tested on Linux Kubernetes 1.10 cluster and could not reproduce
reported issue with fix in place.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Impact to existing users

Improved responsiveness to REST API calls, CLI and UI

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I've read the CONTRIBUTION guide
  • I have signed-off my commits with git commit -s
  • I have added tests to cover my changes.
  • All new and existing tests passed.
Fix rate limiting latency for list endpoint
- the list endpoint of the REST API was making too many API calls
in a short period of time (1 per existing function in cluster)
so we hit rate limiting and the Kubernetes API server started to
rate limit us causing a huge delay to the end-user and timeouts.

This issue was found when Bart S tried out OpenFaas Cloud with
the Operator which makes use of the REST API. The UI would have
had the same issue causing a poor and confusing experience for
our users.

An initial fix would have used List instead of Get to reduce API
calls, but this fix follows an idea from Stefan to make use of
a shared Informer which already has a cache of the state of
Deployments in the cluster.

Tested on Linux Kubernetes 1.10 cluster and could not reproduce
reported issue.

Signed-off-by: Alex Ellis (VMware) <alexellis2@gmail.com>
@@ -13,7 +13,11 @@ import (
func makeDeleteHandler(namespace string, client clientset.Interface) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
if r.Body != nil {

This comment has been minimized.

@alexellis

alexellis Dec 3, 2018

Member

Additional belt and braces added for preventing trailing connections.

@stefanprodan

LGTM

@alexellis alexellis merged commit 21fc310 into master Dec 3, 2018

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@alexellis

This comment has been minimized.

Member

alexellis commented Dec 3, 2018

Thanks for the review both.

@alexellis alexellis deleted the alexellis/fix-rate-limiting branch Dec 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment