Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Excessive latency when serving function list endpoint #64
I've noticed excessive latency when serving function list endpoint when compared to faas-netes. For instance we have 100 function deployments on the community cluster right now running with faas-netes, but calling the 127.0.0.1:8080/system/functions endpoint takes
The operator is taking a similar amount of time some of the time, but on subsequent requests can be taking 3-5 seconds or longer, or even timing out.
Reproduce the issue
Could this be caused by locking or the way we are looking up services?
changed the title from
Excessive latency when querying
Excessive latency when serving function list endpoint
Dec 2, 2018
The functions I deployed were from the certifier so some have embedded annotations/labels. Even with store functions: figlet, certinfo and nodeinfo - calling the endpoint seems to give more latency when called several times.
This is the kind of latency I'm seeing when I added tracing points with glog and time.Start/Since:
It seems like the "getReplicas" function is a source of latency adding about 0.5s per function in the cluster. https://github.com/openfaas-incubator/openfaas-operator/blob/master/pkg/server/replicas.go#L52
@stefanprodan suggested this behaviour may be related to rate limiting enforced by the Kubernetes API Server.
Given my approach reduces API calls this would make sense.
I'll test this first with