High Latency observed between staging pod completed and stateful set creation

[PavanVasist]

Describe the bug

As a followup for the scalability tests we performed a max concurrent Push test.
During the tests we observed a strange behaviour when we perform concurrent pushes(20) we see huge latency between staging pod completion time in cf-workloads-staging namespace and creation of statefulset in cf-workloads namespace.

We already ensured that cf-api-server has enough resources to handle the load.

Below graph shows latency with X-Axis having Number of Apps deployed vs Y-Axis Latency in seconds(Tme difference between staging pod completed time and Stateful set creation time ).

To Reproduce*

Steps to reproduce the behavior:

1. Try to push concurrent apps with 20~40 at a time.
2. Now we used the below method to extract the time difference.

guid=$(kubectl get pods -n cf-workloads-staging $app -o json | jq '.metadata.labels | to_entries | .[] | select(.key=="cloudfoundry.org/app_guid") | .value ' | cut -d'"' -f 2)

stage_completion_time=$(kubectl get pods -n cf-workloads-staging $app -o json | jq '.status.conditions[0].lastTransitionTime' | cut -d'"' -f 2)

sts_creation_time=$(kubectl get sts -n cf-workloads -l  cloudfoundry.org/app_guid=$guid -o json | jq '.items[0].metadata.creationTimestamp' | cut -d'"' -f 2)

Let us know if the issue is from Eirini, we can open the issue over there.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175081091

The labels on this github issue will be updated when the story is started.

[heycait]

Given the conversation around converging on a shared CF on K8s vision, we won’t be focused on scaling the current cf for k8s and are going to close the issue.

We’d recommend either commenting on the document to discuss scaling needs or attending the Cloud Foundry for Kubernetes SIG meeting on the calendar here: https://www.cloudfoundry.org/community-calendar/