KF 1.7 Run metadata not updating

[savemuri]

/kind bug

What steps did you take and what happened:
I am running kubeflow 1.7 and when I clone an existing run and open /pipeline/?ns=${SOME_NAMESPACE}#/runs/details/${UUID}, it takes too long to update run status (around 300s). Until then it is stuck in Unknown status

Note:

• The workflow pod does get created and executes correctly.
• Runs created by scheduled workflow does not have the same issue.

What did you expect to happen:
Expect to see run status updated with task information.

Anything else you would like to add:
Tried scaling up ml-pipeline and persistence-agent replicas but it did not help.

Environment:

• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): 1.7
• kfctl version: (use kfctl version): n/a
• Kubernetes platform: (e.g. minikube): EKS
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[savemuri]

Resolved after cleaning up old workflows that were not deleted. Persistence agent was becoming a bottleneck because workflow GC did not work as intended.