/api/v1/applications takes significant amount of time

[cedricboesiger]

Describe the bug
The application list page uses GET /api/v1/applications?fields=... which, depending on the number of projects holding OIDC groups, requires more time to answer the request.
In our environment, it already takes between 12s-14s to answer the request.

We count:

• Between 50 to 70 OIDC groups per user
• 25 Appprojects each with 2 roles with up to 10 policies that are assigned to 1 oidc group.
• 215 Applications
• We use v1.5.6 in production

To Reproduce

• Create an empty ArgoCD installation, v1.7.4
• Create a couple 100 Appprojects, each with 2 roles that are assigned to an OIDC group
• Create a couple 100 Applications assigned to the Appprojects
• Use an OpenID User to login
• Check that the response time from /api/v1/applications in the browser console increases with the number of projects with assigned groups.

Expected behavior

A response time that is less dependent on the number of applications, projects and groups involved.

[asvasyanin]

Same issue, we are using v1.7.4

[asvasyanin]

Also, I've notice, that if login using built-in account to the same api server response more quickly.

[asvasyanin]

We have about 800+ applications in 2 clusters.
My results with SSO login and RBAC:

Using built-in administrator account:

[jessesuen]

NOTE: The admin is special because it has no groups and therefore enforcement is simpler.

Between 50 to 70 OIDC groups per user

I think we need an optimization to use a temporary map to lookup oidc groups rather than current implementation to iterate the groups list

[boskoop]

Great news if this is solved with 1.8!

[kshamajain99]

Looking into this.

[rbreeze]

#5342 is a workaround for this, but ideally we should still implement a map for OIDC group lookup

[rbreeze]

Discussed with @jessesuen and we feel pagination addresses the user experience of slow loading apps.

[alexmt]

Hello @cedricboesiger, @boskoop ,

The v2.1 has a optimization that is supposed to significantly improve API server performance. The change is available in https://github.com/argoproj/argo-cd/releases/tag/v2.1.0-rc1. Can you give it a try please?

[alexmt]

Closing and will reopen if someone reproduce issue again

[limoges]

We had very high hopes that this would be fixed with v2.1 and I've come around to making the update to v2.1.2.

Unfortunately, the performance issue is still there. I didn't measure before the update but I don't notice any difference.

We have ~1000 applications.

[matoous]

Can confirm, the load time is still slow

[alexmt]

Thank you @matoous and @limoges ! Reopening ticket. Will attack it in 2.2 again

[yydzhou]

@alexmt is there any progress on the overall application list slowness issue? we have 4k applications now and the application listing cost about 8-10 seconds at least, even if we only assign an admin role to the user.

[alexmt]

@yydzhou the API was improved in v2.2. During performance testing, we've reduced API response time from 10 seconds to ~0.5 seconds. Can you please give it a try?

[amitde69]

still happens on v2.3.3 with 23 clusters and 1612 applications.

[wgzhao]

still happens on v2.4.6 with 10 clusters and 50 applications

[sidewinder12s]

With v2.4.17 it can still take 5+ seconds for 1000+ apps to load

[jujubetsz]

Same here. 3k+ apps, 1 cluster, v2.4.11

Sometimes it take 30s+ :sad:

[crenshaw-dev]

Would y'all be willing to test this? #11884

[jenting]

Does the endpoint /api/v1/applications support pagination?