Context deadline exceeded when having more than 30 apps

[peterbosalliandercom]

Describe the bug
with version 1.0.0
When using more than 20 apps it appears a lot of the time that argo cannot refresh or sync and gives context deadline exceeded errors.

To Reproduce
Use more applications and the problem appears

Expected behavior
Quick response without errors

Screenshots
If applicable, add screenshots to help explain your problem.

Version

1.0.0

Logs

Paste any relevant application logs here.

Additional context
Add any other context about the problem here.

[alexmt]

I'm almost sure this is because of this change #1528

@peterbosalliandercom , do you store all applications in a single repo? In this case repo server responses might be slow because it hold lock per repo during manifest generation.

As a quick fix you can increase timeout using '--repo-server-timeout-seconds' flag in argocd-application-controller deployment. By default it is 60 seconds

[alexmt]

You can also increase number of repo servers to improve performance.

Meanwhile we should optimize template generation in case when repo has a lots of applications.

[peterbosalliandercom]

We will test thism thanks for quick response

[alexmt]

@peterbosalliandercom did you have a chance to check if '--repo-server-timeout-seconds' helps?

[sahil87]

We were facing the same issue. This helped instantly.
For reference, we used the following overlay for deploying argocd:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-application-controller
spec:
  template:
    spec:
      containers:
      - name: argocd-application-controller
        command:
        - argocd-application-controller
        - --status-processors
        - "20"
        - --operation-processors
        - "10"
        - --repo-server-timeout-seconds
        - "180"

[peterbosalliandercom]

Looks fine with these settings. I am wondering what the other settings do, the status processors and operation processors

[alexmt]

Thanks for confirming! The status-processors flag specifies how many unique apps can be reconciled concurrently. The operation-processors flag specifies max number of concurrent sync operations.