Deployment Restart via ArgoCD not working.

[agido-heppe]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Hey, since I upgraded to ArgoCD to 2.9.4 via Helmchart version 5.53.1, I cannot restart deployments via the UI. First, I tried to restart a deployment via the regular UI on the deployment object. During this attempt an error message popped up: Unable to execute resource action: Invalid content type. To rule out errors with permissions i tried the same as admin and got the same error.

I also tried this on multiple apps/deployments.

To Reproduce

Restart an app via ArgoUI on Version 2.9.4 (deployed by the helm chart version 5.53.1)

Expected behavior

The deployment should restart like usual.

Screenshots

Version

Unfortunately, I already perfomed a rollback before opening this issue.
Details:
ArgoCD 2.9.4
HelmChart: 5.53.1

If you need detailed build details, I could deploy the newer version later this day.

Logs

I didn't find any related logs, if you could provide me with a hint on what to grep for, I could provide some logs later.

Edit:

• Might be related to fix: enforce content type header for API requests #16860.
  I guess, I have to allow application/grpc and/or others?

[baileydoestech]

We are also seeing this issue upgrading to 2.9.4 - seems to be blocking all post actions in the UI such as deleting a replica set or terminating a sync.

The HTTP error is 415 unsupported media error

[nobbs]

Running the ArgoCD server with the cli argument

--api-content-types application/json;application/x-www-form-urlencoded

does fix that, as the POST requests are using application/x-www-form-urlencoded and as of v2.9.4 only application/json is allowed for non-GET requests. This does fix the "invalid content type" issue for triggering actions from the UI, but I'm not sure whether this does not fully disable the protection for the vulnerability that was fixed in v2.9.4 - that's something someone from the ArgoCD team will need to clarify.

Note from crenshaw-dev: please don't do this long-term, it does defeat the security fix.

[rubensheik]

Running the ArgoCD server with the cli argument

--api-content-types application/json;application/x-www-form-urlencoded

does fix that, as the POST requests are using application/x-www-form-urlencoded and as of v2.9.4 only application/json is allowed for non-GET requests. This does fix the "invalid content type" issue for triggering actions from the UI, but I'm not sure whether this does not fully disable the protection for the vulnerability that was fixed in v2.9.4 - that's something someone from the ArgoCD team will need to clarify.

Hello! I'm getting this same issue with the version v2.9.0+c7bf064 always that i try to perform any exclusion by UI, how can i fix this? I'm using eks HA deploy. Thanks.

Maybe i should edit Deploy Argocd-Server with
kubectl edit deploy argocd-server -n argocd

But can you help how to add these argumments in the file?

[nobbs]

@rubensheik Depends on what you used to deploy ArgoCD. Their community-maintained helm chart from https://github.com/argoproj/argo-helm? Then the following values will work:

server:
  # FIX: this is a workaround for https://github.com/argoproj/argo-cd/issues/16923
  extraArgs:
    - "--api-content-types"
    - "application/json;application/x-www-form-urlencoded"

If it's another way of deployment, you'll need to check how to best modify the args of the container to look like this:

...
    spec:
...
      containers:
        - args:
            - /usr/local/bin/argocd-server
            - '--api-content-types'
            - application/json;application/x-www-form-urlencoded
            - ...
...

[rubensheik]

@rubensheik Depends on what you used to deploy ArgoCD. Their community-maintained helm chart from https://github.com/argoproj/argo-helm? Then the following values will work:

server:
  # FIX: this is a workaround for https://github.com/argoproj/argo-cd/issues/16923
  extraArgs:
    - "--api-content-types"
    - "application/json;application/x-www-form-urlencoded"

If it's another way of deployment, you'll need to check how to best modify the args of the container to look like this:

...
    spec:
...
      containers:
        - args:
            - /usr/local/bin/argocd-server
            - '--api-content-types'
            - application/json;application/x-www-form-urlencoded
            - ...
...

Hi Noobs, thanks for fast answer, i'm using this deploy.

Available in: https://argo-cd.readthedocs.io/en/stable/operator-manual/installation/

[azhurbilo]

@nobbs does it help for you? #16923 (comment)

I still receive error from Argocd UI (post, delete actions)

[nobbs]

@nobbs does it help for you? #16923 (comment)

I still receive error from Argocd UI (post, delete actions)

yes - it does for me. I can restart deployments, delete applications / resources. All works fine after adding that additional argument.

[azhurbilo]

looks I found another issue

it works for PUT as it contains in UI

Request Headers

Content-Type:
application/x-www-form-urlencoded

but DELETE actions still not working

[nobbs]

@azhurbilo You're right. Delete on individual resources doesn't work. I tried to delete an application, that worked.

The delete on individual resources doesn't send a content-type at all, so it's blocking those requests.

[rubensheik]

maybe we should consider version downgrade ?

[crenshaw-dev]

I'm not sure whether this does not fully disable the protection for the vulnerability that was fixed in v2.9.4 - that's something someone from the ArgoCD team will need to clarify.

Allowing application/x-www-form-urlencoded does in fact defeat the security fix.

Downgrade is the only option for now. Working on a patch. Thanks for the reports!

[danivendetta]

Here the same problem. Downgrading to 2.9.3 It's working again.

[crenshaw-dev]

Please let me know if I'm missing issues:

• Deleting resources
• Running resource actions
• Patching resources
• Terminating syncs
• Deleting certificates
• Deleting clusters
• Deleting account tokens
• Deleting repositories
• Deleting GPG keys
• Deleting projects

[crenshaw-dev]

The patch has been reviewed, tested, and approved. I'll push 2.10.0-rc3, 2.9.5, and 2.8.9 the moment CI checks pass. Apologies for the issues, and thank you all for your patience!