Cannot apply CustomResourceDefinitions from namespaced install eventhough RBAC allows it

[wilmardo]

Describe the bug

We are using ArgoCD in a managed cluster where we have rights within our namespaces. So we use the namespaced ArgoCD install with a cluster with namespaces defined:

---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cluster-kubernetes.default.svc
  namespace: argocd
  annotations:
    managed-by: argocd.argoproj.io
  labels:
    argocd.argoproj.io/secret-type: cluster
stringData:
  config: '{"tlsClientConfig":{"insecure":false}}'
  name: iks-cluster
  namespaces: argocd,playground,tools
  server: https://kubernetes.default.svc

We also have the right to create CRDs in the clusterscope, but just CRDs nothing else. This gives us the ability to install applications within our namespaces that define CRDs, like ArgoCD or Gloo.
This gives us an issue with Argo, in the namespaced setup it errors when applying CRDs:

Cluster level ClusterIssuer "letsencrypt-production" can not be managed when in namespaced mode

When I set namespaces: * Argo cannot connect to the cluster anymore since RBAC doesn't allow clusterresources other then the CRDs:

failed to sync cluster https://10.43.0.1:443: failed to load initial state of resource Deployment.apps: deployments.apps is forbidden: User "system:serviceaccount:argocd:argocd-application-controller" cannot list resource "deployments" in API group "apps" in the namespace "*

I traced the change back to this PR:
argoproj/gitops-engine#143

It shipped with ArgoCD 1.7.7, we are now pinned to 1.7.6. In that version it applies the CRDs fine, it just has no diff and is permanently OutOfSync

To Reproduce

• Setup ArgoCD as a namespaced install with a namespace set for the cluster.
• Create the RBAC underneath to give ArgoCD permission to apply CRDs at the clusterscope.
• Apply an CRD, I included a ClusterIssuer as an example.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
  name: argocd:admin
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  - configmaps
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - argoproj.io
  resources:
  - applications
  - appprojects
  verbs:
  - create
  - get
  - list
  - watch
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - list
---
apiVersion: cert-manager.io/v1alpha3
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: info@example.org
    privateKeySecretRef:
      name: le-account-key
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
            class: nginx

Expected behavior

ArgoCD to apply the CRD and have a Healthy status.

Version

argocd: v1.4.2+48cced9
  BuildDate: 2020-01-24T01:04:04Z
  GitCommit: 48cced9d925b5bc94f6aa9fa4a8a19b2a59e128a
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.7.4+f8cbd6b
  BuildDate: 2020-09-05T02:45:44Z
  GitCommit: f8cbd6bf432327cc3b0f70d23b66511bb906a178
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.17.8

[alexmt]

As of v2.1 it is possible to enable cluster level resources in a namespaced mode using:

argocd cluster add <context> --cluster-resources=true --namespace my-namespace

[wilmardo]

I just gave it a try but that feature doesn't cover this issue. It tries to add a new cluster level service-account:

# argocd cluster add staging-1 --cluster-resources=true --namespace test
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `imax-staging-1` with full cluster level admin privileges. Do you want to continue [y/N]? y
FATA[0002] Failed to create service account "argocd-manager" in namespace "kube-system": serviceaccounts is forbidden: User "u-gglgl5os6k" cannot create resource "serviceaccounts" in API group "" in the namespace "kube-system"

Which doesn't work and also isn't needed. The current serviceAccount for ArgoCD has the RBAC rights to adds some clusterlevel resources. It currently still complains with the same error:

Cluster level ClusterIssuer "letsencrypt-staging" can not be managed when in namespaced mode

Which isn't true, if Argo is enabled (with RBAC) to manage the CRDs.

[nnachefski]

I'm getting the same issue....

ComparisonError: Cluster level Namespace "elastic-test" can not be managed when in namespaced mode

ServiceAccount has cluster-wide policy. this whole thing worked fine until the auto-update on my argocd cluster/operator that occurred today.

[nnachefski]

Ugh, i just checked on my other "prod" clusters for a different customer... same issue. broke everything.

[magf]

Ugh, i just checked on my other "prod" clusters for a different customer... same issue. broke everything.

are you done that?

Add a label to the namespace your application is deployed in so that the Argo CD instance in the openshift-gitops namespace can manage it:

$ oc label namespace spring-petclinic argocd.argoproj.io/managed-by=openshift-gitops

it was described here

[Cuball0]

ServiceAccount has cluster-wide policy. this whole thing worked fine until the auto-update on my argocd cluster/operator that occurred today.

We are experiencing the sames issues, after the auto-update of ArgoCd everything is messed up, apparently ArgoCD cannot manage the namespaces anymore.

Namespace "yyyy" for ConfigMap "jenkins-rrrrr" is not managed

Did you find a solution for this? We are running ArgoCD v2.1.6 in Openshift 4.6

[jaipalkomakula]

Hello, Any solution for this issue: Namespace "yyyy" for "resources" "jenkins-rrrrr" is not managed

[gkmahendran09]

We are able to fix this issue by adding the below key/value in the argocd-default-cluster-config secret.

clusterResources: true

We are running ArgoCD v2.1.6 in Openshift 4.7.30

[Cuball0]

We are using the ArgoCD Operator (in Openshift). I'm not sure, but it could be that the operator reconciles the default secret.

We solved it by adding

config:
    env:
      - name: ARGOCD_CLUSTER_CONFIG_NAMESPACES
        value: argocd

the following env variable to the Subscription of the Operator (where the value is the namespace of the argocd installation)

[patchon]

We are able to fix this issue by adding the below key/value in the argocd-default-cluster-config secret.

clusterResources: true

We are running ArgoCD v2.1.6 in Openshift 4.7.30

Thanks a lot for the tip @gkmahendran09, really saved our day.
We have been struggling with this issue for some time now, and what it boiled down to in our case, was actually two parameters.

1 ) The one you are mentioning here, setting the "clusterResources: true" in the argocd-cluster secret. This, to my understanding, "tells" argocd that it's allowed to manage resources outside its own namespace.

However, doing so, we were faced with a new error, telling us,
"message\":\"Namespace \\\"foo\\\" for LimitRange \\\"bar\\\" is not managed\",\"type\":\"ComparisonError\" which is becuase its still configured via the operator, to only "manage" resources within the namespace it recides in. So even if its allowed via the "clusterResources: true" parameter, it doesn't have any knowledge about the resources outside its own namespace.

2 ) To give argocd this knowledge, you have to provide the operator with the environment variable, as stated by user Cuball0 above. Ie., edit the subscription for the operator like

$ > oc get subscriptions.operators.coreos.com argocd-operator 

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: argocd
  namespace: argocd
spec:
  channel: alpha
  name: argocd-operator
  source: operatorhubio-catalog
  sourceNamespace: olm
  config:
    env:
    - name: ARGOCD_CLUSTER_CONFIG_NAMESPACES
      value: <argocd-namespaces(s)>

where <argocd-namespaces(s)> should be replaced by a comma separated list of namespaces in which you have argocd-instances that should manage cluster-wide-resources (ie. it should not be a list of namespaces you want to manage).

I've had issues finding this in the documentaion, but thanks to you we are all good again with our "namespaced argocd which handles cluster-wide resources".

Hope this helps somebody else and please correct me if I'm wrong about something.

😄 👍 💯 🚀 🥇 🍰

[patchon]

We are able to fix this issue by adding the below key/value in the argocd-default-cluster-config secret.

clusterResources: true

Hm, I noticed today that this secret actually gets deleted when setting the environment variable in the subscription.
So it seems that you only would need to update the subsciption to get desired functionality.

Just thought I should share this knowledge.

[crenshaw-dev]

Note that argocd-default-cluster-config is a feature of the Argo CD Operator.

If you're not using the Argo CD Operator but are using declarative cluster configs, use the clusterResource key. I've opened a PR to document that key.

[ruibinghao]

Thanks for saving my day as well since I ran into the same issue after standing up ArgoCD using Openshift GitOps operator. Just curious is there a parameter you can set to enable the cluster resources for the in-build cluster while bringing up a brand new operator for the first time? Or this is something you need to change afterwards?