Extend namespace filtering to all operations on namespaced resources #1668

2opremio · 2019-01-17T04:59:35Z

Addresses part of #1471

Deprecate and rename flag --kubernetes-namespace-whitelist to ~~--kubernetes-allow-namespace~~ --k8s-allow-namespace
Honor --k8s-allow-namespace for namespaced kubernetes resources in all flux operations.
cluster-global resources cluster-global resources (i.e. resources without an associated namespace) won't be filtered out by this change (they will keep being synced).

2opremio · 2019-01-17T15:29:34Z

I ran into a problem when filtering resources by the namespace in their identifier.

Namespace filtering cannot distinguish cluster-wide resources and resources in the default namespace, just like mentioned at #1442 (comment) .

When the namespace of a resource identifier comes from ...

... a manifest file, cluster-wide resources are not distinguishable from resources mapped (explicitly or implicitly) to the default namespace. This is because empty namespaces are replaced with default
... a cluster resource, cluster-wide resources are not distinguishable from namespaced resources missing an explicit namespace (they are part of default namespace).

(2) is slightly less problematic because, AFAIU, the only cluster resources we currently read are controllers, which are all namespaced.

My suggestion would be to create the following invariant for namespaces in resource identifiers:

"" indicates cluster-wide resource
"default" indicates mapped to the default namespace, either explicitly (the yaml in git or from the cluster contains it) or implicitly (the yaml in git or the cluster doesn't include a namespace)

More specifically, the changes needed are: when constructing resource identifiers ...

... of a manifest file in git. Stop using "default" as the namespace fo cluster-wide resources
... of a cluster resource. Start using "default" as the namespace of resources in the default namespace which don't state it explicitly

squaremo · 2019-01-17T17:45:14Z

"default" indicates mapped to the default namespace, either explicitly (the yaml in git or from the cluster contains it) or implicitly (the yaml in git or the cluster doesn't include a namespace)

default is the name of a namespace, so you would not be able to tell whether this meant "implicitly mapped to the default namespace" (which might not be default) or "explicitly in default".

squaremo · 2019-01-17T17:51:35Z

For those two cases you've put above: in (2), I think the API server will always give you a namespace if the resource is namespaced, and never if it's not. So an empty namespace is a reliable guide to cluster-scoped resources.

But for 1), the only way to know is to check with the API server whether the particular GroupVersionKind is namespaced or not; something which I was able to dodge in #1442 since I don't actually care what the namespace is, so long I can use it to relate resources to manifests (though it would be cleaner if I could give things accurate IDs).

(CODA: #1442 did end up querying the API for resource namespacedness)

cluster/kubernetes/manifests.go

http/client/client.go

integrations/client/clientset/versioned/clientset.go

2opremio · 2019-01-23T00:11:36Z

I am done with the code. I want to do some further high-level testing on a Kubernetes cluster, but it's ready to review.

cluster/kubernetes/scoper.go

cluster/kubernetes/resource/resource.go

cluster/mock.go

cluster/scoper.go

2opremio · 2019-02-08T14:01:38Z

This PR is blocked by #1442 (they have a lot in common and I agreed with @squaremo to get #1442 merged first)

2opremio · 2019-03-06T12:49:42Z

@squaremo this is ready again, PTAL

squaremo

I need to have a think about this, so only comments for now -- sorry 🌵

cluster/kubernetes/kubernetes.go

cluster/cluster.go

daemon/daemon.go

squaremo · 2019-03-14T17:15:11Z

@2opremio That commit is what I had in mind (hopefully clearer in code than in comments!). See what you think ..

2opremio · 2019-03-14T23:06:05Z

The "lowest" we can filter resources by namespace, the better, since it will cover more code paths.

My initial proposal was very similar (to filter out disallowed namespaces when loading the manifests) but it was discarded, see #1668 (comment) .

I recall you ended up coming with a scenario in which it could be a problem. Now I don't recall which one and I can't find it, maybe you shared it privately.

At the very least, it will lead to confusing error messages when trying to edit policies from workload belonging to disallowed namespaces. If we remove the manifests when loading, the error will be workload not found whereas if we don't we can provide a proper message such as workload belongs to disallowed namespace.

PS: We should also merge the code of NamespaceAllowed and IsAllowedResource

squaremo · 2019-03-18T12:23:42Z

My initial proposal was very similar (to filter out disallowed namespaces when loading the manifests) but it was discarded, see #1668 (comment) .

I recall you ended up coming with a scenario in which it could be a problem. Now I don't recall and I can't find it, maybe you shared it privately.

Where we ended up on that (in #1442) was that it was an acceptable mid-way point to leave the parsing as generic (that's cluster/kubernetes/resource/load.go and friends), but do post-hoc filtering on the result (that's cluster/kubernetes/manifest.go). Mind you, the waters are muddy enough now that there may not be much of a distinction to make.

At the very least, it will lead to confusing error messages when trying to edit policies from workload belonging to disallowed namespaces. If we remove the manifests when loading, the error will be workload not found whereas if we don't we can provide a proper message such as workload belongs to disallowed namespace.

That's a fair objection. It's the usual trade-off of reliable enforcement versus being helpful to users. In this case, I don't think it's important to hide from a user the fact of namespaces being blocked, and I do think we can do better at closing down opportunities for accidentally side-stepping the check.

What about if the check against the allowed namespaces happened in cluster/kubernetes/Manifest.UpdateImage and cluster/kubernetes/Manifest.UpdatePolicy? Then it can return a more helpful error, while still being in the narrow bit of the interface.

I'll have another go at my amendment ..

cluster/kubernetes/kubernetes.go

squaremo · 2019-03-20T17:19:17Z

Rebased on master, and removed my extra commit. I've tried out a few scenarios:

Start from scratch with repo and --k8s-allow-namespace=hello
- only hello namespace and things in hello namespace get created, even though there are manifests for other things
changed --k8s-allow-namespace=default and ran with --sync-garbage-collection
- new resources are created, old resources are left, as they aren't considered for GC
list-images and list-workloads both show just the namespaces allowed
release --all acts on only the namespaces allowed, even if others use the image in question

So all good so far! Will think of some tricksier tests, next ..

2opremio · 2019-03-20T23:52:24Z

I did some tests myself, but it's great that you are being thorough. Thanks!!

2opremio · 2019-03-22T15:42:41Z

@squaremo I think I've dealt with the comments (except for #1668 (comment) which I don't know how to address).

PTAL

2opremio · 2019-03-25T17:17:47Z

@squaremo I tested this PR against https://github.com/2opremio/locked-down-flux to see how it behaved with a namespace-locking RBAC and everything worked as expected (no errors in the logs and the example workload was created as expected)

$ kubectl -n flux-system logs flux-58fd7cbd99-n5sjn 
Flag --k8s-namespace-whitelist has been deprecated, changed to --k8s-allow-namespace, use that instead
ts=2019-03-25T17:00:39.817983756Z caller=main.go:167 version=1471-extend-ns-filtering-cb4ad60c
ts=2019-03-25T17:00:39.882144008Z caller=main.go:297 component=cluster identity=/var/fluxd/keygen/identity
ts=2019-03-25T17:00:39.882313951Z caller=main.go:298 component=cluster identity.pub="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7OhlwhU0Hwp3kxyUe+FUWIQ3S127QIlQFnj2m9HUDfb1bpkRDK4gRVBM6JiYBTFj4/YAfGcWck4mPt88SgFIPLQzUYbJC1NoKjMVjH/VbAxP1XoJIZMlzHMd2MztcX2/q0QrCFAYhF60OhbTFu3tuZpqSMeZ0DK/KAmCVevXslaKztpayUqatBnmxhm3C5EM22jAwQZ/yxdN9+s8DZNiJS+2g5YwWVJqWB3WisjT9/p6MT/wjW+fiBbuQDpr3xAp7z6tAnr1/yChqmY7RAvTVsyXaNDtTEgXS9+FAgNmo3wf6m4qpT2H42/E6mY4rjDlPYxlf1bgkpHk8eR4I7qbJ"
ts=2019-03-25T17:00:39.882461841Z caller=main.go:299 component=cluster host=https://10.96.0.1:443 version=kubernetes-v1.13.0
ts=2019-03-25T17:00:39.88267732Z caller=main.go:311 component=cluster kubectl=/usr/local/bin/kubectl
ts=2019-03-25T17:00:39.883574849Z caller=main.go:322 component=cluster ping=true
ts=2019-03-25T17:00:45.150716883Z caller=aws.go:69 component=aws warn="no AWS region configured, or detected as cluster region" err="RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/placement/availability-zone: dial tcp 169.254.169.254:80: connect: connection refused"
ts=2019-03-25T17:00:45.151020082Z caller=main.go:347 warning="AWS authorization not used; pre-flight check failed"
ts=2019-03-25T17:00:45.160827588Z caller=main.go:452 url=git@github.com:2opremio/locked-down-flux.git user="Weave Flux" email=support@weave.works signing-key= sync-tag=flux-sync notes-ref=flux set-author=false
ts=2019-03-25T17:00:45.160897014Z caller=main.go:509 upstream="no upstream URL given"
ts=2019-03-25T17:00:45.16101119Z caller=main.go:538 metrics-addr=:3031
ts=2019-03-25T17:00:45.162781302Z caller=loop.go:90 component=sync-loop err="git repo not ready: git repo has not been cloned yet"
ts=2019-03-25T17:00:45.16283077Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-03-25T17:00:45.162848493Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-03-25T17:00:45.16403101Z caller=main.go:530 addr=:3030
ts=2019-03-25T17:00:45.728458176Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=1.11.0
ts=2019-03-25T17:05:44.941893242Z caller=loop.go:90 component=sync-loop err="git repo not ready: git clone --mirror: fatal: Could not read from remote repository."
ts=2019-03-25T17:05:44.942947035Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-03-25T17:05:44.943062658Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-03-25T17:09:42.250149846Z caller=loop.go:103 component=sync-loop event=refreshed url=git@github.com:2opremio/locked-down-flux.git branch=deploy HEAD=723b91549434575348e4b67affdc857478c5894f
ts=2019-03-25T17:09:42.494560317Z caller=sync.go:416 component=cluster method=Sync cmd=apply args= count=3
ts=2019-03-25T17:09:43.034708957Z caller=sync.go:482 component=cluster method=Sync cmd="kubectl apply -f -" took=539.883683ms err=null output="service/echoserver created\ndeployment.apps/echoserver created\nnetworkpolicy.networking.k8s.io/echoserver created"
ts=2019-03-25T17:09:45.680724239Z caller=warming.go:268 component=warmer info="refreshing image" image=gcr.io/google-samples/hello-app tag_count=2 to_update=2 of_which_refresh=0 of_which_missing=2
ts=2019-03-25T17:09:46.424009606Z caller=warming.go:364 component=warmer updated=gcr.io/google-samples/hello-app successful=2 attempted=2
ts=2019-03-25T17:09:46.424208427Z caller=images.go:17 component=sync-loop msg="polling images"

$ kubectl -n helloworld get all
NAME                             READY   STATUS    RESTARTS   AGE
pod/echoserver-bf95b6849-dp4kg   1/1     Running   0          10m

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/echoserver   ClusterIP   10.110.52.118   <none>        8080/TCP   10m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/echoserver   1/1     1            1           10m

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/echoserver-bf95b6849   1         1         1       10m

squaremo · 2019-03-25T17:21:14Z

I tested this PR against https://github.com/2opremio/locked-down-flux

Lovely! What happens if you tell it to allow a namespace it doesn't have RBAC access to?

2opremio · 2019-03-26T00:55:21Z

Good suggestion. After creating a namespace forbidden by RBAC and included in the whitelist I noticed that flux was creating everything in the namespace anyways.

After quite some digging, it turns out that the Kubernetes installation provided by Docker For Mac makes all service accounts cluster-admins, sigh:

$ kubectl get clusterrolebinding -o yaml docker-for-desktop-binding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2019-03-25T16:29:55Z"
  name: docker-for-desktop-binding
  resourceVersion: "436"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/docker-for-desktop-binding
  uid: 38fe7238-4f1b-11e9-b66a-025000000001
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts

I will retest tomorrow after removing that role binding.

PS: It's really frustrating that the RBAC groups are second-class citizens and are only documented through examples.

2opremio · 2019-03-26T01:26:55Z

I bit the bullet and everything works as expected:

The resources from a RBAC-allowed and whitelisted namespace (helloworld) are created just fine
The resources from a RBAC-disallowed and whitelisted namespace (helloworkd2) are not created and it doesn't abort the sync

That said, although we give a sensible error about not being able to access the helloworld2, subsequent kubectl apply errors are pretty horrible:

$ kubectl logs --namespace=flux-system flux-9b5fb66b7-526rz 
Flag --k8s-namespace-whitelist has been deprecated, changed to --k8s-allow-namespace, use that instead
ts=2019-03-26T01:16:01.725052284Z caller=main.go:169 version=1471-extend-ns-filtering-92db72a4
ts=2019-03-26T01:16:01.764536902Z caller=main.go:312 component=cluster identity=/etc/fluxd/ssh/identity
ts=2019-03-26T01:16:01.76731935Z caller=main.go:313 component=cluster identity.pub="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7OhlwhU0Hwp3kxyUe+FUWIQ3S127QIlQFnj2m9HUDfb1bpkRDK4gRVBM6JiYBTFj4/YAfGcWck4mPt88SgFIPLQzUYbJC1NoKjMVjH/VbAxP1XoJIZMlzHMd2MztcX2/q0QrCFAYhF60OhbTFu3tuZpqSMeZ0DK/KAmCVevXslaKztpayUqatBnmxhm3C5EM22jAwQZ/yxdN9+s8DZNiJS+2g5YwWVJqWB3WisjT9/p6MT/wjW+fiBbuQDpr3xAp7z6tAnr1/yChqmY7RAvTVsyXaNDtTEgXS9+FAgNmo3wf6m4qpT2H42/E6mY4rjDlPYxlf1bgkpHk8eR4I7qbJ"
ts=2019-03-26T01:16:01.767403888Z caller=main.go:314 component=cluster host=https://10.96.0.1:443 version=kubernetes-v1.13.0
ts=2019-03-26T01:16:01.76744863Z caller=main.go:326 component=cluster kubectl=/usr/local/bin/kubectl
ts=2019-03-26T01:16:01.769259121Z caller=main.go:337 component=cluster ping=true
ts=2019-03-26T01:16:07.132247759Z caller=aws.go:69 component=aws warn="no AWS region configured, or detected as cluster region" err="RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/placement/availability-zone: dial tcp 169.254.169.254:80: connect: connection refused"
ts=2019-03-26T01:16:07.132330617Z caller=main.go:362 warning="AWS authorization not used; pre-flight check failed"
ts=2019-03-26T01:16:07.133919183Z caller=main.go:467 url=git@github.com:2opremio/locked-down-flux.git user="Weave Flux" email=support@weave.works signing-key= sync-tag=flux-sync notes-ref=flux set-author=false
ts=2019-03-26T01:16:07.14674934Z caller=main.go:524 upstream="no upstream URL given"
ts=2019-03-26T01:16:07.14715081Z caller=main.go:553 metrics-addr=:3031
ts=2019-03-26T01:16:07.154751956Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-03-26T01:16:07.154995676Z caller=images.go:27 component=sync-loop msg="no automated workloads"
ts=2019-03-26T01:16:07.155227009Z caller=loop.go:90 component=sync-loop err="git repo not ready: git repo has not been cloned yet"
ts=2019-03-26T01:16:07.15584015Z caller=main.go:545 addr=:3030
ts=2019-03-26T01:16:07.203392013Z caller=kubernetes.go:284 component=cluster warning="cannot access allowed namespace" namespace=helloworld2 err="namespaces \"helloworld2\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"helloworld2\""
ts=2019-03-26T01:16:07.601941976Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=1.11.0
ts=2019-03-26T01:16:15.779525806Z caller=loop.go:103 component=sync-loop event=refreshed url=git@github.com:2opremio/locked-down-flux.git branch=master HEAD=c89a1b3362d904300f10a74c6daf3f2cdf900a18
ts=2019-03-26T01:16:15.871514063Z caller=sync.go:455 component=cluster method=Sync cmd=apply args= count=7
ts=2019-03-26T01:16:16.26625243Z caller=sync.go:521 component=cluster method=Sync cmd="kubectl apply -f -" took=394.659317ms err="running kubectl: Error from server (Forbidden): error when retrieving current configuration of:\nResource: \"/v1, Resource=namespaces\", GroupVersionKind: \"/v1, Kind=Namespace\"\nName: \"helloworld2\", Namespace: \"\"\nObject: &{map[\"apiVersion\":\"v1\" \"kind\":\"Namespace\" \"metadata\":map[\"annotations\":map[\"flux.weave.works/sync-checksum\":\"eff788baf5f8906b3879d666e60b5407f7b9f294\" \"kubectl.kubernetes.io/last-applied-configuration\":\"\"] \"labels\":map[\"flux.weave.works/sync-gc-mark\":\"sha256.cFRnOwOXCB_oNpi52QLlUkjvODSLBIo1ntBRfzoPqp0\"] \"name\":\"helloworld2\" \"namespace\":\"\"]]}\nfrom server for: \"STDIN\": namespaces \"helloworld2\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"helloworld2\"\nError from server (Forbidden): error when retrieving current configuration of:\nResource: \"/v1, Resource=services\", GroupVersionKind: \"/v1, Kind=Service\"\nName: \"echoserver\", Namespace: \"helloworld2\"\nObject: &{map[\"apiVersion\":\"v1\" \"kind\":\"Service\" \"metadata\":map[\"labels\":map[\"app\":\"echoserver\" \"flux.weave.works/sync-gc-mark\":\"sha256.3KBCW83dRp2s10Zkls_vLADf8Wl0E9Agzz8sIT3wHkU\"] \"name\":\"echoserver\" \"namespace\":\"helloworld2\" \"annotations\":map[\"flux.weave.works/sync-checksum\":\"24d87352fb027d306ce7b03676b6d723dc59a799\" \"kubectl.kubernetes.io/last-applied-configuration\":\"\"]] \"spec\":map[\"ports\":[map[\"protocol\":\"TCP\" \"targetPort\":'\\u1f90' \"name\":\"http\" \"port\":'\\u1f90']] \"selector\":map[\"app\":\"echoserver\" \"component\":\"echoserver\"] \"type\":\"ClusterIP\"]]}\nfrom server for: \"STDIN\": services \"echoserver\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"services\" in API group \"\" in the namespace \"helloworld2\"\nError from server (Forbidden): error when retrieving current configuration of:\nResource: \"apps/v1, Resource=deployments\", GroupVersionKind: \"apps/v1, Kind=Deployment\"\nName: \"echoserver\", Namespace: \"helloworld2\"\nObject: &{map[\"apiVersion\":\"apps/v1\" \"kind\":\"Deployment\" \"metadata\":map[\"annotations\":map[\"flux.weave.works/sync-checksum\":\"191df8fec32a1deda61e32f995d8ffb7ea160781\" \"flux.weave.works/tag.echoserver\":\"semver:*\" \"kubectl.kubernetes.io/last-applied-configuration\":\"\"] \"labels\":map[\"app\":\"echoserver\" \"component\":\"echoserver\" \"flux.weave.works/sync-gc-mark\":\"sha256.b3ah4IFHE0JJUNAUMz8Q7ikY7f9dDREvkAYx40-0FTA\"] \"name\":\"echoserver\" \"namespace\":\"helloworld2\"] \"spec\":map[\"replicas\":'\\x01' \"selector\":map[\"matchLabels\":map[\"app\":\"echoserver\" \"component\":\"echoserver\"]] \"strategy\":map[\"type\":\"RollingUpdate\"] \"template\":map[\"metadata\":map[\"labels\":map[\"app\":\"echoserver\" \"component\":\"echoserver\"]] \"spec\":map[\"automountServiceAccountToken\":%!q(bool=false) \"containers\":[map[\"securityContext\":map[\"allowPrivilegeEscalation\":%!q(bool=false)] \"image\":\"gcr.io/google-samples/hello-app:1.0\" \"name\":\"echoserver\" \"ports\":[map[\"protocol\":\"TCP\" \"containerPort\":'\\u1f90' \"name\":\"web\"]]]]]]]]}\nfrom server for: \"STDIN\": deployments.apps \"echoserver\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"deployments\" in API group \"apps\" in the namespace \"helloworld2\"\nError from server (Forbidden): error when retrieving current configuration of:\nResource: \"networking.k8s.io/v1, Resource=networkpolicies\", GroupVersionKind: \"networking.k8s.io/v1, Kind=NetworkPolicy\"\nName: \"echoserver\", Namespace: \"helloworld2\"\nObject: &{map[\"apiVersion\":\"networking.k8s.io/v1\" \"kind\":\"NetworkPolicy\" \"metadata\":map[\"annotations\":map[\"kubectl.kubernetes.io/last-applied-configuration\":\"\" \"flux.weave.works/sync-checksum\":\"d98993cf8b5516b417533ebeeb0cfabe5c7ff6fa\"] \"labels\":map[\"app\":\"echoserver\" \"flux.weave.works/sync-gc-mark\":\"sha256.yxyBpR0YQllFHVEBWXu_LxyedQX59j9IYMah3mi6rL4\"] \"name\":\"echoserver\" \"namespace\":\"helloworld2\"] \"spec\":map[\"ingress\":[map[\"ports\":[map[\"port\":'\\u1f90' \"protocol\":\"TCP\"]]]] \"podSelector\":map[\"matchLabels\":map[\"app\":\"echoserver\" \"component\":\"echoserver\"]] \"policyTypes\":[\"Ingress\" \"Egress\"]]]}\nfrom server for: \"STDIN\": networkpolicies.networking.k8s.io \"echoserver\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"networkpolicies\" in API group \"networking.k8s.io\" in the namespace \"helloworld2\"" output="service/echoserver created\ndeployment.apps/echoserver created\nnetworkpolicy.networking.k8s.io/echoserver created"
ts=2019-03-26T01:16:16.523525929Z caller=sync.go:521 component=cluster method=Sync cmd="kubectl apply -f -" took=256.930326ms err="running kubectl: Error from server (Forbidden): error when retrieving current configuration of:\nResource: \"/v1, Resource=namespaces\", GroupVersionKind: \"/v1, Kind=Namespace\"\nName: \"helloworld2\", Namespace: \"\"\nObject: &{map[\"apiVersion\":\"v1\" \"kind\":\"Namespace\" \"metadata\":map[\"annotations\":map[\"flux.weave.works/sync-checksum\":\"eff788baf5f8906b3879d666e60b5407f7b9f294\" \"kubectl.kubernetes.io/last-applied-configuration\":\"\"] \"labels\":map[\"flux.weave.works/sync-gc-mark\":\"sha256.cFRnOwOXCB_oNpi52QLlUkjvODSLBIo1ntBRfzoPqp0\"] \"name\":\"helloworld2\" \"namespace\":\"\"]]}\nfrom server for: \"STDIN\": namespaces \"helloworld2\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"helloworld2\"" output=
ts=2019-03-26T01:16:16.753540835Z caller=sync.go:521 component=cluster method=Sync cmd="kubectl apply -f -" took=229.928726ms err=null output="service/echoserver unchanged"
ts=2019-03-26T01:16:17.020017775Z caller=sync.go:521 component=cluster method=Sync cmd="kubectl apply -f -" took=266.312128ms err="running kubectl: Error from server (Forbidden): error when retrieving current configuration of:\nResource: \"/v1, Resource=services\", GroupVersionKind: \"/v1, Kind=Service\"\nName: \"echoserver\", Namespace: \"helloworld2\"\nObject: &{map[\"metadata\":map[\"annotations\":map[\"kubectl.kubernetes.io/last-applied-configuration\":\"\" \"flux.weave.works/sync-checksum\":\"24d87352fb027d306ce7b03676b6d723dc59a799\"] \"labels\":map[\"app\":\"echoserver\" \"flux.weave.works/sync-gc-mark\":\"sha256.3KBCW83dRp2s10Zkls_vLADf8Wl0E9Agzz8sIT3wHkU\"] \"name\":\"echoserver\" \"namespace\":\"helloworld2\"] \"spec\":map[\"ports\":[map[\"name\":\"http\" \"port\":'\\u1f90' \"protocol\":\"TCP\" \"targetPort\":'\\u1f90']] \"selector\":map[\"app\":\"echoserver\" \"component\":\"echoserver\"] \"type\":\"ClusterIP\"] \"apiVersion\":\"v1\" \"kind\":\"Service\"]}\nfrom server for: \"STDIN\": services \"echoserver\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"services\" in API group \"\" in the namespace \"helloworld2\"" output=
ts=2019-03-26T01:16:17.187479884Z caller=sync.go:521 component=cluster method=Sync cmd="kubectl apply -f -" took=167.164138ms err=null output="deployment.apps/echoserver configured"
ts=2019-03-26T01:16:17.414758158Z caller=sync.go:521 component=cluster method=Sync cmd="kubectl apply -f -" took=226.754625ms err="running kubectl: Error from server (Forbidden): error when retrieving current configuration of:\nResource: \"apps/v1, Resource=deployments\", GroupVersionKind: \"apps/v1, Kind=Deployment\"\nName: \"echoserver\", Namespace: \"helloworld2\"\nObject: &{map[\"spec\":map[\"selector\":map[\"matchLabels\":map[\"component\":\"echoserver\" \"app\":\"echoserver\"]] \"strategy\":map[\"type\":\"RollingUpdate\"] \"template\":map[\"metadata\":map[\"labels\":map[\"app\":\"echoserver\" \"component\":\"echoserver\"]] \"spec\":map[\"automountServiceAccountToken\":%!q(bool=false) \"containers\":[map[\"image\":\"gcr.io/google-samples/hello-app:1.0\" \"name\":\"echoserver\" \"ports\":[map[\"containerPort\":'\\u1f90' \"name\":\"web\" \"protocol\":\"TCP\"]] \"securityContext\":map[\"allowPrivilegeEscalation\":%!q(bool=false)]]]]] \"replicas\":'\\x01'] \"apiVersion\":\"apps/v1\" \"kind\":\"Deployment\" \"metadata\":map[\"name\":\"echoserver\" \"namespace\":\"helloworld2\" \"annotations\":map[\"flux.weave.works/sync-checksum\":\"191df8fec32a1deda61e32f995d8ffb7ea160781\" \"flux.weave.works/tag.echoserver\":\"semver:*\" \"kubectl.kubernetes.io/last-applied-configuration\":\"\"] \"labels\":map[\"app\":\"echoserver\" \"component\":\"echoserver\" \"flux.weave.works/sync-gc-mark\":\"sha256.b3ah4IFHE0JJUNAUMz8Q7ikY7f9dDREvkAYx40-0FTA\"]]]}\nfrom server for: \"STDIN\": deployments.apps \"echoserver\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"deployments\" in API group \"apps\" in the namespace \"helloworld2\"" output=
ts=2019-03-26T01:16:17.57742458Z caller=sync.go:521 component=cluster method=Sync cmd="kubectl apply -f -" took=184.507826ms err="running kubectl: Error from server (Forbidden): error when retrieving current configuration of:\nResource: \"networking.k8s.io/v1, Resource=networkpolicies\", GroupVersionKind: \"networking.k8s.io/v1, Kind=NetworkPolicy\"\nName: \"echoserver\", Namespace: \"helloworld2\"\nObject: &{map[\"kind\":\"NetworkPolicy\" \"metadata\":map[\"annotations\":map[\"flux.weave.works/sync-checksum\":\"d98993cf8b5516b417533ebeeb0cfabe5c7ff6fa\" \"kubectl.kubernetes.io/last-applied-configuration\":\"\"] \"labels\":map[\"app\":\"echoserver\" \"flux.weave.works/sync-gc-mark\":\"sha256.yxyBpR0YQllFHVEBWXu_LxyedQX59j9IYMah3mi6rL4\"] \"name\":\"echoserver\" \"namespace\":\"helloworld2\"] \"spec\":map[\"ingress\":[map[\"ports\":[map[\"port\":'\\u1f90' \"protocol\":\"TCP\"]]]] \"podSelector\":map[\"matchLabels\":map[\"app\":\"echoserver\" \"component\":\"echoserver\"]] \"policyTypes\":[\"Ingress\" \"Egress\"]] \"apiVersion\":\"networking.k8s.io/v1\"]}\nfrom server for: \"STDIN\": networkpolicies.networking.k8s.io \"echoserver\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"networkpolicies\" in API group \"networking.k8s.io\" in the namespace \"helloworld2\"" output=
ts=2019-03-26T01:16:17.764813618Z caller=sync.go:521 component=cluster method=Sync cmd="kubectl apply -f -" took=186.921197ms err=null output="networkpolicy.networking.k8s.io/echoserver unchanged"
ts=2019-03-26T01:16:17.765076931Z caller=loop.go:210 component=sync-loop err="<cluster>:namespace/helloworld2: running kubectl: Error from server (Forbidden): error when retrieving current configuration of:\nResource: \"/v1, Resource=namespaces\", GroupVersionKind: \"/v1, Kind=Namespace\"\nName: \"helloworld2\", Namespace: \"\"\nObject: &{map[\"apiVersion\":\"v1\" \"kind\":\"Namespace\" \"metadata\":map[\"annotations\":map[\"flux.weave.works/sync-checksum\":\"eff788baf5f8906b3879d666e60b5407f7b9f294\" \"kubectl.kubernetes.io/last-applied-configuration\":\"\"] \"labels\":map[\"flux.weave.works/sync-gc-mark\":\"sha256.cFRnOwOXCB_oNpi52QLlUkjvODSLBIo1ntBRfzoPqp0\"] \"name\":\"helloworld2\" \"namespace\":\"\"]]}\nfrom server for: \"STDIN\": namespaces \"helloworld2\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"helloworld2\"; helloworld2:service/echoserver: running kubectl: Error from server (Forbidden): error when retrieving current configuration of:\nResource: \"/v1, Resource=services\", GroupVersionKind: \"/v1, Kind=Service\"\nName: \"echoserver\", Namespace: \"helloworld2\"\nObject: &{map[\"metadata\":map[\"annotations\":map[\"kubectl.kubernetes.io/last-applied-configuration\":\"\" \"flux.weave.works/sync-checksum\":\"24d87352fb027d306ce7b03676b6d723dc59a799\"] \"labels\":map[\"app\":\"echoserver\" \"flux.weave.works/sync-gc-mark\":\"sha256.3KBCW83dRp2s10Zkls_vLADf8Wl0E9Agzz8sIT3wHkU\"] \"name\":\"echoserver\" \"namespace\":\"helloworld2\"] \"spec\":map[\"ports\":[map[\"name\":\"http\" \"port\":'\\u1f90' \"protocol\":\"TCP\" \"targetPort\":'\\u1f90']] \"selector\":map[\"app\":\"echoserver\" \"component\":\"echoserver\"] \"type\":\"ClusterIP\"] \"apiVersion\":\"v1\" \"kind\":\"Service\"]}\nfrom server for: \"STDIN\": services \"echoserver\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"services\" in API group \"\" in the namespace \"helloworld2\"; helloworld2:deployment/echoserver: running kubectl: Error from server (Forbidden): error when retrieving current configuration of:\nResource: \"apps/v1, Resource=deployments\", GroupVersionKind: \"apps/v1, Kind=Deployment\"\nName: \"echoserver\", Namespace: \"helloworld2\"\nObject: &{map[\"spec\":map[\"selector\":map[\"matchLabels\":map[\"component\":\"echoserver\" \"app\":\"echoserver\"]] \"strategy\":map[\"type\":\"RollingUpdate\"] \"template\":map[\"metadata\":map[\"labels\":map[\"app\":\"echoserver\" \"component\":\"echoserver\"]] \"spec\":map[\"automountServiceAccountToken\":%!q(bool=false) \"containers\":[map[\"image\":\"gcr.io/google-samples/hello-app:1.0\" \"name\":\"echoserver\" \"ports\":[map[\"containerPort\":'\\u1f90' \"name\":\"web\" \"protocol\":\"TCP\"]] \"securityContext\":map[\"allowPrivilegeEscalation\":%!q(bool=false)]]]]] \"replicas\":'\\x01'] \"apiVersion\":\"apps/v1\" \"kind\":\"Deployment\" \"metadata\":map[\"name\":\"echoserver\" \"namespace\":\"helloworld2\" \"annotations\":map[\"flux.weave.works/sync-checksum\":\"191df8fec32a1deda61e32f995d8ffb7ea160781\" \"flux.weave.works/tag.echoserver\":\"semver:*\" \"kubectl.kubernetes.io/last-applied-configuration\":\"\"] \"labels\":map[\"app\":\"echoserver\" \"component\":\"echoserver\" \"flux.weave.works/sync-gc-mark\":\"sha256.b3ah4IFHE0JJUNAUMz8Q7ikY7f9dDREvkAYx40-0FTA\"]]]}\nfrom server for: \"STDIN\": deployments.apps \"echoserver\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"deployments\" in API group \"apps\" in the namespace \"helloworld2\"; helloworld2:networkpolicy/echoserver: running kubectl: Error from server (Forbidden): error when retrieving current configuration of:\nResource: \"networking.k8s.io/v1, Resource=networkpolicies\", GroupVersionKind: \"networking.k8s.io/v1, Kind=NetworkPolicy\"\nName: \"echoserver\", Namespace: \"helloworld2\"\nObject: &{map[\"kind\":\"NetworkPolicy\" \"metadata\":map[\"annotations\":map[\"flux.weave.works/sync-checksum\":\"d98993cf8b5516b417533ebeeb0cfabe5c7ff6fa\" \"kubectl.kubernetes.io/last-applied-configuration\":\"\"] \"labels\":map[\"app\":\"echoserver\" \"flux.weave.works/sync-gc-mark\":\"sha256.yxyBpR0YQllFHVEBWXu_LxyedQX59j9IYMah3mi6rL4\"] \"name\":\"echoserver\" \"namespace\":\"helloworld2\"] \"spec\":map[\"ingress\":[map[\"ports\":[map[\"port\":'\\u1f90' \"protocol\":\"TCP\"]]]] \"podSelector\":map[\"matchLabels\":map[\"app\":\"echoserver\" \"component\":\"echoserver\"]] \"policyTypes\":[\"Ingress\" \"Egress\"]] \"apiVersion\":\"networking.k8s.io/v1\"]}\nfrom server for: \"STDIN\": networkpolicies.networking.k8s.io \"echoserver\" is forbidden: User \"system:serviceaccount:flux-system:flux\" cannot get resource \"networkpolicies\" in API group \"networking.k8s.io\" in the namespace \"helloworld2\""
ts=2019-03-26T01:16:17.767684483Z caller=daemon.go:624 component=daemon event="Sync: c89a1b3, <cluster>:namespace/helloworld2, helloworld2:deployment/echoserver, helloworld2:networkpolicy/echoserver, helloworld2:service/echoserver, helloworld:deployment/echoserver, helloworld:networkpolicy/echoserver, helloworld:service/echoserver" logupstream=false
ts=2019-03-26T01:16:20.95148383Z caller=loop.go:441 component=sync-loop tag=flux-sync old= new=c89a1b3362d904300f10a74c6daf3f2cdf900a18
ts=2019-03-26T01:16:22.190745263Z caller=loop.go:103 component=sync-loop event=refreshed url=git@github.com:2opremio/locked-down-flux.git branch=master HEAD=c89a1b3362d904300f10a74c6daf3f2cdf900a18

That said, that's something unrelated to this PR.

* Rename `getAllowedNamespaces()` to `getAllowedAndExistingNamespaces()` * Remove redundant namespace check * Check for namespace existence when syncing

squaremo

This is looking good to me 🥇 💯 🍍

+234 thoroughly discussed lines!

2opremio force-pushed the 1471-extend-ns-filtering branch 2 times, most recently from f060e01 to 752d7b2 Compare January 17, 2019 05:08

2opremio mentioned this pull request Jan 17, 2019

Add option to limit helm-op to a single namespace #1664

Merged

2opremio force-pushed the 1471-extend-ns-filtering branch 2 times, most recently from e3f9e83 to dda60d7 Compare January 17, 2019 15:13

2opremio commented Jan 19, 2019

View reviewed changes

cluster/kubernetes/manifests.go Outdated Show resolved Hide resolved

2opremio mentioned this pull request Jan 21, 2019

Namespaces blocklist #1677

Closed

2opremio force-pushed the 1471-extend-ns-filtering branch 2 times, most recently from 00570c5 to 84a3850 Compare January 22, 2019 02:59

squaremo reviewed Jan 22, 2019

View reviewed changes

http/client/client.go Outdated Show resolved Hide resolved

squaremo reviewed Jan 22, 2019

View reviewed changes

integrations/client/clientset/versioned/clientset.go Outdated Show resolved Hide resolved

2opremio force-pushed the 1471-extend-ns-filtering branch 2 times, most recently from 18cbe9f to a936989 Compare January 22, 2019 21:07

2opremio changed the title ~~[WIP] Extend namespace filtering to all operations on namespaced resources~~ Extend namespace filtering to all operations on namespaced resources Jan 23, 2019

squaremo reviewed Jan 24, 2019

View reviewed changes

cluster/kubernetes/scoper.go Outdated Show resolved Hide resolved

squaremo reviewed Jan 24, 2019

View reviewed changes

cluster/kubernetes/resource/resource.go Outdated Show resolved Hide resolved

squaremo reviewed Jan 24, 2019

View reviewed changes

cluster/mock.go Outdated Show resolved Hide resolved

squaremo reviewed Jan 24, 2019

View reviewed changes

cluster/scoper.go Outdated Show resolved Hide resolved

2opremio mentioned this pull request Jan 28, 2019

Delete resources no longer in git #1442

Merged

11 tasks

2opremio added the blocked label Feb 8, 2019

2opremio removed the blocked label Feb 27, 2019

2opremio changed the title ~~Extend namespace filtering to all operations on namespaced resources~~ [WIP] Extend namespace filtering to all operations on namespaced resources Feb 27, 2019

2opremio requested review from hiddeco and removed request for hiddeco February 28, 2019 02:50

2opremio force-pushed the 1471-extend-ns-filtering branch from 759d954 to e71b52c Compare March 6, 2019 12:05

2opremio changed the title ~~[WIP] Extend namespace filtering to all operations on namespaced resources~~ Extend namespace filtering to all operations on namespaced resources Mar 6, 2019

2opremio force-pushed the 1471-extend-ns-filtering branch from c5706cc to cb4ad60 Compare March 6, 2019 12:48

2opremio requested a review from squaremo March 6, 2019 17:55

squaremo reviewed Mar 7, 2019

View reviewed changes

cluster/kubernetes/kubernetes.go Outdated Show resolved Hide resolved

cluster/kubernetes/kubernetes.go Show resolved Hide resolved

cluster/cluster.go Show resolved Hide resolved

daemon/daemon.go Outdated Show resolved Hide resolved

2opremio force-pushed the 1471-extend-ns-filtering branch 2 times, most recently from 1ea6920 to 47d75cf Compare March 13, 2019 16:10

zeeZ mentioned this pull request Mar 14, 2019

flux 1.11.0 no longer syncs without ClusterRole #1830

Closed

squaremo force-pushed the 1471-extend-ns-filtering branch from 5bf61b4 to 92db72a Compare March 20, 2019 15:44

squaremo reviewed Mar 20, 2019

View reviewed changes

cluster/kubernetes/kubernetes.go Show resolved Hide resolved

2opremio and others added 3 commits March 26, 2019 16:45

Replace k8s-namespace-whitelist by k8s-allow-namespace

8320236

Honor allowed namespaces in all cluster/git operations

1c81531

Fix multiple namespace filtering issues

a61e228

* Rename `getAllowedNamespaces()` to `getAllowedAndExistingNamespaces()` * Remove redundant namespace check * Check for namespace existence when syncing

2opremio force-pushed the 1471-extend-ns-filtering branch from 0696022 to a61e228 Compare March 26, 2019 15:46

Fix tests

0aaca0b

2opremio force-pushed the 1471-extend-ns-filtering branch from f2272f7 to 0aaca0b Compare March 26, 2019 17:43

squaremo approved these changes Mar 27, 2019

View reviewed changes

2opremio merged commit 0e601a6 into fluxcd:master Mar 27, 2019

2opremio deleted the 1471-extend-ns-filtering branch March 27, 2019 10:42

2opremio mentioned this pull request Nov 14, 2019

Gitops-engine branch doesn't set default namespaces properly #2614

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend namespace filtering to all operations on namespaced resources #1668

Extend namespace filtering to all operations on namespaced resources #1668

2opremio commented Jan 17, 2019 •

edited

2opremio commented Jan 17, 2019 •

edited

squaremo commented Jan 17, 2019

squaremo commented Jan 17, 2019 •

edited

2opremio commented Jan 23, 2019 •

edited

2opremio commented Feb 8, 2019

2opremio commented Mar 6, 2019

squaremo left a comment

squaremo commented Mar 14, 2019

2opremio commented Mar 14, 2019 •

edited

squaremo commented Mar 18, 2019 •

edited

squaremo commented Mar 20, 2019

2opremio commented Mar 20, 2019

2opremio commented Mar 22, 2019 •

edited

2opremio commented Mar 25, 2019 •

edited

squaremo commented Mar 25, 2019

2opremio commented Mar 26, 2019 •

edited

2opremio commented Mar 26, 2019 •

edited

squaremo left a comment

Extend namespace filtering to all operations on namespaced resources #1668

Extend namespace filtering to all operations on namespaced resources #1668

Conversation

2opremio commented Jan 17, 2019 • edited

2opremio commented Jan 17, 2019 • edited

squaremo commented Jan 17, 2019

squaremo commented Jan 17, 2019 • edited

2opremio commented Jan 23, 2019 • edited

2opremio commented Feb 8, 2019

2opremio commented Mar 6, 2019

squaremo left a comment

Choose a reason for hiding this comment

squaremo commented Mar 14, 2019

2opremio commented Mar 14, 2019 • edited

squaremo commented Mar 18, 2019 • edited

squaremo commented Mar 20, 2019

2opremio commented Mar 20, 2019

2opremio commented Mar 22, 2019 • edited

2opremio commented Mar 25, 2019 • edited

squaremo commented Mar 25, 2019

2opremio commented Mar 26, 2019 • edited

2opremio commented Mar 26, 2019 • edited

squaremo left a comment

Choose a reason for hiding this comment

2opremio commented Jan 17, 2019 •

edited

2opremio commented Jan 17, 2019 •

edited

squaremo commented Jan 17, 2019 •

edited

2opremio commented Jan 23, 2019 •

edited

2opremio commented Mar 14, 2019 •

edited

squaremo commented Mar 18, 2019 •

edited

2opremio commented Mar 22, 2019 •

edited

2opremio commented Mar 25, 2019 •

edited

2opremio commented Mar 26, 2019 •

edited

2opremio commented Mar 26, 2019 •

edited