Integrations, IntegrationKits, IntegrationPlatforms are not owned by Namespaces

[phantomjinx]

When running the e2e tests, I found that despite the namespaces being deleted after each test, the integrations, integrationKits and integrationPlatforms are not. Leaving a cluster for some time leads to an accumulation of these resources and the performance of a global operator degrades since on starting up, it first has to reconcile all of these resources currently installed but not necessarily 'visible' (due to their namespace having been deleted).

$ kubectl get namespaces | grep test

NAME	STATUS	COMMENT
test-618a87b4-b8fb-496e-adbf-90510372faa6	Active	Left over from testing
test-b0a72b56-3909-43f4-8a65-43f47127f68e	Terminating	Process of being deleted
test-d37b5d1c-c9d1-4360-b731-aa5b6c76ce6f	Terminating	Process of being deleted

$ kubectl get integrationplatforms --all-namespaces

NAMESPACE	NAME	PHASE	COMMENT
openshift-operators	camel-k	Ready	Only 'valid' integrationplatform that should still be around
test-3c9e5d26-67a0-46da-9118-40415180fd1b	secondary	Ready	Should have been delected by now
test-97103a00-4a22-4ddd-b28b-4769d59c2f52	camel-k	Ready	Should be being deleted by terminating namespace

oc get integrationkits --all-namespaces

NAMESPACE	NAME	PHASE	TYPE	IMAGE
test-97103a00-4a22-4ddd-b28b-4769d59c2f52	kit-ca8hkn95ggpi4b4o0gbg	Ready	platform	image-registry.openshift-image-registry.svc:5000/test-97103a00-4a22-4ddd-b28b-4769d59c2f52/camel-k-kit-[.....]4fa8a3
test-97103a00-4a22-4ddd-b28b-4769d59c2f52	kit-ca8hm0idkk603440sajg	Ready	platform	image-registry.openshift-image-registry.svc:5000/test-97103a00-4a22-4ddd-b28b-4769d59c2f52/camel-k-kit-[.....]dc7b21

Both should have been deleted by terminating namespace?

oc get integrations --all-namespaces

NAMESPACE	NAME	PHASE	KIT	REPLICAS
test-02610ee1-f79e-4456-a8bc-0c9c42816575	display	Building Kit
test-02d27e76-90b8-4413-ae18-f4d87384d3cc	yaml	Building Kit
test-08581fd1-d69f-468e-a782-52efb74797b9	yaml	Building Kit
test-0a6f7e5e-bbbf-4bc6-be5d-d9a22064bd5b	service-binding	Running	kit-c921mmhj5m71rdvhut60	1
test-123faa08-1d51-4ecb-b4d1-1d6ea4e605f7	first	Running	kit-c7orcon7vfna3ammv0ug	1
test-123faa08-1d51-4ecb-b4d1-1d6ea4e605f7	second	Running	kit-c7orcon7vfna3ammv0ug	1
test-1a27e89f-5994-4a14-9d05-e7fd0c209925	yaml	Building Kit
test-1e86a177-e143-489b-8db8-9b62000cc195	petstore	Running	kit-c93nje5iadjqhe238cv0	0
test-1e97efd0-b18d-47de-9048-38b5cb8f82e3	podtest-knative1	Running	kit-c7grdq79fpis8lk5k250	1
test-1e97efd0-b18d-47de-9048-38b5cb8f82e3	podtest-knative2	Running	kit-c7grcuf9fpis8lk5k24g	1
test-28c56377-e4f4-4362-8f63-c03e8349a124	first	Running	kit-c7o73e2tqbcb2nde0vag	1
test-28c56377-e4f4-4362-8f63-c03e8349a124	second	Running	kit-c7o73e2tqbcb2nde0vag	1
.....
..... 50+ more
......

I notice that none of these resouces have the owner property assigning them either to the operator or to the namespace. Perhaps, adding this will ensure that deleting either the operator or namespace (depending on which is chosen) will then remove them.

Question is whether to attach them to the operator or to the namespace?

[tadayosi]

Good finding!

Talking about whether to add ownership to them, I thought the kits are intentionally left even after the operator is uninstalled, since they work as build caches and may be reused with another operator installation.

As for namespaces, I think deleting them should delete all resources under them:
https://kubernetes.io/docs/tasks/administer-cluster/namespaces/#deleting-a-namespace
If it doens't then probably it's because some resources are blocking, i.e. the operator is not cleaned up.

If the issue is only for the e2e testing, then adding owner references might be a bit intrusive; instead we could perhaps just make sure WithNewTestNamespace test helper clean up all kamel resources and the operator to resolve it.

[phantomjinx]

As for namespaces, I think deleting them should delete all resources under them:

Seems not. Some of the namespaces were deleted some time ago but if others could check their clusters to confirm ...

f the issue is only for the e2e testing, then adding owner references might be a bit intrusive; instead we could perhaps just make sure WithNewTestNamespace test helper clean up all kamel resources and the operator to resolve it.

Already added a cleanup-resources stage in the refactoring I'm doing but concerned that the global operator is being increasingly affected so becomes an issue of installation performance in user clusters rather than just confined to testing.

[tadayosi]

I did a simple test that created a new project (namespace) and then installed an operator and ran a few integrations (examples/languages/{Sample.java,simple.js,routes.yaml}) in it. After they were up I deleted the project without cleaning up any Camel K resources and all the resources were wiped out with the deleted project. So we already have the behaviour we expect.

If the namespace deletion is stuck, then probably there are some problems with the finalizers for some remaining resources. You can force deleting those resources like this:

kubectl patch ... --type=merge -p '{"metadata": {"finalizers":null}}'

but I think it should be specific to the settings of some tests and we should investigate why the finalizer for the resource is stuck in the tests.

[astefanutti]

I did a simple test that created a new project (namespace) and then installed an operator and ran a few integrations (examples/languages/{Sample.java,simple.js,routes.yaml}) in it. After they were up I deleted the project without cleaning up any Camel K resources and all the resources were wiped out with the deleted project. So we already have the behaviour we expect.

If the namespace deletion is stuck, then probably there are some problems with the finalizers for some remaining resources. You can force deleting those resources like this:

kubectl patch ... --type=merge -p '{"metadata": {"finalizers":null}}'

but I think it should be specific to the settings of some tests and we should investigate why the finalizer for the resource is stuck in the tests.

+1, namespaced resources, such as integrations, are deleted by the GC controller when their namespace is deleted. The problem, as outlined by @tadayosi, is that some namespaces are still pending deletion, e.g.:

test-b0a72b56-3909-43f4-8a65-43f47127f68e | Terminating | Process of being deleted

Finalizers set on resources within the namespace may be a possible cause.

[phantomjinx]

Closing as proven not required.