Updating the monitoring namespace in the DSCI CR leaves resources running

[lugi0]

Describe the bug
Updating the monitoring namespace in the DSCI CR will trigger the creation of new monitoring pods in the new namespace, but the old namespace will not be delete and the pods will be left running there

To Reproduce
Steps to reproduce the behavior:

1. Go to DSCI CR
2. Enable monitoring in a namespace
3. Wait for monitoring pods/resources to be deployed in that namespace
4. Update DSCI CR to point to new monitoring namespace
5. Wait for monitoring pods/resources to be created in new namespace
6. See old pods still running in the old monitoring namespace

Expected behavior
old pods are stopped, resources are removed, optionally namespace is deleted (not sure, might point to a namespace the user wants to keep)

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[VaishnaviHire]

Note: This is same for applicationsNamespace as well.

An expected solution here is to ensure, application pods are cleaned up from the old namespace. We should not be deleting the previous namespace.

[zdtsw]

so the scope for this ticket is:
if we change applicationnamespace or monitoringnamespace, operator should keep the old namespace, but delete all resource in it.
correct?

[VaishnaviHire]

Correct, delete the components that were enabled.

[zdtsw]

why add rhods-2.4 to this ticket? @dchourasia

[zdtsw]

i think part of this issue has been done in #637 for application namespace.
but not for the monitoring namespace.

[lugi0]

@zdtsw Can you confirm this should not be tracked as "Ready for QA" in 2.4?

[zdtsw]

@zdtsw Can you confirm this should be tracked as "Ready for QA" in 2.4?

i dont think the scope of this issue is completely covered , also the "application" part did not included in downstream 2.4. so the "rhods-2.4" label should be removed.

[zdtsw]

did a test based on the current logic which already have "cleanup missing resourced in DSC" part.
changing namespace does not get resourced in the old namespace get cleaned up.

in our component logic, it is purely if enabled.
so a change namespace is not == to set to disable then enable again, so the cleanup wont be triggered by the reconcile.

[zdtsw]

2 cents from me:

do we really need to support this configable namespace?
e.g the integration feature we have with servicemesh and serving, namespace is fixed even users can modify it from API or OCM but this is only gonna break the feature.
unless this is a requirement that user can config monitoring and application namespace from the default one,
why should we add support for this.
rather than use documenation to mention this , a validation on the operator logic, say this value cannot be changed will be much simple.

[AjayJagan]

moved this issue to JIRA ref: https://issues.redhat.com/browse/RHOAIENG-403

[AjayJagan]

closing this as it is migrated to the JIRA