Being able to keep resources when gitRepo is deleted

[StrongMonkey]

Today when gitRepo is deleted, assciated bundles and underlying resources are being deleted. This can cause cascading failure if user deletes their gitRepo accidently, or removed their bundles. A possible solution is that we can keep the resource by default when gitRepo/bundle is deleted, and only clean up resources when there is purge: true set on gitRepo.

[erSitzt]

Not sure if it fits here, but my issue here rancher/rancher#35987 causes similar problems... and should not delete anything without a warning/confirmation.

Im not sure if manual removal of a label has the same result ?

[0x4c6565]

This would be massively beneficial for those using 'App of Apps' pattern (GitRepo resource creating more GitRepo resources). Currently when this GitRepo is deleted, all child GitRepo resources (and their bundles) are removed.

[puffitos]

This would also be beneficial in case you don't want the resources to get deleted. Image if you're managing CRDs with Fleet and then for some reason you have to make a change to the way you're managing the CRDs, or just wanna stop managing them with Fleet. Deleting the GitRepo would delete all CRDs and all custom resources based on those CRDs...

An option to be able to "orphan" the resources and then bind them to another GitRepo would be ideal :)

[atsai1220]

I wonder if one could add annotations to resources so Fleet does not delete them (since it is using Helm underneath Tekton).

https://helm.sh/docs/howto/charts_tips_and_tricks/#tell-helm-not-to-uninstall-a-resource

kind: Secret
metadata:
  annotations:
    "helm.sh/resource-policy": keep

[raulcabello]

@atsai1220 yes you can use the helm.sh/resource-policy: keep annotation to keep the resources. Keep in mind that the helm.sh/resource-policy: keep annotation is currently only used if it is included in the rendered manifests of the last deployed release. That means that executing kubectl annotate <type> <name> helm.sh/resource-policy=keep directly will not work.

[atsai1220]

@raulcabello ty!

We used that annotation on CRDs that are part of the chart and has worked well.

This may not work well if CRDs are part of a job generated by the chart.

Lastly, we were able to remove the gitrepo and keep the resources by using these manual steps:

1. Scale down fleet-agent on down stream clusters
2. Delete Helm secret for your application in downstream cluster
3. Delete gitrepo object in local cluster
4. Scale up fleet-agent in downstream cluster

[sowmyav27]

@kkaempf Need info/design doc for determining QA LOE

[raulcabello]

@rancher/qa

Additionnal QA

Problem

There is no way of keeping all resources created by fleet when a GitRepo or Bundle is deleted

Solution

New field keepResources added to GitRepo and Bundle. If keepResources is true resources will be keep after deleting the GitRepo or Bundle, helm release secrets will be removed instead.

All Bundles created from a GitRepo will get the keepResources value from the GitRepo. Additionally, keepResources can be specified in the fleet.yaml

Testing

Engineering Testing

Manual Testing

• Create a GitRepo with keepResources: true
• Remove the GitRepo
• Verify resources are still present, and helm release is not (this can be checked with helm ls)
• Create the same GitRepo
• Resources should be adopted. That means that we should see the same resources in the new Bundle created without being recreated.

---

• Create a GitRepo with keepResources: false that points to a git repo that contains keepResources: true in the fleet.yaml
• Remove the GitRepo
• Verify resources are still present, and helm release is not (this can be checked with helm ls)
• Create the same GitRepo
• Resources should be adopted. That means that we should see the same resources in the new Bundle created without being recreated.

Automated Testing

• E2E test

[kkaempf]

SURE-5008, SURE-5925, SURE-4920

[w13915984028]

@raulcabello

For existing managedchart, if we patch it with keepResources: true, will it effectively patch all downstreams bundles,bundledeplyments and other objects ? and delete the managedchart will keep all resources. thanks.

[raulcabello]

@w13915984028 no. keepResources was added to GitRepo and Bundles but no to ManagedChart.
Maybe you could manually set keepResources to true to the Bundles created by the ManagedChart, but I'm not sure how ManagedChart works in Rancher. It might find the change in the Bundle and remove the keepResources field. I would recommend to create an issue in Rancher to add support for keepResources in ManagedChart

ManagedChart is a Rancher resource not a Fleet resource

[olblak]

This fix is available for testing in https://github.com/rancher/fleet/releases/tag/v0.7.0-AGENT-rc.1 but not yet available in Rancher RC

[sbulage]

QA TEST PLAN

Scenarios

Scenario	Test Case
1	Test whether resources are not removed after the GitRepo deleted when keepResources: true used in the GitRepo yaml file.
2	Test resources are not present, when GitRepo is deleted. (without using keepResources: true)
3	Test resources are present after adding keepResources: true to existing GitRepo and delete GitRepo afterwards.

[sbulage]

Verified first two test cases QE Test Plan in above comment.

Rancher Version: 2.7.5-rc1
fleet version: 0.7.0-rc.2

TEST RESULT

Scenarios

Scenario	Test Case	Result
1	Test whether resources are not removed after the GitRepo deleted when keepResources: true used in the GitRepo yaml file.	✅
2	Test resources are not present, when GitRepo is deleted. (without using keepResources: true)	✅
3	Test resources are present after adding keepResources: true to existing GitRepo and delete GitRepo afterwards.	✅

REPRO STEPS

Scenario 1

1. Created a GitRepo via cli using https://github.com/sbulage/test-fleet/blob/main/keep-resources/keep-resources/gitrepo.yaml
2. Navigate to the Cluster -> Workloads -> deployment.
3. Checked that GitRepo and resources are created.
4. Delete the GitRepo.
5. Navigate to the Cluster -> Workloads -> deployment.
6. Verified that the deployement is present.

Scenario 2

1. Created a GitRepo via cli using https://github.com/sbulage/test-fleet/blob/main/keep-resources/do-not-keep-resources/gitrepo.yaml
2. Navigate to the Cluster -> Workloads -> deployment.
3. Checked that GitRepo and resources are created.
4. Delete the GitRepo.
5. Verified that the GitRepo and deployement gets deleted.

Scenario 3

1. Created a GitRepo via cli using https://github.com/sbulage/test-fleet with path nginx-helm.
2. Navigate to the Cluster -> Workloads -> deployment.
3. Wait for the resource to be created.
4. Edit GitRepo as YAML by adding keepResources: true in spec
5. Delete the GitRepo.
6. Navigate to the Cluster -> Workloads -> deployment.
7. Verified that the deployement is present.