Explicit DeleteStrategy to prevent deleting resources

[sebastianrosch]

A misconfiguration in Faros or the GitOps repository it watches could potentially lead to all resources managed by Faros being deleted.

When the branch name or subpath was changed in the repository or the branch name or subpath was changed in GitTrack:

• new branch/subpath doesn’t exist → nothing happens
• new branch/subpath empty → nothing happens
• new branch/subpath that contains .yaml files (can be empty) → all cluster resources not in Git folder will be deleted ⚠️

Proposal:
We propose to be more explicit about deletes, and therefore propose to introduce a DeleteStrategy at the GitTrack level.

apiVersion: faros.pusher.com/v1alpha1
kind: GitTrack
metadata:
  name: faros-gittrack
spec:
  deployKey: ...
  reference: master
  repository: https://github.com/...
  subPath: deploy
  deleteStrategy: none

DeleteStrategy can be one of the following:

• cleanup: the currently implemented behavior, deleting resources that no longer exist in the repository (default)
• resource-state: resource-specific annotation (similar to our previous PR)
• never: for production environments that don’t expect deletes to ever happen

[tshak]

After some thinking there is one downside to the resource-state annotation approach: If someone were to kubectl apply -f the git folder it would recreate deleted items. An alternative may be to create a Tombstone CRD that describe a resource to be deleted. So deletion would be two step process: 1) removing the file that contains the resource that you want deleted and 2) creating a Tombstone with the name, kind, and namespace (if applicable) of the object that should be deleted.