Consider DaemonSet replacement

[peterbom]

This relates to #23. The concern is that by design, DaemonSet pods stay running, despite the fact they finish their useful work shortly after starting.

Others have pointed out the need for Kubernetes to support job-like behaviour that runs on all nodes, but so far there is no official Kubernetes resource type, and not much consensus about workarounds. Some suggestions are:

• Dysproz/DaemonJob
• AmitKumarDas/metac

Neither of the above appear to be actively maintained.

Some considerations:

• How much of a big change is this in terms of work involved?
• How will it affect existing users?
• Can we find a resource that we can trust, or will we have to roll our own?
• If we use a third party dependency, will that affect customers' willingness to deploy Periscope to their clusters?

[peterbom]

I opened this before noticing #101 - much of that discussion may be relevant here.

[Tatsinnit]

Nice, also further to share and just for idea sake, there is another way we can clean it, and its here: https://github.com/Tatsinnit/aks-periscope-porter-poc

[davidkydd]

could there be any value in adding a "triggering" capability to allow periscope to be rerun on demand when the DaemonSet is left installed and running? Wouldn't be very useful on its own but if it could then be triggered by some kind of automated node-change-detection (or even just cron schedule) then you've got a use case for leaving periscope installed, providing periodic snapshots

[peterbom]

If the concern is more around making it easy to re-trigger (rather than the pods hanging around) there might be some ideas there we could discuss.

I see there's a cross-platform filesystem watcher here, which could be used in conjunction with a command-line argument specifying a watch-path (which could maybe be set up as a volume that a cronjob could update). That's just my initial thought, but any suggestions for a simpler (or more K8s idiomatic) approaches welcome!

Note I'm not yet familiar enough with the code to confidently assess how achievable this is...