adopt Ark to provide backup and DR functionality

[danielhartnell]

https://github.com/heptio/ark

Ark gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes. Ark lets you:

Take backups of your cluster and restore in case of loss.
Copy cluster resources to other clusters.
Replicate your production environment for development and testing environments.

I did not know that this provided a simple way to copy cluster resources to other clusters. This should be a pretty useful tool for us.

[the-smooth-operator]

I'm starting to work in this one. What I plan to do:

• Create with Terraform the AWS resources needed: S3 bucket for the backup, IAM role, IAM policy and trust policy.
• Create the Kubernetes resources: Ark server, credentials and other configurations.
• Decide which Namespaces, PV etc have to be backed up.
• Create an scheduled backup running periodically.
• Add docs on Managing Ark, Disaster Recovery & Cluster Migration.
• Finally, together with other stakeholders, test one failure scenario and recovery.

[the-smooth-operator]

Sad note: Ark doesn't currently support encryption-at-rest vmware-tanzu/velero#434
I'll be looking if it's possible to let s3 take care of encrypting all files inside a bucket at rest, and if this doesn't break Ark functionality.

[the-smooth-operator]

While Ark doesn't provide a standard encryption-at-rest solution, it is compatible with AWS KMS. See https://github.com/heptio/ark/blob/master/docs/api-types/backupstoragelocation.md#config