CLI option for IAM Role in annotations

[tlkamp]

Describe the solution you'd like
Following the AWS Config instructions for using kube2iam enabled EKS clusters, we have to edit the Velero Deployment object after it is generated, either interactively through kubectl edit deploy/velero or through another means (creating and applying a patch for the deployment, using a mutating admission webhook to intercept the deployment); this is not great for deploying Velero in automation.

It would be ideal if the velero install ... command provided an option to set the value of the iam.amazonaws.com/role annotation to a given value to generate the Deployment fully.

I would like to propose a workflow like this:

velero install --provider aws \
  --aws-iam-role velero-role \
  --secret-file ./credentials-velero \
...etc

that results in a Deployment containing the following:

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
    namespace: velero
    name: velero
spec:
    replicas: 1
    template:
        metadata:
            labels:
                component: velero
            annotations:
                iam.amazonaws.com/role: velero-role

Anything else you would like to add:
If there is a better way to handle this situation, I would also be open to suggestions.

Environment:
AWS EKS with kube2iam

• Velero version (use velero version): 1.0.0
• Kubernetes version (use kubectl version): 1.10.13
• Kubernetes installer & version:
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Amazon Linux 2 (CentOS)

[skriss]

Thanks for logging the issue @tlkamp. We're trying to balance keeping velero install simple/not too many CLI flags, with giving users what they need. We'll definitely take a look at this and see if/how we can add it. Alternately, I'm curious if you've looked at the Helm chart as another means of installation?

[tlkamp]

@skriss thanks for getting back to me!

Please forgive me as I have no working experience with Helm (we are not using it), just an idea of what it is/does.

Ideally, we would like to deploy Velero onto our EKS clusters when they are built to ensure the cluster "base load" includes backup and migration/DR utilities. In order to enable Helm deployments of Velero as part of the base infrastructure load for us, we would need to also add Helm/Tiller into the base load for this single deployment. It would be a little cumbersome to do this if we could accomplish the same resulting deployment with a single extra CLI option. If I'm missing some key points to a better solution here, please please please let me know.

I also understand the desire to keep the installation options minimal. Perhaps a better alternative would be to support a generic --annotations key1=value1,key2=value2 option rather than supporting each individual cloud provider/annotation desire.

[nrb]

@tlkamp Would generating the YAML via velero install --dry-run -o yaml and keeping that in a git repo be usable with your automation?

[tlkamp]

@nrb it would be, yes, in the same way that adding Helm to my automation would be "usable"; it just creates a really terrible user experience when a new version of Velero is released. The workflow becomes:

1. Generate the yaml file
2. Manually add the annotation to the yaml
3. Deploy through automation
4. Wait for a new version of Velero to release
5. Return to step 1

This is the only issue we're encountering with our use of Velero -- the deployment not being generated "fully" and it' all due to a single annotation missing in the file.

I'm more than happy to submit a PR for this myself, assuming it would be accepted.

[nrb]

Yeah, that's a valid problem.

Truthfully, I'm very torn about extending velero install to include more features, but at the same time it's definitely a blocker for kube2iam/kiam uses.

For myself, I wouldn't reject a PR for this and would be happy to review it.

[skriss]

I think the generic approach of adding a --pod-template-annotations flag is reasonable. If we start adding provider-specific flags (e.g. --aws-iam-role), then the possible number of new flags explodes, but if we keep things generic, there's a limit to how many additional useful flags there are.

[nrb]

@tlkamp Are you using restic? Do you want/need different pod template annotations on the restic daemonset as opposed to the velero server deployment?

[tlkamp]

@nrb we are not using restic. We're only using Velero for our backups. The Velero documentation for AWS indicates that it needs some IAM grants in order to backup the cluster and upload said backup to S3.

We're using EKS with AWS IAM instance profiles so the credentials are auto-rotated by the system. We use kube2iam to make the connection between the applications and the IAM grants they require. kube2iam works by reading the annotations on the pod to associate the role, hence needing the annotation on the Velero pod, which is defined in the Deployment :)

[nrb]

@tlkamp Thanks! Just making sure, I can foresee that there may be a use case for adding different annotations to the restic daemonset. For now, I'm 👍 to having a single flag that adds the same annotations to both the daemonset and velero deployment.