Terraform rancher2 agent customization

[a-blender]

Issue:

#1097

Problem

Cluster/Fleet agent deployment customization is not supported by the TFP.

Solution

Add cluster_agent_deployment_customization and fleet_agent_deployment_customization (and nested) fields tolerations, affinity, and resource requirements to the rancher2 provider so users will be able to define those values for RKE, RKE2/K3s, and EKS clusters.

Error handling

For agent deployment customization, we have a unique scenario with error handling. Most tf resources do not add error handling in the expander, but for rke clusters I have to marshal v1 Tolerations and Affinity strings into objects that exist on the rancher backend. They are being passed as strings because 1) it's too complex to struct out all the nested fields, 2) this is how terraform harvester encodes node affinity. If the Toleration or Affinity string passed via the tf config is invalid, expanding the field will fail. I argue that this should cause a cluster create/update to also fail.

My solution to this is to add error handling to the expander. If expanding an agent deployment customization returns nil with no error, customization is not defined and a create/update will pass. If it returns nil with an error, a create/update will fail and the user needs to fix their config.

Testing

6/2 update: Not fully tested yet, but requesting preliminary review of backend logic for the rke case.

6/13 update: Finished testing rke with all new fields and fixing bugs. Working on adding EKS, testing v2 prov and EKS and updating tests.

6/16 update: Added Toleration v2 and ResourceRequirement resources, updated backend logic. Added/updated tests which turned out to be tricky. Need to re-test

Engineering Testing

Manual Testing

Smoke testing

• rke single all-role node + agent customization
  • remove agent customization, verify removed
  • add it back, verify customization is added
  • update customization, verified it got updated on the v3 cluster spec and the agent deployment yaml
• rke2 (v2 prov) single all-role node + agent customization
  • same as above
• eks 2 node + agent customization
  • same as above

Note: you may not see cluster go into Updating state in the UI for added/updated agent customization or minor updates because the redeploy is fast.

Automated Testing

QA Testing Considerations

Regressions Considerations

[jakefhyde]

Few nits