Implement Terraform Runner

[chanwit]

Terraform Runner is a sub system of TF-controller.
It is responsible for running Terraform commands at the different stages.
Terraform Runner is a part of supporting multi-tenancy #43

Requirements

• It must be an gRPC server wrapping around TFExec Implement local RPC server for TF Runner #68
• Its local version must be able to use inside the TF-controller, and all current tests pass Implement local RPC server for TF Runner #68
• the gRPC server must support executing a command for each step in the life-cycle Implement local RPC server for TF Runner #68
  • receive tar.gz
  • init
  • plan for drift detection
  • plan
  • apply
  • show
  • output
• Separate runner as a Pod Implement runner main program #70
• Implement Pod invocation logics #71
• the gRPC server must support mTLS #72
• TF-controller must be able to rotate certificates for every Terraform Runner Pod #73
• A Runner Pod must be placed inside the namespace of the Terraform object only #105
• A Runner Pod of a Terraform Object must use the Service Account specified by the Terraform object #106

High Level Diagram

[phoban01]

Couple of additional thoughts on this:

• Will we aim to support persistence for the Terraform Runner Pod? This would be required for caching providers etc... Introduces a little complexity as may require a TerraformStatefulRunner in addition to an (ephemeral/one-shot) TerraformRunner.
• Is it our intention to handle TLS generation/rotation within the controller code a la Gatekeeper or can we offload this to cert-manager? It may reduce the initial effort if cert-manager can handle it.
• How is reconcile state distributed between controller & runner, i.e. what does .status.conditions look like for the runner? This has knock on implications for how we handle communication and trigger the reconcile loop in tf-controller. I think communicating via status conditions should be the preferred route but it might also be possible to do something similar to flux/notification controller using a gRPC call.

[chanwit]

Will we aim to support persistence for the Terraform Runner Pod? This would be required for caching providers etc... Introduces a little complexity as may require a TerraformStatefulRunner in addition to an (ephemeral/one-shot) TerraformRunner.

This is a good question. I'm thinking of it as a stateless system. Runner Pod is a gRPC server which allows to crash.

Is it our intention to handle TLS generation/rotation

We'll do our own TLS generation, yes. A goal is to make the controller self-contained

How is reconcile state distributed between controller & runner

Reconciliation process happens only inside the controller using the current set of logics.
Runner Pod will be issued a command, and send output back to the Controller, via gRPC, just like we run that command locally.

[chanwit]

I'll get the first version of POC out. We can then discuss and move it forward.

[chanwit]

@phoban01 would you like to further split this epic into other issues?

[phoban01]

@chanwit Looks alright to me.

[chanwit]

Work done enough to close this issue. We'll fix other related issues along the way.