Terraform connecting to controller in k8s cluster

[kian99]

Hi,

The Terraform provider currently accepts a controller address, username, password and CA cert.
In the case of a controller within a kubernetes cluster, the Juju config file controllers.yaml also contains a "proxy-config" section with more details particularly for getting access into the cluster and talking to the controller. Without this extra config I imagine it is impossible to talk to a controller within a k8s cluster.

I'd just like to confirm the above is correct and if so make a feature request for the provider to be able to make use of the extra "proxy-config" information.

[juanmanuel-tirado]

AFAIK the juju connection only requires the API endpoint of the controller (here). The proxy-config is not required.

At this moment, this provider does not bootstrap controllers and this param is not required.

I will close this issue. Please reopen if you have more comments.

[alesstimec]

Actually, won't juju use the proxy? it does create a proxy tunnel...

[alesstimec]

16:05:37 INFO juju.cmd supercommand.go:56 running juju [2.9.38 6d211be0d72d6f4d625c61c7c4ddb4e9325226c8 gc go1.18.9]
16:05:37 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/21790/bin/juju", "models", "--debug"}
16:05:37 DEBUG juju.jujuclient proxy.go:65 unmarshalled proxy config for "kubernetes-port-forward"
16:05:37 DEBUG juju.jujuclient proxy.go:65 unmarshalled proxy config for "kubernetes-port-forward"
16:05:37 DEBUG juju.jujuclient proxy.go:65 unmarshalled proxy config for "kubernetes-port-forward"
16:05:37 DEBUG juju.jujuclient proxy.go:65 unmarshalled proxy config for "kubernetes-port-forward"
16:05:37 INFO juju.juju api.go:86 connecting to API addresses: [10.152.183.197:17070]
16:05:37 DEBUG juju.api apiclient.go:625 starting proxier for connection
16:05:37 DEBUG juju.api apiclient.go:629 tunnel proxy in use at localhost on port 35967
16:05:37 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://localhost:35967/api"
16:05:37 INFO juju.api apiclient.go:1054 cannot resolve "localhost": lookup localhost: operation was canceled
16:05:37 INFO juju.api apiclient.go:687 connection established to "wss://localhost:35967/api"
Controller: micro

Model Cloud/Region Type Status Units Access Last connection
controller microk8s/localhost kubernetes available - admin just now
test microk8s/localhost kubernetes available 1 admin 11 minutes ago
test-charm-ts15* microk8s/localhost kubernetes available 4 admin 12 minutes ago

16:05:37 DEBUG juju.api monitor.go:35 RPC connection died
16:05:37 INFO cmd supercommand.go:544 command finished

[kian99]

Hi,

Building on what @alesstimec said, I believe there is another type of connector that is used when connecting in more complex scenarios in Juju, for example when connecting to a controller residing within a k8s cluster.

The connector is here in Juju.

I ran into this issue when trying to use Terraform to connect to a controller that is within Kubernetes, the error was as below. In this scenario the controller is running and can be access with regular juju commands but not via Terraform.

│ Error: dial tcp 10.85.0.121:17070: i/o timeout
│ 
│   with provider["terraform.local/local/juju"],
│   on providers.tf line 1, in provider "juju":
│    1: provider "juju" {
│ 
│ Connection error, please check the controller_addresses property set on the provider

I hope that makes sense.

[juanmanuel-tirado]

I reopen this to start some additional research on how to proceed.

[kian99]

@juanmanuel-tirado Just a note that we ended up needing this again and came up with a workaround using kubectl port-forward which is what I believe the Juju CLI does when talking to a controller in k8s.

For anyone else encountering this issue, you can use the following command + tweak to enable the Terraform provider to talk to a controller in k8s.

1. kubectl port-forward <controller-pod-name> 17070:17070 # Note you must be in the namespace of the controller, this will open a proxy to the controller that can be accessed via localhost.
2. Change your Terraform spec (or just environment variable) so that the controller address is as below (might also work if you change it to localhost).

provider "juju" {
  controller_addresses = "0.0.0.0:17070"
  ...
}

Thanks to @kelkawi-a for testing this.

[juanmanuel-tirado]

Thanks @kian99 for the workaround!!

[cderici]

I just published a post about this on discourse. I'll close this particular issue, but feel free to comment/reply either here or on the discourse post to further the discussion 👍