Raft cluster_addr env variable is hardcoded to use https

[jmuleiro]

Is your feature request related to a problem? Please describe.
Feature. I've been trying to setup raft storage. After deleting one of the peers manually because of some debugging, I'm unable to re-add the node to the cluster. I've noticed that the environment variable VAULT_CLUSTER_ADDR is hardcoded to use https as seen in the server-statefulset.yaml template.

Describe the solution you'd like
Even though I've specified the cluster address and the leader API's address within the retry_join blocks, the pods still use a semi-hardcoded environment variable to try and reach the other nodes. I believe this may lead to not being able to re-join the cluster once a peer has been removed. If not using https, which is my case, we should be able to specify.

Describe alternatives you've considered
Tried hardcoding the environment variable a couple times but the way it is configured within the template does not allow for any changes when it comes to the https protocol.

Additional context
My high availability configuration looks like this, currently:

  ha:
    enabled: true
    replicas: 2
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true
        api_addr = "http://POD_IP:8200"
        cluster_addr = "http://POD_IP:8201"
        disable_mlock = true
        
        listener "tcp" {
          tls_disable = 1
          address = "0.0.0.0:8200"
          cluster_address = "0.0.0.0:8201"
        }

        storage "raft" {
          path = "/vault/data"
          node_id = "vault-0"
          retry_join {
            leader_api_addr = "http://vault-0.vault-internal:8200"
          }

          retry_join {
            leader_api_addr = "http://vault-1.vault-internal:8200"
          }

          autopilot {
            cleanup_dead_servers = "true"
            last_contact_threshold = "200ms"
            last_contact_failure_threshold = "10m"
            max_trailing_logs = 250000
            min_quorum = 2
            server_stabilization_time = "10s"
          }
        }
        
        service_registration "kubernetes" {}

        seal "gcpckms" {
          project = "REDACTED"
          region = "global"
          key_ring = "REDACTED"
          crypto_key = "REDACTED"
        }

The api_addr variable is picked up and used to configure the pods' VAULT_API_ADDR variable, contrary to cluster_addr which is ignored.

[sharkannon]

This issue can pop up as well when you're dealing with Vault being spread over multiple kube clusters. We currently have 2 clusters setup, one in us-central1 and one in us-east1 (GKE), and they can't talk to each other on internal service addresses, due to the fact that they are 2 different kube clusters.

I'll take a crack at adding a change to fix this, later today or maybe monday, shouldn't be to hard to fix.

[sharkannon]

#709 I've put that PR in, hopefully I can get some review on it and it can get merged soon.. it's causing a production issue right now for me, I don't really want to have to use the fork I created.

[jmuleiro]

The PR looks great to me, let's see if we can get someone else to review and approve it.