use SSL_CERT_DIR environment variable for snapshot agent

[kschoche]

This PR changes the way that we used to echo a custom ca-cert to the system which is used for system level operations (like interacting with s3 storage classes). Previously we wrote directly to /etc/ssl however in OpenShift there is no root access to do this write. Instead we will write the file to an emptyDir volume that we have access to, and add its path to the container as an environment variable that overrides SSL_CERT_DIR

Changes proposed in this PR:

• Update the snapshot agent deployment to specify the SSL_CERT_DIR environment variable when client.snapshotAgent.caCert is specified.
• add a new emptydir volume to the pod and echo the custom ca pem file to the emptydir
• slight reorganization of the volumeMounts is needed so we can have this enabled with global.tls.enabled=false because client.snapshotAgent.caCert is a caCert for the system, not for Consul.

How I've tested this PR:

• Manually confirmed that we do not see the permission denied errors in openshift while still echo'ing the pem file to the new volume
• confirmed the environment variable is set properly
  Following the guidance in this comment : Support custom CA cert for snapshot agent consul-helm#481 (comment)

I ran the following:

consul tls ca create
onsul tls cert create -server -additional-dnsname=0.tcp.ngrok.io
mv dc1-server-consul-0-key.pem server.key
mv dc1-server-consul-0.pem server.crt

and then copied the CA cert to the deployments command.

I ran the following server code locally on my laptop using go run server.go

demo $ cat server.go
package main

import (
	"log"
	"net/http"
)

func HelloServer(w http.ResponseWriter, req *http.Request) {
	w.Header().Set("Content-Type", "text/plain")
	w.Write([]byte("This is an example server.\n"))
}

func main() {
	http.HandleFunc("/hello", HelloServer)
	err := http.ListenAndServeTLS(":80", "server.crt", "server.key", nil)
	if err != nil {
		log.Fatal("ListenAndServe: ", err)
	}
}

I ran ngrok locally on my laptop:

demo $  ngrok tcp 80

This output like this:

ngrok                                                      (Ctrl+C to quit)

Session Status                online
<snip>
Forwarding                    tcp://0.tcp.ngrok.io:10598 -> localhost:80

I then built a container using the following dockerfile:

demo $ cat container/Dockerfile
# syntax=docker/dockerfile:1

FROM golang:1.17-alpine as builder
WORKDIR /app
COPY . .
RUN go mod download
RUN go build -o ./app

FROM alpine:3.15.0
RUN apk add --no-cache curl
RUN apk add --no-cache openssl
COPY --from=builder /app/app /bin/app
ENTRYPOINT ["/bin/app"]

and app:

demo $ cat container/app.go
package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"time"
)

func main() {
	for {
		time.Sleep(1 * time.Second)
		makeCall()
	}
}

func makeCall() {
	resp, err := http.Get("https://0.tcp.ngrok.io:10598/hello")   // Note the url is the same as the ngrok url
	if err != nil {
		fmt.Printf("err: %s\n", err)
		return
	}

	defer resp.Body.Close()
	body, err := ioutil.ReadAll(resp.Body)
	fmt.Printf("resp: %s\n", string(body))
}

And then deployed the app to Kind k8s cluster:

demo $ cat demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  labels:
    app: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
        - name: test
          image: "kyleschochenmaier/ngrok-test"
          env:
          - name: SSL_CERT_DIR
            value: "/etc/ssl/certs:/tmp/"
          command:
            - "/bin/sh"
            - "-ec"
            - |
              cat <<EOF > /tmp/custom-ca.pem
              -----BEGIN CERTIFICATE-----
<SNIP --- use the CA cert >
              -----END CERTIFICATE-----
              EOF
              /bin/app

And then checked the logs from the client pod:

demo $ k logs -f test-54d6ccd748-v8wdh

resp: This is an example server.

resp: This is an example server.

resp: This is an example server.

How I expect reviewers to test this PR:
bats tests and acceptance pass.

Checklist:

• Tests added
• CHANGELOG entry added

  HashiCorp engineers only, community PRs should not add a changelog entry.
  Entries should use present tense (e.g. Add support for...)

[kschoche]

This is just moved a bit down.

[jmurret]

Looks good. 🎉 changelog?

[curtbushko]

Looks good!