RKE should allow setting extra_env/etc. on the ingress provider

[Oats87]

Currently, it is not possible to set extra_env/extra_binds on the ingress provider. We should allow this functionality, in order to allow users to perform customizations to the nginx ingress controller. This can include things like adding proxies, adding custom file-based certs, etc.

[superseb]

rancher/rancher#16868

[cloudnautique]

This should be under similar keys to the upstream helm chart: https://github.com/helm/charts/tree/master/stable/nginx-ingress

Specifically the functionality introduced by:
controller.extraVolumeMounts
controller.extraEnvs

[mrajashree]

The ingress config now accepts three more fields: ExtraEnvs, ExtraVolumes and ExtraVolumeMounts.
They refer the k8s native types EnvVar, Volume and VolumeMount respectively.

For example:

  ingress:
    provider: nginx
    extra_envs:
      - name: MY_NODE_NAME
        valueFrom:
          fieldRef:
            fieldPath: spec.nodeName
      - name: testenvvar
        value: testvalue
    extra_volume_mounts:
      - mountPath: /testmountpath
        name: testvol
    extra_volumes:
      - emptyDir: {}
        name: testvol

This should result in creating 2 env vars in the ingress nginx pod:

1. $MY_NODE_NAME with the node name on which it's running as the value
2. $testenvvar with testvalue as value.

And a dir named testmountpath will be created within the pod at /testmountpath

[soumyalj]

Tested with rancher version v2.3-head from Nov 11
Created a DO cluster and specified the extra_envs, extra_volume_mounts, extra_volumes in cluster.yml

ingress:
    provider: nginx
    extra_envs:
      - name: soumyanode1
        valueFrom:
          fieldRef:
            fieldPath: spec.nodeName
      - name: testenvvar
        value: testvalue
    extra_volume_mounts:
      - mountPath: /testmountpath
        name: testvol
    extra_volumes:
      - emptyDir: {}
        name: testvol

The cluster got created successfully. In the pod , nginx-ingress-controller-g8jqp, the volume mount /testmountpath was created as expected. Read/Write to the volume was successful.
The env variables were also verified as below in the pod by typing env command

KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_PORT=443
HOSTNAME=soumyanode1
DEFAULT_HTTP_BACKEND_PORT_80_TCP_PROTO=tcp
POD_NAME=nginx-ingress-controller-g8jqp
POD_NAMESPACE=ingress-nginx
soumyanode1=soumyanode1
PWD=/etc/nginx
DEFAULT_HTTP_BACKEND_SERVICE_HOST=abcd
HOME=/var/www
KUBERNETES_PORT_443_TCP=tcp://abcd:443
DEFAULT_HTTP_BACKEND_PORT_80_TCP_ADDR=abcd
testenvvar=testvalue
TERM=xterm-256color
SHLVL=1
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=abcd
KUBERNETES_SERVICE_HOST=abcd

Tested with rke version v1.0.0-rc3: rke up with the config.yml with ingress config as above had the warning below

INFO[0116] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0116] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0116] [addons] Executing deploy job rke-ingress-controller 
WARN[0147] Failed to deploy addon execute job [rke-ingress-controller]: Failed to get job complete status for job rke-ingress-controller-deploy-job in namespace kube-system 
INFO[0147] [addons] Setting up user addons              
INFO[0147] [addons] no user addons defined              
INFO[0147] Finished building Kubernetes cluster successfully 

nginx-ingress-controller pod did not come up successfully.

[sangeethah]

@soumyalj v1.0.0-rc3 has the fix for testing rke cli use case.

[soumyalj]

Tested with rke version v1.0.0-rc3
Created a rke cluster using the cluster.yml file below:

auth:
  strategy: x509
nodes:
  - address: abcd
    user: root
    role: [etcd,controlplane,worker]
  - address: efgh
    user: root
    role: [etcd,controlplane,worker]
  - address: ijkl
    user: root
    role: [etcd,controlplane,worker]
ingress:
    provider: nginx
    extra_envs:
      - name: MYNODENAME
        valueFrom:
          fieldRef:
            fieldPath: spec.nodeName
      - name: testenvvar
        value: testvalue
    extra_volume_mounts:
      - mountPath: /data
        name: testvol
    extra_volumes:
      - emptyDir: {}
        name: testvol

Cluster came up successfully. rancher/nginx-ingress-controller pod was created successfully on each node. Exec'ed into each of the containers and verified that the volume mount /data exists.
Write and read to the volume was successful.

env variables were also verifed

HOME=/var/www
OLDPWD=/etc/nginx
DEFAULT_HTTP_BACKEND_SERVICE_HOST=abcd
TERM=xterm
POD_NAME=nginx-ingress-controller-mrnvg
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
DEFAULT_HTTP_BACKEND_SERVICE_PORT=80
DEFAULT_HTTP_BACKEND_PORT=tcp://abcd:80
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/openresty/luajit/bin:/usr/local/openresty/nginx/sbin:/usr/local/openresty/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
DEFAULT_HTTP_BACKEND_PORT_80_TCP_ADDR=efgh
DEFAULT_HTTP_BACKEND_PORT_80_TCP_PORT=80
DEFAULT_HTTP_BACKEND_PORT_80_TCP_PROTO=tcp
MYNODENAME=157.245.231.221
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://abcd:443
POD_NAMESPACE=ingress-nginx
KUBERNETES_SERVICE_HOST=10.43.0.1
LUA_CPATH=/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;/usr/local/openresty/site/lualib/?.so;/usr/local/openresty/lualib/?.so;./?.so;/usr/local/lib/lua/5.1/?.so;/usr/local/openresty/luajit/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so;/usr/local/openresty/luajit/lib/lua/5.1/?.so;;
PWD=/data
testenvvar=testvalue
DEFAULT_HTTP_BACKEND_PORT_80_TCP=tcp://ancd:80