Proposal: Use dhall for yaml generation instead of (mostly) python

[s-zeng]

I know that this has been talked about for a while, but I would like to formally make my proposal to use Dhall as our kubernetes yaml-templating solution. I have a working implementation to demonstrate as well, which can be found on the branch s-zeng/dhall_config. See below for more

Problem

Python, as a general purpose language, can often be not the most ergonomic language to write configurations in.

Our current process with mkchain.py involves quite a number of steps - we have a variety of existing yaml templates, which we consume as python data structures to edit a handful of them in place, and then spit everything back out as yaml again.

In particular, we have a main loop which involves looking through every yaml template, wherein we check each template for the appropriate name to apply a particular patch on. Essentially:

if safeget(k, "metadata", "name") == <a>:
    # apply patch <a>
    ...
if safeget(k, "metadata", "name") == <b>:
    # apply patch <b>
    ...

This style, in my opinion, obfuscates the relationship between the original templates and the logic that goes into adjusting them, and can make it hard to follow how the inputs become the final outputted yaml file.

Why Dhall

Unlike Python, Dhall is purpose-built as a configuration language, with a number of features designed specifically to this end. A few for summary:

• Essentially JSON or YAML, but with functions
• Strong types: Has a type system in the vein of ML/Haskell. Even has dependent types (!) if one is so inclined to go that far. Allows for much stronger guarantees about the structure of data
• Pure and non-turing complete: unlike general-purpose languages, guarantees lack of IO, and guarantees termination. Dhall is a strongly-normalizing language, so every expression is guaranteed to normalize into a most-reduced form
• Can output to always-correct JSON or YAML

Below are snippets from my prototype dhall implementation, along with comparisons to the original python if applicable.

• main mkchain function (only have mkchain create implemented right now but should be trivial to implement invite:

let tezos = ./tezos/tezos.dhall

let k8s = tezos.common.k8s.Resource

let create =
      λ(opts : tezos.chainConfig.Type) →
        [ k8s.Namespace tezos.namespace
        , k8s.Secret (tezos.chainConfig.makeK8sSecret opts)
        , k8s.ConfigMap tezos.utilsConfigMap
        , k8s.ConfigMap (tezos.chainConfig.makeChainParams opts)
        , k8s.StatefulSet (tezos.makeNode opts)
        , k8s.Job (tezos.bootstrap.makeActivateJob opts)
        , k8s.Service tezos.bootstrap.port_services.rpc
        , k8s.Service tezos.bootstrap.port_services.p2p
        , k8s.Deployment (tezos.bootstrap.makeDeployment opts)
        , k8s.PersistentVolumeClaim tezos.bootstrap.pvc
        , k8s.PersistentVolumeClaim tezos.zerotier.pvc
        , k8s.ConfigMap (tezos.zerotier.makeZTConfig opts)
        , k8s.DaemonSet tezos.zerotier.bridge
        ]

in  { tezos, create }

• Sample config + running (current weakness: stil needs an external script to generate things like keys and such. see below for more info):

let mkchain = ./mkchain.dhall

let opts =
      mkchain.tezos.chainConfig.ChainOpts::{
      , chain_name = "simons_chain"
      , baker = True
      , keys = toMap
          { baker_secret_key = "..."
          , bootstrap_account_1_secret_key = "..."
          , bootstrap_account_2_secret_key = "..."
          , genesis_secret_key = "..."
          }
      , bootstrap_timestamp = "2020-10-27T20:44:49.522093+00:00"
      , genesis_chain_id = "BL4D6Gg3XArdsawVRcvMiyny2QEhWtUzi4CXgzemUKkEnuedDQG"
      , additional_nodes = 4
      , zerotier_network = "6ab565387a53269c"
      , zerotier_token = "agnshaxe0mqF1WAiWDVHw1lIRbds9xyq"
      , zerotier_hostname = "46305149-64cb-4f74-aea0-7f249324b574"
      }

in  mkchain.create opts

The above can be used to output yaml with the following command:

dhall-to-yaml --documents <<< sample.dhall

• Templating activate job in dhall:

let makeActivateJob =
      λ(opts : chainConfig.Type) →
        let varVolume =
              merge
                { minikube = common.volumes.bootstrap_pvc_var
                , eks = common.volumes.var
                }
                opts.cluster

        let spec =
              k8s.JobSpec::{
              , template = k8s.PodTemplateSpec::{
                , metadata = k8s.ObjectMeta::{ name = Some "activate-job" }
                , spec = Some k8s.PodSpec::{
                  , initContainers = Some
                    [ jobs.make_import_key_job opts
                    , jobs.config_generator
                    , jobs.wait_for_node
                    , jobs.make_activate_container opts
                    , jobs.make_bake_once opts
                    ]
                  , containers =
                    [ k8s.Container::{
                      , name = "job-done"
                      , image = Some "busybox"
                      , command = Some
                        [ "sh", "-c", "echo \"private chain activated\"" ]
                      }
                    ]
                  , restartPolicy = Some "Never"
                  , volumes = Some
                    [ common.volumes.config, varVolume, common.volumes.utils ]
                  }
                }
              }

        in  k8s.Job::{
            , metadata = common.tqMeta "activate-job"
            , spec = Some spec
            }

Compared to python yaml template + if statement:

apiVersion: batch/v1
kind: Job
metadata:
  name: activate-job
  namespace: "tqtezos"
spec:
  template:
    metadata:
      name: activate-job
    spec:
      initContainers:
      - name: import-keys
        command: ['sh', '/opt/tqtezos/import_keys.sh']
        envFrom:
        - secretRef:
            name: tezos-secret
        volumeMounts:
        - name: tqtezos-utils
          mountPath: /opt/tqtezos
        - name: var-volume
          mountPath: /var/tezos
      - imagePullPolicy: Always
        name: tezos-config-generator
        image: python:alpine
        command: ["python", "/opt/tqtezos/generateTezosConfig.py"]
        envFrom:
        - configMapRef:
            name: tezos-config
        volumeMounts:
        - name: config-volume
          mountPath: /etc/tezos
        - name: tqtezos-utils
          mountPath: /opt/tqtezos
        - name: var-volume
          mountPath: /var/tezos
      - name: wait-for-node
        image: busybox
        command: ['sh', '-c', 'until nslookup tezos-bootstrap-node-rpc; do echo waiting for tezos-bootstrap-node-rpc; sleep 2; done;']
      - name: activate
        command: ["/usr/local/bin/tezos-client"]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/tezos
        - name: var-volume
          mountPath: /var/tezos
      - name: bake-once
        command: ["/usr/local/bin/tezos-client"]
        args: ["-A", "tezos-bootstrap-node-rpc", "-P", "8732", "-d", "/var/tezos/client", "-l", "bake", "for", "baker", "--minimal-timestamp"]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/tezos
        - name: var-volume
          mountPath: /var/tezos
      containers:
      - name: job-done
        image: busybox
        command: ['sh', '-c', 'echo "private chain activated"']
      restartPolicy: Never
      volumes:
      - name: config-volume
        emptyDir: {}
      - name: var-volume
        emptyDir: {}
      - name: tqtezos-utils
        configMap:
          name: tqtezos-utils

                if safeget(k, "metadata", "name") == "activate-job":
                    k["spec"]["template"]["spec"]["initContainers"][0][
                        "image"
                    ] = c["docker_image"]
                    k["spec"]["template"]["spec"]["initContainers"][3][
                        "image"
                    ] = c["docker_image"]
                    k["spec"]["template"]["spec"]["initContainers"][3]["args"] = [
                        "-A",
                        "tezos-bootstrap-node-rpc",
                        "-P",
                        "8732",
                        "-d",
                        "/var/tezos/client",
                        "-l",
                        "--block",
                        "genesis",
                        "activate",
                        "protocol",
                        c["protocol_hash"],
                        "with",
                        "fitness",
                        "-1",
                        "and",
                        "key",
                        "genesis",
                        "and",
                        "parameters",
                        "/etc/tezos/parameters.json",
                    ]
                    k["spec"]["template"]["spec"]["initContainers"][4][
                        "image"
                    ] = c["docker_image"]

                    if args.cluster == "minikube":
                        k["spec"]["template"]["spec"]["volumes"][1] = {
                           "name": "var-volume",
                           "persistentVolumeClaim": {
                             "claimName": "tezos-bootstrap-node-pv-claim" } }

[s-zeng]

With the introduction of optional zerotier containers and init containers present across the bootstrap nodes and the statefulset nodes from #42 , I feel that we've hit the limit of what Dhall is capable of. Until dhall-lang/dhall-haskell#2040 is merged, there simply is no ergonomic way to create generic helm charts with Dhall. While it is a nice and idealistic configuration paradigm, it is in a too immature state for our purposes right now. As such, closing until further notice.