HCL2 split function doesn't evaluate its parameters

[axsuul]

Nomad version

1.2.3

Operating system and Environment details

Ubuntu 20.04 LTS

Issue

With meta.variants_comma_separated being set to "base,uhd", using HCL2, the output doesn't appear to be split properly. For example,

join(":", [for v in split(",", meta.variants_comma_separated) : "/mnt/media-variant-${v}"]) returns /mnt/media-variant-base,uhd

However, if the value is hardcoded

join(":", [for v in split(",", "base,uhd") : "/mnt/media-variant-${v}"]) returns /mnt/media-variant-base:/mnt/media-variant-uhd

Reproduction steps

/etc/nomad.d/nomad.hcl

client {
  enabled = true

  meta {
    variants_comma_separated = "base,uhd"
  }
}

with job file

job "issue" {
  datacenters = ["dc1"]

  group "issue" {
    task "issue" {
      driver = "raw_exec"

      env {
        VARIANTS_COMMA_SEPARTED = "${meta.variants_comma_separated}"
        VARIANT_PATHS_1 = join(":", formatlist("/mnt/media-variant-%s", split(",", "base,uhd")))
        VARIANT_PATHS_2 = join(":", [for v in split(",", "base,uhd") : "/mnt/media-variant-${v}"])
        META_VARIANT_PATHS_1 = join(":", formatlist("/mnt/media-variant-%s", split(",", meta.variants_comma_separated)))
        META_VARIANT_PATHS_2 = join(":", [for v in split(",", meta.variants_comma_separated) : "/mnt/media-variant-${v}"])
      }

      config {
        command = "/bin/bash"
        args = ["${NOMAD_TASK_DIR}/test.sh"]
      }

      template {
        data = <<-EOF
          #!/bin/bash 

          echo $VARIANTS_COMMA_SEPARTED
          echo $VARIANT_PATHS_1
          echo $VARIANT_PATHS_2
          echo $META_VARIANT_PATHS_1
          echo $META_VARIANT_PATHS_2

          # Keep script running
          tail -f /dev/null
        EOF
        destination = "local/test.sh"
      }
    }
  }
}

Expected Result

STDOUT should return

base,uhd
/mnt/media-variant-base:/mnt/media-variant-uhd
/mnt/media-variant-base:/mnt/media-variant-uhd
/mnt/media-variant-base:/mnt/media-variant-uhd
/mnt/media-variant-base:/mnt/media-variant-uhd

Actual Result

STDOUT instead returns

base,uhd
/mnt/media-variant-base:/mnt/media-variant-uhd
/mnt/media-variant-base:/mnt/media-variant-uhd
/mnt/media-variant-base,uhd
/mnt/media-variant-base,uhd

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[Amier3]

Hey @axsuul

Thanks for finding this! We'll have to look into this and see if it's an issue with HCL or the nomad parser 🤔 . We'll keep this issue updated with what we find

[axsuul]

Hey just wanted to follow up on this 👍

[axsuul]

Bumping this again

[axsuul]

I believe this is still an issue, bumping

[tgross]

Hi @axsuul there's no need to bump issues. We'll update when there's an update to be had. Thanks.

[heatzync]

I'm also experiencing an issue where the split function isn't working as expected, so I'm adding the details of this scenario here...

Nomad version

1.6.0-beta.1

Operating system and Environment details

Manjaro Ultima Thule 23.0.0
Kernel: 6.1.31-2

Issue

Using a template to read key -> value pairs from a nomad variable into environment variables and then splitting one of the string ENV VARs on a comma doesn't work.

Reproduction steps

Define a nomad variable that contains multiple key value pairs, with one of them being:

{
"extra_hosts": "extra-host:172.17.0.1,demo.extra-host:172.17.0.1"
}

job "job-name-same-as-var-name" {
  group "group1" {
    task "task1" {
      template {
        change_mode          = "restart"
        error_on_missing_key = true
        destination          = "${NOMAD_SECRETS_DIR}/.env"
        env                  = true
        data                 = <<EOT
{{- with nomadVar (env "NOMAD_JOB_NAME") -}}
EXTRA_HOSTS = {{ .extra_hosts }}
{{- end -}}
EOT
      }

      driver = "docker"

      config {
        image       = "path.to/image:tag"
        extra_hosts = flatten(["hard-coded-host:123.45.67.8", split(",", EXTRA_HOSTS)])
      }
    }
  }
}

Expected result
EXTRA_HOSTS to be split and returned as an array:

[
  "extra-host:172.17.0.1",
  "demo.extra-host:172.17.0.1"
]

Actual result

Driver Failure: failed to create container: API error (400): invalid IP address in add-host: "172.17.0.1,demo.extra-host:172.17.0.1"

[axsuul]

@tgross I'm still seeing this issue in Nomad 1.6.1

[tgross]

Hi @axsuul there's no need to bump issues. We'll update when there's an update to be had. Thanks.

[tgross]

Doing a bit of issue cleanup and dug into the syntax used here. I've tried this jobspec with a couple of variations:

jobspec

locals {
  variants = "${meta.variants}"
}

job "example" {

  group "issue" {
    task "issue" {
      driver = "raw_exec"

      env {
        VARIANTS_COMMA_SEPARTED = "${meta.variants}"
        VARIANT_PATHS_1 =      join(":", formatlist("/mnt/media-variant-%s", split(",", "base,uhd")))
        VARIANT_PATHS_2 =      join(":", [for v in split(",", "base,uhd") : "/mnt/media-variant-${v}"])
        META_VARIANT_PATHS_1 = join(":", formatlist("/mnt/media-variant-%s", split(",", "${meta.variants}")))
        META_VARIANT_PATHS_2 = join(":", [for v in split(",", "${meta.variants}") : "/mnt/media-variant-${v}"])
        META_VARIANT_PATHS_3 = join(":", formatlist("/mnt/media-variant-%s", split(",", "${local.variants}")))
        META_VARIANT_PATHS_4 = join(":", [for v in split(",", "${local.variants}") : "/mnt/media-variant-${v}"])
        OTHER_FUNCTIONS = join(":", ["${meta.variants}", local.variants])

      }

      config {
        command = "/bin/bash"
        args = ["${NOMAD_TASK_DIR}/test.sh"]
      }

      template {
        data = <<-EOF
          #!/bin/bash

          echo $VARIANTS_COMMA_SEPARTED
          echo $VARIANT_PATHS_1
          echo $VARIANT_PATHS_2
          echo $META_VARIANT_PATHS_1
          echo $META_VARIANT_PATHS_2
          echo $META_VARIANT_PATHS_3
          echo $META_VARIANT_PATHS_4
          echo $OTHER_FUNCTIONS

          # Keep script running
          tail -f /dev/null
        EOF
        destination = "local/test.sh"
      }
    }
  }
}

That returns:

$ nomad alloc logs eda8f9fa
base,uhd
/mnt/media-variant-base:/mnt/media-variant-uhd
/mnt/media-variant-base:/mnt/media-variant-uhd
/mnt/media-variant-base,uhd
/mnt/media-variant-base,uhd
/mnt/media-variant-base,uhd
/mnt/media-variant-base,uhd
base,uhd:base,uhd

It looks like the parameters of the split function (specifically) simply aren't being evaluated. This looks like it might be a HCL2 bug rather than a Nomad bug.

I'm going to re-title this for clarity and mark it for roadmapping. The issue will be updated when someone is working on it.

---

For @heatzync's report:

Using a template to read key -> value pairs from a nomad variable into environment variables and then splitting one of the string ENV VARs on a comma doesn't work.

Correct. See the HCL Parsing Context docs. The value you're trying to interpolate into extra_hosts only exists on the client, whereas the HCL2 evaluation is being done on your command line client. That's working as intended.

[axsuul]

Thanks @tgross for the followup!