More clearly indicate where variable interpolation is/isn't valid

[margueritepd]

Nomad version

Nomad v0.8.4 (dbee1d7)

Operating system and Environment details

Ubuntu 16.04

Issue

Sometimes it's not obvious where Nomad environment variables and node variables can be interpolated.

For example, in the past I've had trouble with NOMAD_TASK_DIR, which is interpolated in the template stanza but not in artifact.

The documentation states:

Node attributes are interpretable in constraints, task environment variables and certain driver fields. Runtime environment variables are not interpretable in constraints because they are only defined once the scheduler has placed them on a particular node.

This does not give me enough information to confidently reason about where something like NOMAD_TASK_DIR would be available.

Having clearer docs about where a variable would be interpolated would be great. It would be even better if Nomad would warn or error when Nomad-provided variables were referenced in a place that would not be interpolated.

[tgross]

In #7877 (comment) I wrote:

Interpolating the volume block with information that's only available after placement on the host. This isn't going to be easily done (if at all), because we use the volume.source at the time of scheduling in order to determine where the volume can go. So there's a chicken-and-the-egg situation where the scheduler would not have enough information to place if we allowed the volume.source to include things like allocation index or client metadata. There may be improvements we can do here, but this is non-trivial. (I'm inclined to consider that as a larger Nomad issue than this feature request, because it impacts the entire scheduler.)

I think this also belongs in this documentation item. Especially with HCL2 support having landed in Nomad 1.0-beta, there a few different times that interpolation can happen:

• At job submission time, which allows for HCL2 functions that can include environment variables or arguments from the command line. This can be used to interpolate almost anything.
• At job validation time on the server.
• After job placement with TaskEnv, which includes client metadata or allocation index.

We should probably break down https://www.nomadproject.io/docs/runtime/interpolation with more explicit information about what data is available to interpolate at any given point in time.

[shantanugadgil]

Besides what I have already used and "known to be working previously", if I try out something new wrt interpolation, its always a hit-or-miss.
Knowing which variable (or set of variables) would work (or not work) beforehand, would be awesome. 👍