Pipeline parsing attempts to interpolate invalid environment variables

[DazWorrall]

Trying to upload a pipeline with buildkite-agent pipeline upload containing this snippet:

- trigger: some-pipeline
  build:
    message: Don't create remote checkouts for $0 checkouts
    commit: 1eda429965fd220878dd60d14cf1313988dcb319
    branch: master

Fails with:

DEBUG  Debug mode enabled
INFO   Reading pipeline config from "/tmp/buildkite-pipeline-20180403-1053-88yil1"
DEBUG  Parsing pipeline configuration as YAML
FATAL  Pipeline parsing of "buildkite-pipeline-20180403-1053-88yil1" failed (Expected identifier to start with a letter, got 0)

It's trying to interpolate the $0 I guess. As a digit it not a valid variable name I feel like this shouldn't break the parser - it really shouldn't attempt to parse it - wdyt @lox?

[lox]

It's a tricky one. I'd say it probably makes sense to ignore numeric vars, but I'd suggest escaping dollar signs in your commit messages or content for generated pipelines, otherwise it will crop up some other way.

[keithpitt]

@DazWorrall that pipeline snippet, was that a literal "Don't create remote checkouts for $0 checkouts" in your pipeline.yml? Or was it interpolated in there from another pipeline upload?

[DazWorrall]

Sorry for the delay @keithpitt, I was travelling. It was a literal $0, we're doing something like:

{ trigger: 'my-pipeline', build: { message: ENV['BUILDKITE_MESSAGE'] }}.to_yaml

2 small and 1 larger thing here I think:

• $0 is not a valid environment variable, and should not attempt to be parsed
• We should be escaping/encoding in our pipeline generation above
• A limitation of our current approach (first step compiles a pipeline for the rest of the build) is that we can't (that I can see) disable the interpolation at pipeline upload time, so we can do that at runtime on the job. This is out of scope here though, just a related niggle :)

[keithpitt]

Ohh, right right. I see now. A hack for now is to change:

ENV['BUILDKITE_MESSAGE']

...to...

$BUILDKITE_MESSAGE

So our interpolation gear picks it up. But I totally agree that we should be ignoring $0 and not throwing an error.

[lox]

Could we add a “—no-interpolation” to the upload command?

…

On 11 Apr 2018, at 9:56 am, Keith Pitt ***@***.***> wrote: Ohh, right right. I see now. A hack for now is to change: ENV['BUILDKITE_MESSAGE'] ...to... $BUILDKITE_MESSAGE So our interpolation gear picks it up. But I totally agree that we should be ignoring $0 and not throwing an error. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[DazWorrall]

Could we add a “—no-interpolation” to the upload command?

We would gobble that right up for sure.

[keithpitt]

I think that's a good idea as well!