Support templated expressions in Azure pipelines files

[davewhitters]

Some lines in a yml/json file may not adhere to a schema. (azure devops preprocesses their yml pipelines prior to running their schema against the .yml files)

Is it possible to just add a comment syntax to a line to avoid flagging the line as an error?

E.g.

 ---
  valid_yml
  valid_yml 
  invalid_yml # !!! Skip this during validation
  valid_yml

[sirosen]

Is it not possible for you to run whatever preprocessing you want in a separate script, and then pass the result to check-jsonschema? Where does that approach fail?

azure devops preprocesses their yml pipelines prior to running their schema against the .yml files

Can you point me at where this happens? I'm curious about what they're doing, but wasn't able to find anything obvious in the azure pipelines vscode repo which does this.

---

My initial reaction to this is that I don't want to support it. check-jsonschema is primarily built as a way of checking that a schema validates on a file. If the file is not valid because of <external-very-good-reason>, that's out of scope.

That said, if we can specify this in a way which is simple enough to implement, and the azure use-case is compelling enough, I could possibly be convinced that it's a good idea.

[davewhitters]

The "Each" keyword is where I ran into difficulty.
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops

I've been using this to preprocess the files, but then realized the schema validation is also occurring on their end as well which was a nice side-effect. https://docs.microsoft.com/en-us/rest/api/azure/devops/pipelines/runs/run-pipeline?view=azure-devops-rest-6.0

I'd prefer to not use a REST API and use your tool instead though.

[sirosen]

I haven't had a chance to look into this more deeply, but let me see if I'm understanding you correctly.
You're sending a pipeline file to the run-pipeline API with previewRun=true to get back a new YAML document with any templating steps in Azure already applied. Is that right?

Can you provide a simplified example of a file which is failing to validate under the schema, but which validates correctly under the run-pipeline call?

check-jsonschema is pulling the Azure Pipelines schema from a Microsoft project which maintains a schema for use in VSCode:
https://github.com/microsoft/azure-pipelines-vscode
If it doesn't validate here, then it would presumably fail to validate in VSCode. If it fails here and does validate in VSCode, that would be worth looking into.

[davewhitters]

Any file that utilizes the each keyword will fail locally as it is expanded by the server prior to schema validation.

E.g.

---
parameters:
  - name: vals
    type: object
    default:
       - 1
       - 2

jobs:
  - ${{ each val in parameter.vals }}:
    - job:
      displayName: Job
      steps:
        - bash: echo ${{ val }}

[sirosen]

I asked in the pipelines-vscode repo about how they handle this, and they directed me to the appropriate source, in the pipelines language-server code.

I see there are helpers in which their language-server code is modifying any expressions:
https://github.com/microsoft/azure-pipelines-language-server/blob/71b20f92874c02dfe82ad2cc2dcc7fa64996be91/language-service/src/parser/yamlParser.ts#L182

Basically, in the case of the each loop, they're "lifting" the contents up one level.
That is

jobs:
  - ${{ each val in parameter.vals }}:
    - job: foo
      steps:
        - bash: echo ${{ val }}

is becoming

jobs:
  - job: foo
    steps:
      - bash: echo ${{ val }}

which will pass schema validation.

There's a similar transform which I haven't quite grasped yet for arrays with expressions like conditionals.

I will have to think more about this, but I understand this use-case a lot better now. I'm thinking about possibly implementing a new feature to transform data prior to passing it to schema validation.
Off the top of my head, the feature I imagine is a new flag something like

check-jsonschema --builtin-schema vendor.azure-pipelines --instance-transform azure-pipelines azure-pipelines.yml

This would be built into the pre-commit hook, and I would have to give some thought to how to make it extensible.

@davewhitters, does that approach, having a transform which does the same work as what Microsoft already does for VSCode, sound satisfactory to you?

[dwhitters]

This approach sounds great; it should satisfy the use-case I described.

[sirosen]

I have a version of this up and running now, and I've worked out enough issues with it that I think I'll be comfortable publishing it soon, after a bit more testing.

There was one thing that I wanted to note about the sample pipeline you provided:

---
parameters:
  - name: vals
    type: object
    default:
       - 1
       - 2

jobs:
  - ${{ each val in parameter.vals }}:
    - job:
      displayName: Job
      steps:
        - bash: echo ${{ val }}

This fails validation with an error even on the branch of check-jsonschema that adds the azure pipelines transformation:

$ check-jsonschema --builtin-schema vendor.azure-pipelines --data-transform azure-pipelines ./pipeline.yaml
Schema validation errors were encountered.
  ./pipeline.yaml::$: {'parameters': [{'name': 'vals', 'type': 'object', 'default': [1, 2]}], 'jobs': [{'job': None, 'displayName': 'Job', 'steps': [{'bash': 'echo ${{ val }}'}]}]} is not valid under any of the given schemas

At first I thought there was something wrong with the code, but then I found the explanation when I added an option to show all errors. Because the structure of the azure pipelines schema uses many nested anyOf and oneOf clauses, there are a lot of failed matches (and therefore distinct errors). To pull the relevant bits from that:

    $.jobs[0].job: None is not of type 'string'
    $.parameters[0].default[0]: 1 is not valid under any of the given schemas
    $.parameters[0].default[0]: 1 is not of type 'string'
    $.parameters[0].default[0]: 1 is not of type 'array'
    $.parameters[0].default[0]: 1 is not of type 'object'
    $.parameters[0].default[1]: 2 is not valid under any of the given schemas
    $.parameters[0].default[1]: 2 is not of type 'string'
    $.parameters[0].default[1]: 2 is not of type 'array'
    $.parameters[0].default[1]: 2 is not of type 'object'

That is, the schema from azure pipelines job to be a string, not null, and default values can be a list of strings (or several other types), but a list of ints is no good.

The pipeline will pass if we explicitly set job to a string -- even '' works -- and quote the default array values:

---
parameters:
  - name: vals
    type: object
    default:
       - '1'
       - '2'

jobs:
  - ${{ each val in parameter.vals }}:
    - job: 'foo'
      displayName: Job
      steps:
        - bash: echo ${{ val }}

[sirosen]

I've just released v0.11.0 with support for this. There isn't very much I can do about the fact that different yaml parsers may handle ambiguous parses differently, or cases like the above where an integer is probably accepted by the service but is not listed in the schema.

However, if you run into issues using it which you think I can solve, please let me know!