CircleCI command line verify verifies invalid config file(s)

[haikusw]

Description

The circleci command line tool config verify command doesn't seem to do a very thorough job or actually validate against the configuration file schema in the way that the actual build system does.

For example, the following config file passes verification:

tyler$ circleci config validate
Config file at .circleci/config.yml is valid.

config file:

vesion: 2
jobs:
  build: 
    macos:
      xcode: "11.0.0"
    
    steps:
      - checkout
    
      - run:
          name: Build and run tests
          command: fastlane scan
          environment:
            SCAN_DEVICE: iPhone Xʀ (13.0)
            SCAN_SCHEME: test_iOS_NoSwiftUI
        
      - store_test_results:
          path: test_output/report.xml
          
      - store_artifacts:
          path: /tmp/test-results
          destination: scan-test-results
          
      - store_artifacts:
          path: ~/Library/Logs/scan
          destination: scan-logs

however "version" is misspelled as "vesion" and gets a warning in the "Configuration" section of the CircleCI dashboard:

# -------- WARNING ---------
# Your config does not conform to the schema.
# Violations are listed below:
# 
# ERROR IN CONFIG FILE:
# [#] required key [version] not found
vesion: 2
jobs:
  build:
    macos:
      xcode: 11.0.0
    steps:
    - checkout
    - run:
        name: Build and run tests
        command: fastlane scan
        environment:
          SCAN_DEVICE: iPhone Xʀ (13.0)
          SCAN_SCHEME: test_iOS_NoSwiftUI
    - store_test_results:
        path: test_output/report.xml
    - store_artifacts:
        path: /tmp/test-results
        destination: scan-test-results
    - store_artifacts:
        path: ~/Library/Logs/scan
        destination: scan-logs
workflows:
  version: 2
  workflow:
    jobs:
    - build

It seems like the circleci config validate command should emit these same warnings.

---

More importantly though, a config file which will cause a build failure also passes circleci config validate :-/

If you put fewer spaces in your indentation for the - run entry like this:

version: 2
jobs:
..build: 
....macos:
......xcode: "11.0.0"
....
....steps:
......- checkout
....
......- run:
........name: Build and run tests
........command: fastlane scan
........environment:
..........SCAN_DEVICE: iPhone Xʀ (13.0)
..........SCAN_SCHEME: test_iOS_NoSwiftUI
........
......- store_test_results:
........path: test_output/report.xml
..........
......- store_artifacts:
........path: /tmp/test-results
........destination: scan-test-results
..........
......- store_artifacts:
........path: ~/Library/Logs/scan
........destination: scan-logs

it again passes validation:

tyler$ circleci config validate
Config file at .circleci/config.yml is valid.

but the build will fail with a ton of issues in the output:

# -------- WARNING ---------
# Your config does not conform to the schema.
# Violations are listed below:
# 
# ERROR IN CONFIG FILE:
# [#/jobs/build] only 1 subschema matches out of 2
# 1. [#/jobs/build/steps] 21 schema violations found
# |   1. [#/jobs/build/steps/1] 0 subschemas matched instead of one
# |   |   1. [#/jobs/build/steps/1] 6 schema violations found
# |   |   |   1. [#/jobs/build/steps/1] extraneous key [command] is not permitted
# |   |   |   |   Permitted keys:
# |   |   |   |     - persist_to_workspace
# |   |   |   |     - save_cache
# |   |   |   |     - run
# |   |   |   |     - checkout
# |   |   |   |     - attach_workspace
# |   |   |   |     - store_test_results
# |   |   |   |     - restore_cache
# |   |   |   |     - store_artifacts
# |   |   |   |     - add_ssh_keys
# |   |   |   |     - deploy
# |   |   |   |     - setup_remote_docker
# |   |   |   |   Passed keys:
# |   |   |   |     - run
# |   |   |   |     - name
# |   |   |   |     - command
# |   |   |   |     - environment
# |   |   |   2. [#/jobs/build/steps/1] extraneous key [environment] is not permitted
# |   |   |   |   Permitted keys:
# |   |   |   |     - persist_to_workspace
# |   |   |   |     - save_cache
# |   |   |   |     - run
# |   |   |   |     - checkout
# |   |   |   |     - attach_workspace
# |   |   |   |     - store_test_results
# |   |   |   |     - restore_cache
# |   |   |   |     - store_artifacts
# |   |   |   |     - add_ssh_keys
# |   |   |   |     - deploy
# |   |   |   |     - setup_remote_docker
# |   |   |   |   Passed keys:
# |   |   |   |     - run
# |   |   |   |     - name
# |   |   |   |     - command
# |   |   |   |     - environment
# |   |   |   3. [#/jobs/build/steps/1] extraneous key [name] is not permitted
# |   |   |   |   Permitted keys:
# |   |   |   |     - persist_to_workspace
# |   |   |   |     - save_cache
# |   |   |   |     - run
# |   |   |   |     - checkout
# |   |   |   |     - attach_workspace
# |   |   |   |     - store_test_results
# |   |   |   |     - restore_cache
# |   |   |   |     - store_artifacts
# |   |   |   |     - add_ssh_keys
# |   |   |   |     - deploy
# |   |   |   |     - setup_remote_docker
# |   |   |   |   Passed keys:
# |   |   |   |     - run
# |   |   |   |     - name
# |   |   |   |     - command
# |   |   |   |     - environment
# |   |   |   4. [#/jobs/build/steps/1] maximum size: [1], found: [4]
# |   |   |   |   SCHEMA:
# |   |   |   |     maxProperties: 1
# |   |   |   |   INPUT:
# |   |   |   |     run: {}
# |   |   |   |     name: Build and run tests
# |   |   |   |     command: fastlane scan
# |   |   |   |     environment:
# |   |   |   |       SCAN_DEVICE: iPhone X\u0280 (13.0)
# |   |   |   |       SCAN_SCHEME: test_iOS_NoSwiftUI
# |   |   |   5. [#/jobs/build/steps/1/run] 0 subschemas matched instead of one
# |   |   |   |   1. [#/jobs/build/steps/1/run] expected type: String, found: Mapping
# |   |   |   |   |   {run \"...\"} is shorthand for `{run {command \"...\"}}`
# |   |   |   |   |   SCHEMA:
# |   |   |   |   |     type: string
# |   |   |   |   |   INPUT:
# |   |   |   |   |     null
# |   |   |   |   2. [#/jobs/build/steps/1/run] required key [command] not found
# |   |   2. [#/jobs/build/steps/1] Input not a valid enum value
# |   |   |   Steps without arguments can be called as strings
# |   |   |     enum:
# |   |   |     - checkout
# |   |   |     - setup_remote_docker
# |   |   |     - add_ssh_keys
# |   2. [#/jobs/build/steps/2] 0 subschemas matched instead of one
# |   |   1. [#/jobs/build/steps/2] 3 schema violations found
# |   |   |   1. [#/jobs/build/steps/2] extraneous key [path] is not permitted
# |   |   |   |   Permitted keys:
# |   |   |   |     - persist_to_workspace
# |   |   |   |     - save_cache
# |   |   |   |     - run
# |   |   |   |     - checkout
# |   |   |   |     - attach_workspace
# |   |   |   |     - store_test_results
# |   |   |   |     - restore_cache
# |   |   |   |     - store_artifacts
# |   |   |   |     - add_ssh_keys
# |   |   |   |     - deploy
# |   |   |   |     - setup_remote_docker
# |   |   |   |   Passed keys:
# |   |   |   |     - store_test_results
# |   |   |   |     - path
# |   |   |   2. [#/jobs/build/steps/2] maximum size: [1], found: [2]
# |   |   |   |   SCHEMA:
# |   |   |   |     maxProperties: 1
# |   |   |   |   INPUT:
# |   |   |   |     store_test_results: {}
# |   |   |   |     path: test_output/report.xml
# |   |   |   3. [#/jobs/build/steps/2/store_test_results] required key [path] not found
# |   |   2. [#/jobs/build/steps/2] Input not a valid enum value
# |   |   |   Steps without arguments can be called as strings
# |   |   |     enum:
# |   |   |     - checkout
# |   |   |     - setup_remote_docker
# |   |   |     - add_ssh_keys
# |   3. [#/jobs/build/steps/3] 0 subschemas matched instead of one
# |   |   1. [#/jobs/build/steps/3] 4 schema violations found
# |   |   |   1. [#/jobs/build/steps/3] extraneous key [destination] is not permitted
# |   |   |   |   Permitted keys:
# |   |   |   |     - persist_to_workspace
# |   |   |   |     - save_cache
# |   |   |   |     - run
# |   |   |   |     - checkout
# |   |   |   |     - attach_workspace
# |   |   |   |     - store_test_results
# |   |   |   |     - restore_cache
# |   |   |   |     - store_artifacts
# |   |   |   |     - add_ssh_keys
# |   |   |   |     - deploy
# |   |   |   |     - setup_remote_docker
# |   |   |   |   Passed keys:
# |   |   |   |     - store_artifacts
# |   |   |   |     - path
# |   |   |   |     - destination
# |   |   |   2. [#/jobs/build/steps/3] extraneous key [path] is not permitted
# |   |   |   |   Permitted keys:
# |   |   |   |     - persist_to_workspace
# |   |   |   |     - save_cache
# |   |   |   |     - run
# |   |   |   |     - checkout
# |   |   |   |     - attach_workspace
# |   |   |   |     - store_test_results
# |   |   |   |     - restore_cache
# |   |   |   |     - store_artifacts
# |   |   |   |     - add_ssh_keys
# |   |   |   |     - deploy
# |   |   |   |     - setup_remote_docker
# |   |   |   |   Passed keys:
# |   |   |   |     - store_artifacts
# |   |   |   |     - path
# |   |   |   |     - destination
# |   |   |   3. [#/jobs/build/steps/3] maximum size: [1], found: [3]
# |   |   |   |   SCHEMA:
# |   |   |   |     maxProperties: 1
# |   |   |   |   INPUT:
# |   |   |   |     store_artifacts: {}
# |   |   |   |     path: /tmp/test-results
# |   |   |   |     destination: scan-test-results
# |   |   |   4. [#/jobs/build/steps/3/store_artifacts] required key [path] not found
# |   |   2. [#/jobs/build/steps/3] Input not a valid enum value
# |   |   |   Steps without arguments can be called as strings
# |   |   |     enum:
# |   |   |     - checkout
# |   |   |     - setup_remote_docker
# |   |   |     - add_ssh_keys
# |   4. [#/jobs/build/steps/4] 0 subschemas matched instead of one
# |   |   1. [#/jobs/build/steps/4] 4 schema violations found
# |   |   |   1. [#/jobs/build/steps/4] extraneous key [destination] is not permitted
# |   |   |   |   Permitted keys:
# |   |   |   |     - persist_to_workspace
# |   |   |   |     - save_cache
# |   |   |   |     - run
# |   |   |   |     - checkout
# |   |   |   |     - attach_workspace
# |   |   |   |     - store_test_results
# |   |   |   |     - restore_cache
# |   |   |   |     - store_artifacts
# |   |   |   |     - add_ssh_keys
# |   |   |   |     - deploy
# |   |   |   |     - setup_remote_docker
# |   |   |   |   Passed keys:
# |   |   |   |     - store_artifacts
# |   |   |   |     - path
# |   |   |   |     - destination
# |   |   |   2. [#/jobs/build/steps/4] extraneous key [path] is not permitted
# |   |   |   |   Permitted keys:
# |   |   |   |     - persist_to_workspace
# |   |   |   |     - save_cache
# |   |   |   |     - run
# |   |   |   |     - checkout
# |   |   |   |     - attach_workspace
# |   |   |   |     - store_test_results
# |   |   |   |     - restore_cache
# |   |   |   |     - store_artifacts
# |   |   |   |     - add_ssh_keys
# |   |   |   |     - deploy
# |   |   |   |     - setup_remote_docker
# |   |   |   |   Passed keys:
# |   |   |   |     - store_artifacts
# |   |   |   |     - path
# |   |   |   |     - destination
# |   |   |   3. [#/jobs/build/steps/4] maximum size: [1], found: [3]
# |   |   |   |   SCHEMA:
# |   |   |   |     maxProperties: 1
# |   |   |   |   INPUT:
# |   |   |   |     store_artifacts: {}
# |   |   |   |     path: ~/Library/Logs/scan
# |   |   |   |     destination: scan-logs
# |   |   |   4. [#/jobs/build/steps/4/store_artifacts] required key [path] not found
# |   |   2. [#/jobs/build/steps/4] Input not a valid enum value
# |   |   |   Steps without arguments can be called as strings
# |   |   |     enum:
# |   |   |     - checkout
# |   |   |     - setup_remote_docker
# |   |   |     - add_ssh_keys
version: 2
jobs:
  build:
    macos:
      xcode: 11.0.0
    steps:
    - checkout
    - run: {}
      name: Build and run tests
      command: fastlane scan
      environment:
        SCAN_DEVICE: iPhone Xʀ (13.0)
        SCAN_SCHEME: test_iOS_NoSwiftUI
    - store_test_results: {}
      path: test_output/report.xml
    - store_artifacts: {}
      path: /tmp/test-results
      destination: scan-test-results
    - store_artifacts: {}
      path: ~/Library/Logs/scan
      destination: scan-logs
workflows:
  version: 2
  workflow:
    jobs:
    - build

Configuration files which cause the build system to fail like this don't seem like they should bass a circleci config validate

It also seems like the indentation after a - run: of two spaces -- like it is everywhere else -- would be a reasonable and more consistent requirement rather than the 4 spaces it seems to be requiring now (?).

---

Checked the version of the circleci CLI I installed via homebrew:

tyler$ circleci version
0.1.5810+ac0d85f

and does appear that Homebrew doesn't is behind by a minor version. However, the only commits between that version (1.5810) and the current version (1.5830) is a contributor added to CONTRIBUTORS.md and the merge of that PR).

Checked the diagnostics to make sure circleci cli was working properly:

tyler$ circleci diagnostic
---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: /Users/tyler/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API... 
Ok.
Hello, Tyler.

and it appears to be.

[haikusw]

It seems like there must be some validation code in the hosted CircleCI repo that verifies against a schema and that the command line tool isn't but could potentially benefit from sharing that code.

[marcomorain]

Hi @haikusw – the underlying problem here is that the 2.0 config format has schema problems that we addressed in config 2.1 - this post explains a little more. If you upgrade to config version: 2.1 you will get far more robust schema checking. If you upgrade to version 2.1 does the tool behave as expected?