Implement validation of configuration changes. (#628)

If the "validate_config" option is set in the pipeline configuration, all changes to pipeline configuration files in the .bazelci folder will be validated. Currently this means that CI tries to print a project pipeline using that configuration.

Examples:

- https://buildkite.com/bazel/fwe-test/builds/35: No validation step since no configs were modified.
- https://buildkite.com/bazel/fwe-test/builds/36: Successful validation of a commit that changed two configuration files correctly.
- https://buildkite.com/bazel/fwe-test/builds/38: Validation failed due to bug in the configuration.

buildkite/README.md

@@ -329,3 +329,19 @@ tasks:
     test_targets:
     - //...
 ```
+
+### Validating changes to pipeline configuration files
+
+You can set the top-level `validate_config` option to ensure that changes to pipeline configuration files in the `.bazelci` directory will be validated.
+With this option, every build for a commit that touches a configuration file will contain an additional validation step for each modified configuration file.
+
+Example usage:
+
+```yaml
+---
+validate_config: 1
+tasks:
+  macos:
+    build_targets:
+    - "..."
+```

buildkite/bazelci.py

@@ -388,6 +388,8 @@
 
 SKIP_TASKS_ENV_VAR = "CI_SKIP_TASKS"
 
+CONFIG_FILE_EXTENSIONS = set([".yml", ".yaml"])
+
 
 class BuildkiteException(Exception):
     """
@@ -1559,6 +1561,37 @@ def SetEnvVar(config_key, env_var_name):
             )
         )
 
+    if "validate_config" in configs:
+        output = execute_command_and_get_output(
+            [
+                "git",
+                "diff-tree",
+                "--no-commit-id",
+                "--name-only",
+                "-r",
+                os.getenv("BUILDKITE_COMMIT"),
+            ]
+        )
+        config_files = [
+            l
+            for l in output.split("\n")
+            if l.startswith(".bazelci/") and os.path.splitext(l)[1] in CONFIG_FILE_EXTENSIONS
+        ]
+        platform = DEFAULT_PLATFORM
+        for f in config_files:
+            pipeline_steps.append(
+                create_step(
+                    label=":cop: Validate {}".format(f),
+                    commands=[
+                        fetch_bazelcipy_command(),
+                        "{} bazelci.py project_pipeline --file_config={}".format(
+                            python_binary(platform), f
+                        ),
+                    ],
+                    platform=platform,
+                )
+            )
+
     print(yaml.dump({"steps": pipeline_steps}))