DAG-2731 Add "--display-variant-name" option (#16)

evergreen-ci · Jul 31, 2023 · bf4fc39 · bf4fc39
1 parent 6e2d161
commit bf4fc39
Show file tree

Hide file tree

Showing 17 changed files with 1,124 additions and 792 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,8 @@
 # Changelog
 
+## 4.0.0 - 2023-07-31
+- Add `--display-variant-name` optional argument.
+
 ## 3.0.6 - 2022-11-07
 - Change default build variants to required for mongo projects.
 

diff --git a/README.md b/README.md
@@ -129,10 +129,11 @@ Usage: git-co-evg-base [OPTIONS]
 
   Examples
 
-  Working on a fix for a task 'replica_sets' on the build variants 'enterprise-rhel-80-64-bit' and
-  'enterprise-windows', to ensure the task has been run on those build variants:
+  Working on a fix for a task 'replica_sets_jscore_passthrough' on the build variants '! Amazon
+  Linux 2 arm64 (all feature flags)' and '! Enterprise Windows (all feature flags)', to ensure the
+  task has been run on those build variants:
 
-      git co-evg-base --run-task replica_sets --build-variant enterprise-rhel-80-64-bit --build-variant enterprise-windows
+      git co-evg-base --run-task replica_sets_jscore_passthrough --display-variant-name "^! Amazon Linux 2 arm64 (all feature flags)$" --display-variant-name "^! Enterprise Windows (all feature flags)$"
 
   Starting a new change, to ensure that there are no systemic failures on the base commit:
 
@@ -146,18 +147,21 @@ Options:
   --run-threshold FLOAT           Specify the percentage of tasks that need to be run.
   --pass-threshold FLOAT          Specify the percentage of tasks that need to be successful.
   --fail-threshold FLOAT          Specify the percentage of tasks that need to be failed.
-  --evg-config-file PATH          File containing evergreen authentication information
-                                  [default=~/.evergreen.yml].
-  --evg-project TEXT              Evergreen project to query against [default=mongodb-mongo-
-                                  master].
+  --evg-config-file PATH          File containing evergreen authentication information.  [default:
+                                  /Users/<username>/.evergreen.yml]
+  --evg-project TEXT              Evergreen project to query against.  [default: mongodb-mongo-
+                                  master]
   --build-variant TEXT            Regex of Build variants to check (can be specified multiple
-                                  times) [default=.*-required$].
-  --commit-lookback INTEGER       Number of commits to check before giving up [default=50].
+                                  times).
+  --display-variant-name TEXT     Regex of Build variant display names to check (can be specified
+                                  multiple times).  [default: .*]
+  --commit-lookback INTEGER       Number of commits to check before giving up.  [default: 50]
   --timeout-secs INTEGER          Number of seconds to search for before giving up.
   --commit-limit TEXT             Oldest commit to check before giving up.
   --git-operation [checkout|rebase|merge|none]
-                                  Git operations to perform with found commit [default=none].
-  -b, --branch TEXT               Name of branch to create on checkout.
+                                  Git operations to perform with found commit.  [default: none]
+  -b, --branch TEXT               Name of branch to create on checkout. When specified, git-
+                                  operation is set to checkout by default.
   --save-criteria TEXT            Save the specified criteria rules under the specified name for
                                   future use.
   --use-criteria TEXT             Use previously save criteria rules.
@@ -167,7 +171,7 @@ Options:
   --export-file PATH              File to write exported rules to.
   --import-criteria PATH          Import previously exported criteria.
   --output-format [plaintext|yaml|json]
-                                  Format of the command output [default=plaintext].
+                                  Format of the command output.  [default: plaintext]
   --verbose                       Enable debug logging.
   --help                          Show this message and exit.
 ```
@@ -178,11 +182,11 @@ Checkout using the default criteria:
 $ git co-evg-base
 ```
 
-Checkout with successful tasks 'auth' and 'auth_audit' on builds 'enterprise-windows' and
-'enterprise-rhel-80-64-bit' and 95% of the tasks are passing.
+Checkout with successful tasks 'auth' and 'auth_audit' on builds '! Enterprise Windows (all feature flags)'
+and '! Amazon Linux 2 arm64 (all feature flags)' and 95% of the tasks are passing.
 
 ```bash
-$ git co-evg-base --passing-task auth --passing-task auth_audit --run-threshold 0.95 --build-variant enterprise-windows --build-variant enterprise-rhel-80-64-bit
+$ git co-evg-base --passing-task auth --passing-task auth_audit --run-threshold 0.95 --display-variant-name "^! Enterprise Windows (all feature flags)$" --display-variant-name "^! Amazon Linux 2 arm64 (all feature flags)$"
 ```
 
 ## Contributor's Guide

diff --git a/docs/usage.md b/docs/usage.md
@@ -92,18 +92,23 @@ git co-evg-base --pass-threshold 0.9 --passing-task noPassthrough --passing-task
 ### Applying checks to build variants
 
 In projects with multiple build variants, you may not desire to apply the criteria to every build
-variant. The `--build-variant` option allows you to control which build variants the checks should
-apply. The option takes a regular expression as an argument. Any build variants that match against
-the regular express will have their criteria checked.
+variant. The `--display-variant-name` and/or `--build-variant` options allow you to control which
+build variants the checks should apply. Options take a regular expressions as an argument. Any build
+variants that match against the regular express will have their criteria checked.
 
-The `--build-variant` option can be specified multiple times to provide multiple regular expression
-to check against.
+The `--display-variant-name` and/or `--build-variant` option can be specified multiple times to
+provide multiple regular expression to check against.
 
-For example, to check that a task was successful on builds that end with "-required" and "-suggested",
-I would run the following:
+For example, to check that a task was successful on builds that end with "-required" and "-suggested"
+and which display names start with "! " and "* ", I would run the following:
 
 ```bash
-git co-evg-base --passing-task compile_dist_test --build-variant ".*-required" --build-variant ".*-suggested"
+git co-evg-base \
+  --passing-task compile_dist_test \
+  --display-variant-name "^! .*" \
+  --display-variant-name "^\* .*" \
+  --build-variant ".*-required$" \
+  --build-variant ".*-suggested$"
 ```
 
 ## Specifying the Evergreen project
@@ -144,7 +149,7 @@ For example, to create a branch named `my-branch`, use the following:
 ```bash
 git co-evg-base --git-operation checkout --branch my-branch
 ```
-The same result can be achieved by using the `-b` or `--branch` option without `--git-operation`, in this 
+The same result can be achieved by using the `-b` or `--branch` option without `--git-operation`, in this
 case, since the branch name is specified, git operation is set to **checkout** by default:
 ```bash
 git co-evg-base --branch my-branch
@@ -160,9 +165,9 @@ Find and print a revision that meets the criteria, but perform no actions on the
 ```bash
 git co-evg-base --pass-threshold 0.85 --git-operation none
 ```
-Git operation is by default **none**. In some cases git operation may be set to other action than **none**, 
-for instance when using `--branch` it is set to **checkout** by default. Unless you have to override those, 
-you can omit the `--git-operation none` part. The same result as above can be achieved by using this: 
+Git operation is by default **none**. In some cases git operation may be set to other action than **none**,
+for instance when using `--branch` it is set to **checkout** by default. Unless you have to override those,
+you can omit the `--git-operation none` part. The same result as above can be achieved by using this:
 ```bash
 git co-evg-base --pass-threshold 0.85
 ```
@@ -217,15 +222,15 @@ different build variants.
 Save a "b-grade" criteria with a pass threshold of 80% on all build variants.
 
 ```bash
-git co-evg-base --pass-threshold 0.8 --build-variant ".*" --save-criteria b-grade
+git co-evg-base --pass-threshold 0.8 --display-variant-name ".*" --save-criteria b-grade
 ```
 
 Save a required criteria with a pass threshold of 95% on all required build variants and the
-"compile_dist_test" task passing on the "enterprise-macos" build variant.
+"compile_dist_test" task passing on the "Enterprise macOS" build variant.
 
 ```bash
-git co-evg-base --pass-threshold 0.95 --build-variant ".*-required" --save-criteria required
-git co-evg-base --passing-task compile_dist_test --build-variant "^enterprise-macos$" --save-criteria required
+git co-evg-base --pass-threshold 0.95 --display-variant-name "^! .*" --save-criteria required
+git co-evg-base --passing-task compile_dist_test --display-variant-name "^Enterprise macOS$" --save-criteria required
 ```
 
 ### Using saved criteria
@@ -250,20 +255,20 @@ saved criteria.
 
 ```bash
 git co-evg-base --list-criteria
-                                  b-grade
-┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
-┃ Build Variant Regexes ┃ Success % ┃ Run % ┃ Successful Tasks ┃ Run Tasks ┃
-┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
-│ .*                    │ 0.8       │       │                  │           │
-└───────────────────────┴───────────┴───────┴──────────────────┴───────────┘
-                                  required
-┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
-┃ Build Variant Regexes ┃ Success % ┃ Run % ┃ Successful Tasks  ┃ Run Tasks ┃
-┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
-│ .*-required           │ 0.95      │       │                   │           │
-├───────────────────────┼───────────┼───────┼───────────────────┼───────────┤
-│ ^enterprise-macos$    │           │       │ compile_dist_test │           │
-└───────────────────────┴───────────┴───────┴───────────────────┴───────────┘
+                                              b-grade
+┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
+┃ Build Variant Regexes ┃ Display Name Regexes ┃ Success % ┃ Run % ┃ Successful Tasks ┃ Run Tasks ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
+│                       │ .*                   │ 0.8       │       │                  │           │
+└───────────────────────┴──────────────────────┴───────────┴───────┴──────────────────┴───────────┘
+                                              required
+┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
+┃ Build Variant Regexes ┃ Display Name Regexes ┃ Success % ┃ Run % ┃ Successful Tasks  ┃ Run Tasks ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
+│                       │ ^! .*                │ 0.95      │       │                   │           │
+├───────────────────────┼──────────────────────┼───────────┼───────┼───────────────────┼───────────┤
+│                       │ ^Enterprise macOS$   │           │       │ compile_dist_test │           │
+└───────────────────────┴──────────────────────┴───────────┴───────┴───────────────────┴───────────┘
 ```
 
 ## Sharing Criteria

diff --git a/goodbase-docs/content/concepts/criteria.md b/goodbase-docs/content/concepts/criteria.md
@@ -2,21 +2,22 @@
 weight: 1
 ---
 
-`git-co-evg-base` allows you to specify criteria that define what a matching evergreen 
+`git-co-evg-base` allows you to specify criteria that define what a matching evergreen
 version will look like.
 
-There are 4 types of criteria that can be specified.
+There are 5 types of criteria that can be specified.
 
 * passing tasks
 * run tasks
 * pass threshold
+* fail threshold
 * run threshold
 
 ## Passing tasks
 
 Passing tasks are tasks that must have completed successfully in Evergreen to meet the criteria.
 They are specified with the `--passing-task` option. This option can be specified more than once
-to include multiple tasks. 
+to include multiple tasks.
 
 For example, to ensure I get a revision with the following three tasks passing: `auth`, `auth_audit`,
 and `noPassthrough`, I would run the following:
@@ -27,7 +28,7 @@ git co-evg-base --passing-task auth --passing-task auth_audit --passing-task noP
 
 ## Run tasks
 
-Run tasks are similar to passing tasks except they only need to have been executed, they do not 
+Run tasks are similar to passing tasks except they only need to have been executed, they do not
 need to have been successful. This can be useful when working on a fix for a known task failure.
 They are specified with the `--run-task` option. This option can be specified more than once
 to include multiple tasks.
@@ -43,14 +44,25 @@ git co-evg-base --run-task jsCore --run-task aggregation
 
 Pass threshold ensures that some percentage of tasks in the build variants must have passed to
 consider the revision. This can help minimize the number of unrelated failures that might show up
-in patch builds. This is specified with the `--pass-threshold` option. 
+in patch builds. This is specified with the `--pass-threshold` option.
 
 For example, to ensure that 85% of tasks have passed, I would run the following:
 
 ```bash
 git co-evg-base --pass-threshold 0.85
 ```
 
+## Fail threshold
+
+Fail threshold is opposite to the pass threshold. This can help to find the build with certain
+amount of failures.
+
+For example, to ensure that 15% of tasks have been failed, I would run the following:
+
+```bash
+git co-evg-base --fail-threshold 0.15
+```
+
 ## Run threshold
 
 Run threshold is similar to the pass threshold, but tasks only need to have been executed, they do
@@ -73,4 +85,3 @@ check that certain tasks pass:
 ```bash
 git co-evg-base --pass-threshold 0.9 --passing-task noPassthrough --passing-task buildscripts_test
 ```
-
diff --git a/goodbase-docs/content/concepts/criteria_application.md b/goodbase-docs/content/concepts/criteria_application.md
@@ -4,18 +4,23 @@ weight: 2
 ## Applying checks to build variants
 
 In projects with multiple build variants, you may not desire to apply the criteria to every build
-variant. The `--build-variant` option allows you to control which build variants the checks should
-apply. The option takes a regular expression as an argument. Any build variants that match against
-the regular express will have their criteria checked. 
+variant. The `--display-variant-name` and/or `--build-variant` options allow you to control which
+build variants the checks should apply. Options take a regular expressions as an argument. Any build
+variants that match against the regular express will have their criteria checked.
 
-The `--build-variant` option can be specified multiple times to provide multiple regular expression
-to check against.
+The `--display-variant-name` and/or `--build-variant` option can be specified multiple times to
+provide multiple regular expression to check against.
 
-For example, to check that a task was successful on builds that end with "-required" and "-suggested", 
-I would run the following:
+For example, to check that a task was successful on builds that end with "-required" and "-suggested"
+and which display names start with "! " and "* ", I would run the following:
 
 ```bash
-git co-evg-base --passing-task compile_dist_test --build-variant ".*-required" --build-variant ".*-suggested"
+git co-evg-base \
+  --passing-task compile_dist_test \
+  --display-variant-name "^! .*" \
+  --display-variant-name "^\* .*" \
+  --build-variant ".*-required$" \
+  --build-variant ".*-suggested$"
 ```
 
 ## Specifying the Evergreen project