📖 Review and update some probe documentation (#4023)

* polish some probe yaml definitions

Signed-off-by: Spencer Schrock <sschrock@google.com>

* update references to probe naming and outcomes

now that #3654 is addressed, the naming restrictions can be relaxed.

Signed-off-by: Spencer Schrock <sschrock@google.com>

---------

Signed-off-by: Spencer Schrock <sschrock@google.com>

probes/README.md

@@ -1,22 +1,24 @@
-# Scorecard probes
+# OpenSSF Scorecard Probes
 
-This directory contains all the Scorecard probes.
+This directory contains all the Scorecard probes. Each probe has its own subdirectory.
 
-A probe is an assessment of a focused, specific heuristic typically isolated to a particular ecosystem. For example, Scorecards fuzzing check consists of many different probes that assess particular ecosystems or aspects of fuzzing.
+A probe is an individual heuristic, which provides information about a distinct behavior a project under analysis may or may not be doing.
+The probes follow a camelcase naming convention that describe the exact heuristic a particular probe assesses. 
+The name should be phrased in a way that can be answered by "true" or "false".
 
-Each probe has its own directory in `scorecard/probes`. The probes follow a camelcase naming convention that describe the exact heuristic a particular probe assesses. 
-
-Probes can return multiple or a single finding, where a finding is a piece of data with an outcome, message, and optionally a location. Probes should be designed in such a way that a `finding.OutcomePositive` reflects a positive result, and `finding.OutcomeNegative` reflects a negative result. Scorecard has other `finding.Outcome` types available for other results; For example, the `finding.OutcomeNotAvailable` is often used for scenarios, where Scorecard cannot assess a project with a given probe. In addition, probes should also be named in such a way that they answer "yes" or "no", and where "yes" answers positively to the heuristic, and "no" answers negatively. For example, probes that check for SAST tools in the CI are called `toolXXXInstalled` so that `finding.OutcomePositive` reflects that it is positive to use the given tool, and that "yes" reflects what Scorecard considers the positive outcome. For some probes, this can be a bit trickier to do; The `notArchived` probe checks whether a project is archived, however, Scorecard considers archived projects to be negative, and the probe cannot be called `isArchived`. These naming conventions are not hard rules but merely guidelines. Note that probes do not do any formal evaluation such a scoring; This is left to the evaluation part once the outcomes have been produced by the probes.
+Probes can return one or more findings, where a finding is a piece of data with an outcome, message, and optionally a location where the behavior was observed. 
+The primary outcomes are `finding.OutcomeTrue` and `finding.OutcomeFalse`, but other outcomes are available as well.
+For example, the `finding.OutcomeNotAvailable` is often used for scenarios, where Scorecard cannot assess a behavior because there is no data to analyze. 
 
 A probe consists of three files: 
 
 - `def.yml`: The documentation of the probe. 
 - `impl.go`: The actual implementation of the probe.
-- `impl_test.go`: The probes test.
+- `impl_test.go`: The probe's test.
 
 ## Reusing code in probes
 
-When multiple probes use the same code, the reused code can be placed on `scorecard/probes/internal/utils`
+When multiple probes use the same code, the reused code can be placed in a package under `probes/internal/`
 
 ## How do I know which probes to add?
 
@@ -39,7 +41,7 @@ And then in `impl.go` add the following metadata:
 ```golang
 f, err := finding.NewWith(fs, Probe,
 	"Message", nil,
-	finding.OutcomePositive)
+	finding.OutcomeTrue)
 f = f.WithRemediationMetadata(map[string]string{
 	"dataToDisplay": "this is the text we will display",
 })
@@ -55,7 +57,7 @@ and the probe sets the following metadata:
 ```golang
 f, err := finding.NewWith(fs, Probe,
 	"Message", nil,
-	finding.OutcomePositive)
+	finding.OutcomeTrue)
 f = f.WithRemediationMetadata(map[string]string{
 	"oss-fuzz-integration-status": "is",
 })

probes/archived/def.yml

@@ -15,7 +15,7 @@
 id: archived
 short: Check that the project is archived
 motivation: >
-  A project which is not active might not be patched, have its dependencies patched, or be actively tested and used. However, a lack of active maintenance is not necessarily always a problem. Some software, especially smaller utility functions, does not normally need to be maintained. For example, a library that determines if an integer is even would not normally need maintenance unless an underlying implementation language definition changed. A lack of active maintenance should signal that potential users should investigate further to judge the situation.
+  An archived project will not received security patches, and is not actively tested or used.
 implementation: >
   The probe checks the Archived Status of a project.
 outcome:

probes/blocksForcePushOnBranches/def.yml

@@ -15,7 +15,7 @@
 id: blocksForcePushOnBranches
 short: Check that the project blocks force push on its branches.
 motivation: >
-  Allowing non-admins to force push to branches could allow untrusted users to make insecure changes to the behavior of the project.
+  Allowing force pushes to branches could allow those with write access to make insecure changes to the behavior of the project.
 implementation: >
   Checks the protection rules of default and release branches.
 outcome:

probes/branchProtectionAppliesToAdmins/def.yml

@@ -13,9 +13,9 @@
 # limitations under the License.
 
 id: branchProtectionAppliesToAdmins
-short: Check that the projects branch protection rules apply to project admins.
+short: Check that the project's branch protection rules apply to project admins.
 motivation: >
-  Admins can make malicious code changes that users will be unaware of (see CVE-2022-23812); consuming software where this is disabled is encouraged.
+  Admins may be able to bypass branch protection settings which could defeat the purpose of having them.
 implementation: >
   Checks the protection rules of default and release branches.
 outcome:

probes/branchesAreProtected/def.yml

@@ -13,9 +13,9 @@
 # limitations under the License.
 
 id: branchesAreProtected
-short: Check that the projects branches are protected.
+short: Check that the project uses protected branches.
 motivation: >
-  Branches that are not protected may allow excessive actions that could compromise the projects security. 
+  Unprotected branches may allow actions that could compromise the project's security. 
 implementation: >
   Checks the protection rules of default and release branches.
 outcome:
@@ -25,7 +25,7 @@ remediation:
   effort: Low
   text:
     - For Gitlab-hosted project, follow the documentation on how to protect branches, https://docs.gitlab.com/ee/user/project/protected_branches.html
-    - For GitHub-hosted projects, follow [the documentation on protected branches, https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches
+    - For GitHub-hosted projects, follow the documentation on protected branches, https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches
   markdown:
     - For Gitlab-hosted project, follow [the documentation on how to protect branches](https://docs.gitlab.com/ee/user/project/protected_branches.html)
     - For GitHub-hosted projects, follow [the documentation on protected branches](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches)

probes/contributorsFromOrgOrCompany/def.yml

@@ -15,25 +15,24 @@
 id: contributorsFromOrgOrCompany
 short: Checks whether a project has a contributions from users associated with a company or organization.
 motivation: >
-  This probe tries to determine if the project has recent contributors from multiple organizations (e.g., companies). 
-
-  Some projects cannot meet this requirement, such as small projects with only one active participant, or projects with a narrow scope that cannot attract the interest of multiple organizations. See Code Reviews for more information about evaluating projects with a small number of participants.
-  
+  This probe tries to determine if the project has recent contributors from multiple organizations.
+  For some projects, having a diverse group of contributors is an indicator of project health.
 implementation: >
-  The probe looks at the Company field on the user profile for authors of recent commits. To receive the highest score, the project must have had contributors from at least 3 different companies in the last 30 commits.
+  The probe looks at the Company field on the user profile for authors of recent commits. 
+  To receive the highest score, the project must have had contributors from at least 3 different companies in the last 30 commits.
 outcome:
-  - If the project has no contributing organizations or companies, the probe returns 1 OutcomeFalse
-  - If the project has contributing organizations or companies, the probe returns 1 Outcome per org or company.
-  - If the probe fails to process the raw results, it returns nil instead of the findings slice as well as an error.
+  - If the project has no contributing organizations, the probe returns 1 OutcomeFalse
+  - If the project has contributing organizations, the probe returns 1 OutcomeTrue per organization.
 remediation:
   onOutcome: False
   effort: High
   text:
     - Encourage community-driven contributions to your project.
-    - Ask contributors to join their respective organizations, if they have not already. Otherwise, there is no remediation for this probe; it simply provides insight into how many organizations have contributed so that you can make a trust-based decision based on that information.
+    - Ask contributors to join their respective organizations, if they have not already. 
+    - Otherwise, there is no remediation for this probe; it simply provides insight into how many organizations have contributed so that you can make a trust-based decision based on that information.
 ecosystem:
   languages:
     - all
   clients:
     - github
-    - gitlab
+    - gitlab

probes/createdRecently/def.yml

@@ -15,7 +15,8 @@
 id: createdRecently
 short: Checks if the project was created in the last 90 days.
 motivation: >
-  When Scorecard checks the activity of a project in the last 90 days, the project may not have been created before the last 90 days. As such, Scorecard cannot give an accurate score. This probe helps Scorecard assess whether it can give an accurate score when checking the project activity in the last 90 days.
+  Recently created repositories have been used for malicious forks / typosquatting attacks in the past.
+  A newly created repo is not a strong signal on its own, but can be a useful piece of information.  
 implementation: >
   The implementation checks the creation date is within the last 90 days.
 outcome:

probes/dependencyUpdateToolConfigured/def.yml

@@ -16,7 +16,7 @@ id: dependencyUpdateToolConfigured
 short: Check that a dependency update tool config is present.
 motivation: >
   Out-of-date dependencies make a project vulnerable to known flaws and prone to attacks.
-  Tools can automate the process of updating dependencies by scanning for outdated or insecure requirements, and opening a pull request to update them if found.
+  Tools can help the process of updating dependencies by scanning for outdated or insecure requirements, and opening a pull request to update them if found.
 implementation: >
   The implementation looks for the presence of various config files for different dependency update tools.
 outcome:
@@ -31,6 +31,7 @@ remediation:
     - Setup one of [tools we currently detect](https://github.com/ossf/scorecard/blob/main/docs/checks/dependencyupdatetool/README.md).
 ecosystem:
   languages:
+    - c#
     - dockerfile
     - go
     - java
@@ -39,6 +40,9 @@ ecosystem:
     - python
     - ruby
     - rust
+    - scala
+    - swift
+    - typescript
   clients:
     - github
     - gitlab

probes/dismissesStaleReviews/def.yml

@@ -15,7 +15,7 @@
 id: dismissesStaleReviews
 short: Check that the project dismisses stale reviews when new commits are pushed.
 motivation: >
-  When a project does not dismiss stale reviews, contributors can bring their pull requests to an approved state and then make malicious commits.
+  When a project does not dismiss stale reviews, contributors can bring their pull requests to an approved state and then make unreviewed commits.
 implementation: >
   Checks the protection rules of default and release branches.
 outcome:

probes/hasBinaryArtifacts/def.yml

@@ -15,12 +15,12 @@
 id: hasBinaryArtifacts
 short: Checks if the project has any binary files in its source tree.
 motivation: >
-  Binary files are not readable so users can't see what they do. Many programming language systems can generate executables from source code (e.g., C/C++ generated machine code, Java .class files, Python .pyc files, and minified JavaScript). Users will often directly use executables if they are included in the source repository, leading to many dangerous behaviors.
+  Binary files are not human readable so users and reviewers can't easily see what they do.
 implementation: >
   The implementation looks for the presence of binary files. This is a more restrictive probe than "hasUnverifiedBinaryArtifacts" which excludes verified binary files.
 outcome:
-  - If the probe finds binary files, it returns a number of true outcomes equal to the number of binary files found. Each outcome includes a location of the file.
-  - If the probe finds no binary files, it returns a single false outcome.
+  - If the probe finds binary files, it returns one OutcomeTrue for each binary file found.
+  - If the probe finds no binary files, it returns a single OutcomeFalse.
 remediation:
   onOutcome: True
   effort: Medium
@@ -33,3 +33,4 @@ ecosystem:
   clients:
     - github
     - gitlab
+    - localdir

probes/hasDangerousWorkflowScriptInjection/def.yml

@@ -15,19 +15,20 @@
 id: hasDangerousWorkflowScriptInjection
 short: Check whether the project has GitHub Actions workflows that enable script injection.
 motivation: >
-  Script Injection with Untrusted Context Variables: This pattern detects whether a workflow's inline script may execute untrusted input from attackers. This occurs when an attacker adds malicious commands and scripts to a context. When a workflow runs, these strings may be interpreted as code that is executed on the runner. Attackers can add their own content to certain github context variables that are considered untrusted, for example, github.event.issue.title. These values should not flow directly into executable code.
+  Script injections allow attackers to use untrusted input to access privileged resources (code execution, secret exfiltration, etc.)
 implementation: >
-  The probe iterates through the workflows from the raw results and checks the workflow type. If it finds a workflow of the type `DangerousWorkflowScriptInjection`, it returns.
+  The probe analyzes the repository's workflows for known dangerous patterns.
 outcome:
-  - If the project has at least one workflow with possibility of script injection, the probe returns one finding with OutcomeTrue.
-  - If the project does not have a single workflow with possibility of script injection, the probe returns one finding with OutcomeFalse.
+  - The probe returns one finding with OutcomeTrue for each dangerous script injection pattern detected.
+  - If no dangerous patterns are found, the probe returns one finding with OutcomeFalse.
 remediation:
   onOutcome: True
   effort: Low
   text:
     - Avoid the dangerous workflow patterns.
   markdown:
-    - Avoid the dangerous workflow patterns. See [this post](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/) for information on avoiding untrusted code checkouts. See [this document](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#understanding-the-risk-of-script-injections) for information on avoiding and mitigating the risk of script injections.
+    - Avoid the dangerous workflow patterns.
+    - See [this document](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#understanding-the-risk-of-script-injections) for information on avoiding and mitigating the risk of script injections.
 ecosystem:
   languages:
     - all

probes/hasDangerousWorkflowUntrustedCheckout/def.yml

@@ -15,19 +15,22 @@
 id: hasDangerousWorkflowUntrustedCheckout
 short: Check whether the project has GitHub Actions workflows that does untrusted checkouts.
 motivation: >
-  Untrusted Code Checkout: This is the misuse of potentially dangerous triggers. This checks if a pull_request_target or workflow_run workflow trigger was used in conjunction with an explicit pull request checkout. Workflows triggered with pull_request_target / workflow_run have write permission to the target repository and access to target repository secrets. With the PR checkout, PR authors may compromise the repository, for example, by using build scripts controlled by the author of the PR or reading token in memory. This check does not detect whether untrusted code checkouts are used safely, for example, only on pull request that have been assigned a label.
+  GitHub workflows triggered with pull_request_target or workflow_run have write permission to the target repository and access to target repository secrets.
+  Combined with a dangerous checkout of PR contents, attackers may be able to compromise the repository, for example, by using build scripts controlled by the PR author.
 implementation: >
-  The probe iterates through the workflows from the raw results and checks the workflow type. If it finds a workflow of the type `DangerousWorkflowUntrustedCheckout`, it returns.
+  The probe iterates through the workflows looking for pull_request_target and workflow_run triggers which checkout references from a PR.
+  This check does not detect whether untrusted code checkouts are used safely, for example, only on pull request that have been assigned a label.
 outcome:
-  - If the project has at least one workflow with possibility of untrusted code checkout, the probe returns one finding with OutcomeTrue.
-  - If the project does not have a single workflow with possibility of untrusted code checkout, the probe returns one finding with OutcomeFalse.
+  - The probe returns one finding with OutcomeTrue per untrusted checkout.
+  - The probe returns one finding with OutcomeFalse if no untrusted checkouts are detected.
 remediation:
   onOutcome: True
   effort: Low
   text:
     - Avoid the dangerous workflow patterns.
   markdown:
-    - Avoid the dangerous workflow patterns. See [this post](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/) for information on avoiding untrusted code checkouts. See [this document](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#understanding-the-risk-of-script-injections) for information on avoiding and mitigating the risk of script injections.
+    - Avoid the dangerous workflow patterns.
+    - See [this post](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/) for information on avoiding untrusted code checkouts.
 ecosystem:
   languages:
     - all

probes/hasFSFOrOSIApprovedLicense/def.yml

@@ -13,11 +13,13 @@
 # limitations under the License.
 
 id: hasFSFOrOSIApprovedLicense
-short: Check that the project has an FSF or OSI approved license. For more, see [all FSF or OSI approved license formats](https://spdx.org/licenses/).
+short: Check that the project has an FSF or OSI approved license.
 motivation: >
-  A license can give users information about how the source code may or may not be used. The lack of a license will impede any kind of security review or audit and creates a legal risk for potential users.
+  A license can give users information about how the source code may or may not be used.
+  Using a recognized license facilitates security or legal reviews for potential users.
 implementation: >
- The implementation checks whether a license file is present and is of an approved format
+  The implementation checks whether a license file is present and is of an approved format.
+  The list of FSF or OSI approved license is taken from https://spdx.org/licenses/ .
 outcome:
   - If a license file is found and is of an approved format, the probe returns a single OutcomeTrue.
   - If a license file is missing the probe returns a single OutcomeNotApplicable.