📖 Improve rationale for Binary-Artifacts (#1016)

* Improve rationale for Binary-Artifacts

I'm fine with prohibiting binary executables, but
the *rationale* for doing this was completely unclear.

This commit rewrites the rationale to explain, in hopefully
a better way, why they can be a problem.

I prefer "executable" over "binary".
On digital computers, all data (including source code) are binaries :-).
In addition, some executables are simultaneously executables
and source code, e.g., shell scripts.
So I think what is meant here is a "generated binary".

I don't really think this merits a "High" level, but that's
a different dicussion.

Signed-off-by: David A. Wheeler <dwheeler@dwheeler.com>

* Tweak Binary-Artifacts rationale

Tweak Binary-Artifacts text based on comments from
@naveensrinivasan.

Signed-off-by: David A. Wheeler <dwheeler@dwheeler.com>

Co-authored-by: Naveen <172697+naveensrinivasan@users.noreply.github.com>

docs/checks.md

@@ -11,11 +11,14 @@ If you have ideas for things to add, or new ways to detect things,
 please contribute!
 ## Binary-Artifacts 
 
-This check tries to determine if the project has binary artifacts in the source repository.
-Binaries are a threat to auditability and vulnerability management. In addition, a binary could be compromised or malicious. A low score is therefore considered `High` risk. 
+This check tries to determine if the project has generated executable (binary) artifacts in the source repository.
+Including generated executables in the source repository increases user risks. Many programming language systems can generate executables from source code (e.g., C/C++ generated machine code, Java `.class` files, Python `.pyc` files, and minified JavaScript). Users will often directly use executables if they are included in the source repository, leading to many dangerous behaviors. One problem is that reviews become ineffective and thus enable the use of obsolete or maliciously-subverted executables. Reviews generally review source code, not executables, since it's difficult to audit executables to ensure that they correspond. Since it's hard to audit this, over time the included executables might not correspond to the source code. Another problem is that it allows the executable generation process to atrophy, which can lead to an inability to create working executables.  These problems can be countered with verified reproducible builds, but it's easier to implement verified reproducible builds when executables are not included in the source repository (since the executable generation process is less likely to have atrophied).
+It's fine to include, in the source repository, files that are simultaneously reviewable source code and executables, since these are reviewable. (Some interpretive systems, such as many operating system shells, don't have a mechanism for storing generated executables that are different from the source file.) Similarly, we allow including in the source repository source code generated by other tools (e.g., by bison, yacc, flex, and lex). There are potential downsides to generated source code, but generated source code tends to be much easier to review and thus presents a lower risk. Generated source code is also often difficult for external tools to detect.
+We allow including generated documentation in source repositories. Generated documentation is intended for use by humans (not computers), and humans can take context into account. Thus, generated documentation doesn't pose the same level of risk.
+A low score is therefore considered `High` risk. 
 
 **Remediation steps**
-- Remove the binary artifacts from the repository.
+- Remove the generated executable artifacts from the repository.
 - Build from source.
 
 ## Branch-Protection 

docs/checks/internal/checks.yaml

@@ -66,16 +66,54 @@ checks:
   Binary-Artifacts:
     risk: High
     tags: supply-chain, security, dependencies
-    short: Determines if the project has binary artifacts in the source repository.
+    short: Determines if the project has generated executable (binary) artifacts in the source repository.
     description: >-
-      This check tries to determine if the project has binary artifacts in the source repository.
+      This check tries to determine if the project has generated executable (binary) artifacts in the source repository.
+
+      Including generated executables in the source repository
+      increases user risks.
+      Many programming language systems can generate
+      executables from source code (e.g., C/C++ generated machine code,
+      Java `.class` files, Python `.pyc` files, and minified JavaScript).
+      Users will often directly use executables if they are included in the
+      source repository, leading to many dangerous behaviors.
+      One problem is that reviews become ineffective and thus enable the
+      use of obsolete or maliciously-subverted executables.
+      Reviews generally review source code, not executables, since
+      it's difficult to audit executables to ensure that they correspond.
+      Since it's hard to audit this, over time
+      the included executables might not correspond to the source code.
+      Another problem is that it allows the executable generation process to
+      atrophy, which can lead to an inability to create working
+      executables.  These problems can be countered with verified
+      reproducible builds, but it's easier to implement verified
+      reproducible builds when executables are not included in the
+      source repository (since the executable generation process is less
+      likely to have atrophied).
+
+      It's fine to include, in the source repository,
+      files that are simultaneously reviewable source code and executables,
+      since these are reviewable.
+      (Some interpretive systems, such as many operating system shells,
+      don't have a mechanism for storing generated executables that are
+      different from the source file.)
+      Similarly, we allow including in the source repository
+      source code generated by other tools
+      (e.g., by bison, yacc, flex, and lex).
+      There are potential downsides to generated source code, but
+      generated source code tends to be much easier to review and thus
+      presents a lower risk. Generated source code is also often difficult
+      for external tools to detect.
+
+      We allow including generated documentation in source repositories.
+      Generated documentation is intended for use by humans
+      (not computers), and humans can take context into account.
+      Thus, generated documentation doesn't pose the same level of risk.
 
-      Binaries are a threat to auditability and vulnerability management.
-      In addition, a binary could be compromised or malicious.
       A low score is therefore considered `High` risk.
     remediation:
       - >-
-        Remove the binary artifacts from the repository.
+        Remove the generated executable artifacts from the repository.
       - >-
         Build from source.
   Branch-Protection: