Java: Add query to detect special characters in string literals

java/ql/integration-tests/java/query-suite/java-code-quality-extended.qls.expected

@@ -77,6 +77,7 @@ ql/java/ql/src/Violations of Best Practice/Naming Conventions/ConfusingMethodNam
 ql/java/ql/src/Violations of Best Practice/Naming Conventions/ConfusingOverloading.ql
 ql/java/ql/src/Violations of Best Practice/Naming Conventions/LocalShadowsFieldConfusing.ql
 ql/java/ql/src/Violations of Best Practice/Naming Conventions/SameNameAsSuper.ql
+ql/java/ql/src/Violations of Best Practice/SpecialCharactersInLiterals/NonExplicitControlAndWhitespaceCharsInLiterals.ql
 ql/java/ql/src/Violations of Best Practice/Undesirable Calls/CallsToStringToString.ql
 ql/java/ql/src/Violations of Best Practice/Undesirable Calls/DefaultToString.ql
 ql/java/ql/src/Violations of Best Practice/Undesirable Calls/DoNotCallFinalize.ql

java/ql/src/Violations of Best Practice/SpecialCharactersInLiterals/NonExplicitControlAndWhitespaceCharsInLiterals.md

@@ -0,0 +1,111 @@
+## Overview
+
+This query detects non-explicit control and whitespace characters in Java literals.
+Such characters are often introduced accidentally and can be invisible or hard to recognize, leading to bugs when the actual contents of the string contain control characters.
+
+## Recommendation
+
+To avoid issues, use the encoded versions of control characters (e.g. ASCII `\n`, `\t`, or Unicode `U+000D`, `U+0009`).
+This makes the literals (e.g. string literals) more readable, and also helps to make the surrounding code less error-prone and more maintainable.
+
+## Example
+
+The following examples illustrate good and bad code:
+
+Bad:
+
+```java
+char tabulationChar = '	'; // Non compliant
+String tabulationCharInsideString = "A	B";  // Non compliant
+String fooZeroWidthSpacebar = "foo​bar"; // Non compliant
+```
+
+Good:
+
+```java
+char escapedTabulationChar = '\t';
+String escapedTabulationCharInsideString = "A\tB";  // Compliant
+String fooUnicodeSpacebar = "foo\u0020bar";  // Compliant
+String foo2Spacebar = "foo  bar";  // Compliant
+String foo3Spacebar = "foo   bar";  // Compliant
+```
+
+## Implementation notes
+
+This query detects Java literals that contain reserved control characters and/or non-printable whitespace characters, such as:
+
+- Decimal and hexidecimal representations of ASCII control characters (code points 0-8, 11, 14-31, and 127).
+- Invisible characters (e.g. zero-width space, zero-width joiner).
+- Unicode C0 control codes, plus the delete character (U+007F), such as:
+
+  | Escaped Unicode | ASCII Decimal | Description               |
+  | --------------- | ------------- | ------------------------- |
+  | `\u0000`        | 0             | null character            |
+  | `\u0001`        | 1             | start of heading          |
+  | `\u0002`        | 2             | start of text             |
+  | `\u0003`        | 3             | end of text               |
+  | `\u0004`        | 4             | end of transmission       |
+  | `\u0005`        | 5             | enquiry                   |
+  | `\u0006`        | 6             | acknowledge               |
+  | `\u0007`        | 7             | bell                      |
+  | `\u0008`        | 8             | backspace                 |
+  | `\u000B`        | 11            | vertical tab              |
+  | `\u000E`        | 14            | shift out                 |
+  | `\u000F`        | 15            | shift in                  |
+  | `\u0010`        | 16            | data link escape          |
+  | `\u0011`        | 17            | device control 1          |
+  | `\u0012`        | 18            | device control 2          |
+  | `\u0013`        | 19            | device control 3          |
+  | `\u0014`        | 20            | device control 4          |
+  | `\u0015`        | 21            | negative acknowledge      |
+  | `\u0016`        | 22            | synchronous idle          |
+  | `\u0017`        | 23            | end of transmission block |
+  | `\u0018`        | 24            | cancel                    |
+  | `\u0019`        | 25            | end of medium             |
+  | `\u001A`        | 26            | substitute                |
+  | `\u001B`        | 27            | escape                    |
+  | `\u001C`        | 28            | file separator            |
+  | `\u001D`        | 29            | group separator           |
+  | `\u001E`        | 30            | record separator          |
+  | `\u001F`        | 31            | unit separator            |
+  | `\u007F`        | 127           | delete                    |
+
+- Zero-width Unicode characters (e.g. zero-width space, zero-width joiner), such as:
+
+  | Escaped Unicode | Description               |
+  | --------------- | ------------------------- |
+  | `\u200B`        | zero-width space          |
+  | `\u200C`        | zero-width non-joiner     |
+  | `\u200D`        | zero-width joiner         |
+  | `\u2028`        | line separator            |
+  | `\u2029`        | paragraph separator       |
+  | `\u2060`        | word joiner               |
+  | `\uFEFF`        | zero-width no-break space |
+
+The following list outlines the _**explicit exclusions from query scope**_:
+
+- any number of simple space characters (`U+0020`, ASCII 32).
+- an escape character sequence (e.g. `\t`), or the Unicode equivalent (e.g. `\u0009`), for printable whitespace characters:
+
+  | Character Sequence | Escaped Unicode | ASCII Decimal | Description     |
+  | ------------------ | --------------- | ------------- | --------------- |
+  | `\t`               | \u0009          | 9             | horizontal tab  |
+  | `\n`               | \u000A          | 10            | line feed       |
+  | `\f`               | \u000C          | 12            | form feed       |
+  | `\r`               | \u000D          | 13            | carriage return |
+  |                    | \u0020          | 32            | space           |
+
+- character literals (i.e. single quotes) containing control characters.
+- literals defined within "likely" test methods, such as:
+  - JUnit test methods
+  - methods annotated with `@Test`
+  - methods of a class annotated with `@Test`
+  - methods with names containing "test"
+
+## References
+
+- Unicode: [Unicode Control Characters](https://www.unicode.org/charts/PDF/U0000.pdf).
+- Wikipedia: [Unicode C0 control codes](https://en.wikipedia.org/wiki/C0_and_C1_control_codes).
+- Wikipedia: [Unicode characters with property "WSpace=yes" or "White_Space=yes"](https://en.wikipedia.org/wiki/Unicode_character_property#Whitespace).
+- Java API Specification: [Java String Literals](https://docs.oracle.com/javase/tutorial/java/data/characters.html).
+- Java API Specification: [Java Class Charset](https://docs.oracle.com/javase/8/docs/api///?java/nio/charset/Charset.html).

java/ql/src/Violations of Best Practice/SpecialCharactersInLiterals/NonExplicitControlAndWhitespaceCharsInLiterals.ql

@@ -0,0 +1,51 @@
+/**
+ * @id java/non-explicit-control-and-whitespace-chars-in-literals
+ * @name Non-explicit control and whitespace characters
+ * @description Non-explicit control and whitespace characters in literals make code more difficult
+ *              to read and may lead to incorrect program behavior.
+ * @kind problem
+ * @precision medium
+ * @problem.severity warning
+ * @tags quality
+ *       correctness
+ *       maintainability
+ *       readability
+ */
+
+import java
+
+/**
+ * A `Literal` that has a Unicode control character within its
+ * literal value (as returned by `getLiteral()` member predicate).
+ */
+class ReservedUnicodeInLiteral extends Literal {
+  private int indexStart;
+
+  ReservedUnicodeInLiteral() {
+    not this instanceof CharacterLiteral and
+    exists(int codePoint |
+      this.getLiteral().codePointAt(indexStart) = codePoint and
+      (
+        // Unicode C0 control characters
+        codePoint < 32 and not codePoint in [9, 10, 12, 13]
+        or
+        codePoint = 127 // delete control character
+        or
+        codePoint = 8203 // zero-width space
+      )
+    )
+  }
+
+  /** Gets the starting index of the Unicode control sequence. */
+  int getIndexStart() { result = indexStart }
+}
+
+from ReservedUnicodeInLiteral literal, int charIndex, int codePoint
+where
+  literal.getIndexStart() = charIndex and
+  literal.getLiteral().codePointAt(charIndex) = codePoint and
+  not literal.getEnclosingCallable() instanceof LikelyTestMethod and
+  not literal.getFile().isKotlinSourceFile()
+select literal,
+  "Literal value contains control or non-printable whitespace character(s) starting with Unicode code point "
+    + codePoint + " at index " + charIndex + "."

java/ql/test/query-tests/NonExplicitControlAndWhitespaceCharsInLiterals/CharTest.java

@@ -0,0 +1,184 @@
+import java.util.List;
+import java.util.ArrayList;
+
+public class CharTest {
+
+    public static void main(String[] args) {
+        CharTest charTest = new CharTest();
+        NonCompliantStringLiterals nonCompliant = charTest.new NonCompliantStringLiterals();
+        CompliantStringLiterals compliant = charTest.new CompliantStringLiterals();
+        CompliantCharLiterals compliantChar = charTest.new CompliantCharLiterals();
+
+        List<String> nonCompliantStrings = nonCompliant.getNonCompliantStrings();
+        List<String> compliantStrings = compliant.getCompliantStrings();
+        List<Character> compliantChars = compliantChar.getCompliantChars();
+
+        System.out.println("");
+        System.out.println("Non-compliant strings:");
+        for (String s : nonCompliantStrings) {
+            System.out.println(s);
+            System.out.println("");
+        }
+
+        System.out.println("");
+        System.out.println("Compliant strings:");
+        for (String s : compliantStrings) {
+            System.out.println(s);
+            System.out.println("");
+        }
+        System.out.println("");
+
+        System.out.println("");
+        System.out.println("Compliant character literals:");
+        System.out.println("");
+        for (Character c : compliantChars) {
+            System.out.println("\\u" + String.format("%04X", (int) c));
+        }
+        System.out.println("");
+    }
+
+    class CompliantCharLiterals {
+        private List<Character> compliantChars;
+
+        public CompliantCharLiterals() {
+            compliantChars = new ArrayList<>();
+            compliantChars.add('A'); // COMPLIANT
+            compliantChars.add('a'); // COMPLIANT
+            compliantChars.add('\b'); // COMPLIANT
+            compliantChars.add('\t'); // COMPLIANT
+            compliantChars.add('\n'); // COMPLIANT
+            compliantChars.add('\f'); // COMPLIANT
+            compliantChars.add('\r'); // COMPLIANT
+            compliantChars.add('\u0000'); // COMPLIANT
+            compliantChars.add('\u0007'); // COMPLIANT
+            compliantChars.add('\u001B'); // COMPLIANT
+            compliantChars.add(' '); // COMPLIANT
+            compliantChars.add('\u0020'); // COMPLIANT
+            compliantChars.add('\u200B'); // COMPLIANT
+            compliantChars.add('\u200C'); // COMPLIANT
+            compliantChars.add('\u200D'); // COMPLIANT
+            compliantChars.add('\u2028'); // COMPLIANT
+            compliantChars.add('\u2029'); // COMPLIANT
+            compliantChars.add('\u2060'); // COMPLIANT
+            compliantChars.add('\uFEFF'); // COMPLIANT
+        }
+
+        public List<Character> getCompliantChars() {
+            return compliantChars;
+        }
+    }
+
+    class CompliantStringLiterals {
+        private List<String> compliantStrings;
+
+        public CompliantStringLiterals() {
+            compliantStrings = new ArrayList<>();
+            compliantStrings.add(""); // COMPLIANT
+            compliantStrings.add("X__Y"); // COMPLIANT
+            compliantStrings.add("X_ _Y"); // COMPLIANT
+            compliantStrings.add("X_\u0020_Y"); // COMPLIANT
+            compliantStrings.add("X_  _Y"); // COMPLIANT
+            compliantStrings.add("X_\u0020\u0020_Y"); // COMPLIANT
+            compliantStrings.add("X_   _Y"); // COMPLIANT
+            compliantStrings.add("X_    _Y"); // COMPLIANT
+            compliantStrings.add("X_     _Y"); // COMPLIANT
+            compliantStrings.add("X_      _Y"); // COMPLIANT
+            compliantStrings.add("X_\b_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0000_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0001_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0002_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0003_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0004_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0005_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0006_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0007_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0008_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0009_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0010_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0011_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0012_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0013_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0014_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0015_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0016_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0017_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0018_Y"); // COMPLIANT
+            compliantStrings.add("X_\u0019_Y"); // COMPLIANT
+            compliantStrings.add("X_\u001A_Y"); // COMPLIANT
+            compliantStrings.add("X_\u001B_Y"); // COMPLIANT
+            compliantStrings.add("X_\u001C_Y"); // COMPLIANT
+            compliantStrings.add("X_\u001D_Y"); // COMPLIANT
+            compliantStrings.add("X_\u001E_Y"); // COMPLIANT
+            compliantStrings.add("X_\u001F_Y"); // COMPLIANT
+            compliantStrings.add("X_\u007F_Y"); // COMPLIANT
+            compliantStrings.add("X_\u200B_Y"); // COMPLIANT
+            compliantStrings.add("X_\u200C_Y"); // COMPLIANT
+            compliantStrings.add("X_\u200D_Y"); // COMPLIANT
+            compliantStrings.add("X_\u2028_Y"); // COMPLIANT
+            compliantStrings.add("X_\u2029_Y"); // COMPLIANT
+            compliantStrings.add("X_\u2060_Y"); // COMPLIANT
+            compliantStrings.add("X_\uFEFF_Y"); // COMPLIANT
+            compliantStrings.add("X_\uFEFF_Y_\u0020_Z"); // COMPLIANT
+            compliantStrings.add("X_\uFEFF_Y_\uFEFF_Z"); // COMPLIANT
+            compliantStrings.add("X_\u0020_Y_\uFEFF_Z"); // COMPLIANT
+            compliantStrings.add("X_\t_Y"); // COMPLIANT
+            compliantStrings.add("X_\t\t_Y"); // COMPLIANT
+            compliantStrings.add("X_\\b_Y"); // COMPLIANT
+            compliantStrings.add("X_\f_Y"); // COMPLIANT
+            compliantStrings.add("X_\\f_Y"); // COMPLIANT
+            compliantStrings.add("X_\n_Y"); // COMPLIANT
+            compliantStrings.add("X_\n\t_Y"); // COMPLIANT
+            compliantStrings.add("X_\\n_Y"); // COMPLIANT
+            compliantStrings.add("X_\r_Y"); // COMPLIANT
+            compliantStrings.add("X_\\r_Y"); // COMPLIANT
+            compliantStrings.add("X_\t_Y"); // COMPLIANT
+            compliantStrings.add("X_\\t_Y"); // COMPLIANT
+            compliantStrings.add("X_\\u0000_Y"); // COMPLIANT
+            compliantStrings.add("X_\\u0007_Y"); // COMPLIANT
+            compliantStrings.add("X_\\u001B_Y"); // COMPLIANT
+            compliantStrings.add("X_\\u200B_Y"); // COMPLIANT
+            compliantStrings.add("X_\\u200C_Y"); // COMPLIANT
+            compliantStrings.add("X_\\u200D_Y"); // COMPLIANT
+            compliantStrings.add("X_\\u2028_Y"); // COMPLIANT
+            compliantStrings.add("X_\\u2029_Y"); // COMPLIANT
+            compliantStrings.add("X_\\u2060_Y"); // COMPLIANT
+            compliantStrings.add("X_\\uFEFF_Y"); // COMPLIANT
+            compliantStrings.add("lorem ipsum dolor "+"sit amet"); // COMPLIANT
+            compliantStrings.add("lorem ipsum dolor " + "sit amet"); // COMPLIANT
+            compliantStrings.add("lorem ipsum dolor sit amet, consectetur adipiscing elit, " + // COMPLIANT
+            "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.");
+            compliantStrings.add("lorem ipsum dolor sit amet, consectetur adipiscing elit, " + // COMPLIANT
+            "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad "
+            + "minim veniam, quis nostrud exercitation ullamco "+"laboris nisi ut aliquip ex " +
+            "ea commodo consequat.");
+            compliantStrings.add("""
+                lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
+                Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
+            """); // COMPLIANT
+        }
+
+        public List<String> getCompliantStrings() {
+            return compliantStrings;
+        }
+    }
+
+    class NonCompliantStringLiterals {
+        private List<String> nonCompliantStrings;
+
+        public NonCompliantStringLiterals() {
+            nonCompliantStrings = new ArrayList<>();
+            nonCompliantStrings.add("X_​_Y"); // NON_COMPLIANT
+            nonCompliantStrings.add("X_​_Y_​_Z"); // NON_COMPLIANT
+            nonCompliantStrings.add("lorem​ipsum dolor sit amet,​consectetur adipiscing elit, " + // NON_COMPLIANT
+            "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.");
+            nonCompliantStrings.add("""
+                lorem​ipsum dolor sit amet,​consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
+                Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
+            """); // NON_COMPLIANT
+        }
+
+        public List<String> getNonCompliantStrings() {
+            return nonCompliantStrings;
+        }
+    }
+}

java/ql/test/query-tests/NonExplicitControlAndWhitespaceCharsInLiterals/NonExplicitControlAndWhitespaceCharsInLiterals.expected

@@ -0,0 +1,7 @@
+| CharTest.java:170:37:170:43 | "X_\u200b_Y" | Literal value contains control or non-printable whitespace character(s) starting with Unicode code point 8203 at index 3. |
+| CharTest.java:171:37:171:47 | "X_\u200b_Y_\u200b_Z" | Literal value contains control or non-printable whitespace character(s) starting with Unicode code point 8203 at index 3. |
+| CharTest.java:171:37:171:47 | "X_\u200b_Y_\u200b_Z" | Literal value contains control or non-printable whitespace character(s) starting with Unicode code point 8203 at index 7. |
+| CharTest.java:172:37:173:80 | "lorem\u200bipsum dolor sit amet,\u200bconsectetur adipiscing elit, " + // NON_COMPLIANT\n            "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua." | Literal value contains control or non-printable whitespace character(s) starting with Unicode code point 8203 at index 6. |
+| CharTest.java:172:37:173:80 | "lorem\u200bipsum dolor sit amet,\u200bconsectetur adipiscing elit, " + // NON_COMPLIANT\n            "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua." | Literal value contains control or non-printable whitespace character(s) starting with Unicode code point 8203 at index 28. |
+| CharTest.java:174:37:177:15 | """\n                lorem\u200bipsum dolor sit amet,\u200bconsectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.\n                Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.\n            """ | Literal value contains control or non-printable whitespace character(s) starting with Unicode code point 8203 at index 25. |
+| CharTest.java:174:37:177:15 | """\n                lorem\u200bipsum dolor sit amet,\u200bconsectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.\n                Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.\n            """ | Literal value contains control or non-printable whitespace character(s) starting with Unicode code point 8203 at index 47. |

java/ql/test/query-tests/NonExplicitControlAndWhitespaceCharsInLiterals/NonExplicitControlAndWhitespaceCharsInLiterals.qlref

@@ -0,0 +1 @@
+Violations of Best Practice/SpecialCharactersInLiterals/NonExplicitControlAndWhitespaceCharsInLiterals.ql

java/ql/test/query-tests/NonExplicitControlAndWhitespaceCharsInLiterals/options

@@ -0,0 +1 @@
+semmle-extractor-options: --javac-args -source 15 -target 15