From 6c28be98274e081a1bfa524bc1380420d0193fda Mon Sep 17 00:00:00 2001 From: Jon Janego Date: Thu, 13 Mar 2025 11:49:48 -0500 Subject: [PATCH 1/5] Update query-metadata-style-guide.md initial commit of changes starting to add quality tagging standards --- docs/query-metadata-style-guide.md | 56 +++++++++++++++++++++--------- 1 file changed, 39 insertions(+), 17 deletions(-) diff --git a/docs/query-metadata-style-guide.md b/docs/query-metadata-style-guide.md index 3350111db22e..d4704c6c11ed 100644 --- a/docs/query-metadata-style-guide.md +++ b/docs/query-metadata-style-guide.md @@ -121,12 +121,47 @@ Alert queries (`@kind problem` or `path-problem`) support two further properties ## Query tags `@tags` -The `@tags` property is used to define categories that the query relates to. Each alert query should belong to one (or more, if necessary) of the following four top-level categories: +The `@tags` property is used to define categories that the query relates to. As we prepare for the release of a dedicated code quality product, we will use tagging to prepare a stronger delineation between queries that are part of our existing "code security" product offering and those that are better suited as part of a "quality" product offering. Each alert query should belong to one of the following two top-level categories, with additional sub-categories: -* `@tags correctness`–for queries that detect incorrect program behavior. -* `@tags maintainability`–for queries that detect patterns that make it harder for developers to make changes to the code. -* `@tags readability`–for queries that detect confusing patterns that make it harder for developers to read the code. +#### High level category `@tags` * `@tags security`–for queries that detect security weaknesses. See below for further information. +* `@tags quality`–for queries that detect code quality issues. See below for further information. + +#### Security query `@tags` + +If your query is a security query, use one or more `@tags` to associate it with the relevant CWEs. Add `@tags` for the most specific Base Weakness or Class Weakness in [View 1000](https://cwe.mitre.org/data/definitions/1000.html), using the parent/child relationship. For example: + +| `@tags security` | `external/cwe/cwe-022`| +|-|-| +||`external/cwe/cwe-023` | +||`external/cwe/cwe-036` | +||`external/cwe/cwe-073` | + +When you tag a query like this, the associated CWE pages from [MITRE.org](https://cwe.mitre.org/index.html) will automatically appear in the references section of its associated qhelp file. + +> [!NOTE] +> The automatic addition of CWE reference links works only if the qhelp file already contains a `` section. + +#### Quality query sub-category `@tags` + +* `@tags maintainability`–for queries that detect patterns that make it harder for developers to make changes to the code. +* `@tags reliability`–for queries that detect issues that affect whether the code will perform as expected during execution. + +Software quality doesn't have as universally-agreed categorization method as security issues like CWE, so we will do our own categorization instead of using tags like CWE. + +We'll have two "top-level" categories of quality queries, with sub-categories beneath: + +* `@tags maintainability`–for queries that detect patterns that make it harder for developers to make changes to the code. + * `@tags readability`–for queries that detect confusing patterns that make it harder for developers to read the code. + * + + +* `@tags reliability`–for queries that detect issues that affect whether the code will perform as expected during execution. + * `@tags correctness`–for queries that detect incorrect program behavior. + + + + There are also more specific `@tags` that can be added. See, the following pages for examples of the low-level tags: @@ -145,20 +180,7 @@ If necessary, you can also define your own low-level tags to categorize the quer * Use a forward slash / to indicate a hierarchical relationship between tags if necessary. For example, a query with tag `foo/bar` is also interpreted as also having tag `foo`, but not `bar`. * Use a single-word `@tags` name. Multiple words, separated with hyphens, can be used for clarity if necessary. -#### Security query `@tags` - -If your query is a security query, use one or more `@tags` to associate it with the relevant CWEs. Add `@tags` for the most specific Base Weakness or Class Weakness in [View 1000](https://cwe.mitre.org/data/definitions/1000.html), using the parent/child relationship. For example: -| `@tags security` | `external/cwe/cwe-022`| -|-|-| -||`external/cwe/cwe-023` | -||`external/cwe/cwe-036` | -||`external/cwe/cwe-073` | - -When you tag a query like this, the associated CWE pages from [MITRE.org](https://cwe.mitre.org/index.html) will automatically appear in the references section of its associated qhelp file. - -> [!NOTE] -> The automatic addition of CWE reference links works only if the qhelp file already contains a `` section. #### Metric/summary `@tags` From de5d3b6263b25657f6524306ce66b20beaa9a963 Mon Sep 17 00:00:00 2001 From: Jon Janego Date: Thu, 13 Mar 2025 12:42:51 -0500 Subject: [PATCH 2/5] Update query-metadata-style-guide.md --- docs/query-metadata-style-guide.md | 40 +++++++++++++++++------------- 1 file changed, 23 insertions(+), 17 deletions(-) diff --git a/docs/query-metadata-style-guide.md b/docs/query-metadata-style-guide.md index d4704c6c11ed..a13ad200a9a3 100644 --- a/docs/query-metadata-style-guide.md +++ b/docs/query-metadata-style-guide.md @@ -121,9 +121,9 @@ Alert queries (`@kind problem` or `path-problem`) support two further properties ## Query tags `@tags` -The `@tags` property is used to define categories that the query relates to. As we prepare for the release of a dedicated code quality product, we will use tagging to prepare a stronger delineation between queries that are part of our existing "code security" product offering and those that are better suited as part of a "quality" product offering. Each alert query should belong to one of the following two top-level categories, with additional sub-categories: +The `@tags` property is used to define the high level category of problem that the query relates to. Each alert query should belong to one of the following two top-level categories, with additional sub-categories: -#### High level category `@tags` +### High level category `@tags` * `@tags security`–for queries that detect security weaknesses. See below for further information. * `@tags quality`–for queries that detect code quality issues. See below for further information. @@ -149,18 +149,19 @@ When you tag a query like this, the associated CWE pages from [MITRE.org](https: Software quality doesn't have as universally-agreed categorization method as security issues like CWE, so we will do our own categorization instead of using tags like CWE. -We'll have two "top-level" categories of quality queries, with sub-categories beneath: +We'll use two "top-level" categories of quality queries, with sub-categories beneath: * `@tags maintainability`–for queries that detect patterns that make it harder for developers to make changes to the code. * `@tags readability`–for queries that detect confusing patterns that make it harder for developers to read the code. - * - + * `@tags unused-code`-for queries that detect functions that are never used and other instances of unused code + * `@tags complexity`-for queries that detect patterns in the code that lead to unnecesary complexity such as unclear control flow, or high cyclomatic complexity + * `@tags reliability`–for queries that detect issues that affect whether the code will perform as expected during execution. - * `@tags correctness`–for queries that detect incorrect program behavior. - - - + * `@tags correctness`–for queries that detect incorrect program behavior or couse result in unintended outcomes. + * `@tags performance`-for queries that detect code that could impact performance through inefficient algorithms, unnecessary computation, etc + * `@tags concurrency`-for queries that detect concurrency related issues such as race conditions, deadlocks, thread safety, etc + * `@tags error-handling`-for queries that detect issues related to unsafe error handling such as uncaught exceptions, etc There are also more specific `@tags` that can be added. See, the following pages for examples of the low-level tags: @@ -172,22 +173,27 @@ There are also more specific `@tags` that can be added. See, the following pages * [JavaScript queries](https://codeql.github.com/codeql-query-help/javascript/) * [Python queries](https://codeql.github.com/codeql-query-help/python/) -Metric queries (`@kind metric`) may have the `summary` tag. If SARIF output is used, the results of these queries can be found at `run[].properties.metricResults`. -If necessary, you can also define your own low-level tags to categorize the queries specific to your project or organization. When creating your own tags, you should: - -* Use all lower-case letters, including for acronyms and proper nouns, with no spaces. All characters apart from * and @ are accepted. -* Use a forward slash / to indicate a hierarchical relationship between tags if necessary. For example, a query with tag `foo/bar` is also interpreted as also having tag `foo`, but not `bar`. -* Use a single-word `@tags` name. Multiple words, separated with hyphens, can be used for clarity if necessary. +### Severities +Maintainers are expected to add a `@security-severity` tag to security relevant queries that will be run on Code Scanning. There is a documented internal process for generating these `@security-severity` values. +TODO: should we have a severity value for quality queries? -#### Metric/summary `@tags` +### Metric/summary `@tags` Code Scanning may use tags to identify queries with specific meanings across languages. Currently, there is only one such tag: `lines-of-code`. The sum of the results for queries with this tag that return a single number column ([example for JavaScript](https://github.com/github/codeql/blob/c47d680d65f09a851e41d4edad58ffa7486b5431/java/ql/src/Metrics/Summaries/LinesOfCode.ql)) is interpreted by Code Scanning as the lines of code under the source root present in the database. Each language should have exactly one query of this form. +Metric queries (`@kind metric`) may have the `summary` tag. If SARIF output is used, the results of these queries can be found at `run[].properties.metricResults`. -Maintainers are expected to add a `@security-severity` tag to security relevant queries that will be run on Code Scanning. There is a documented internal process for generating these `@security-severity` values. + +### Customizing tags + +If necessary, you can also define your own low-level tags to categorize the queries specific to your project or organization, but if possible, please try to follow the above standards (or propose changes to this style guide). When creating your own tags, you should: + +* Use all lower-case letters, including for acronyms and proper nouns, with no spaces. All characters apart from * and @ are accepted. +* Use a forward slash / to indicate a hierarchical relationship between tags if necessary. For example, a query with tag `foo/bar` is also interpreted as also having tag `foo`, but not `bar`. +* Use a single-word `@tags` name. Multiple words, separated with hyphens, can be used for clarity if necessary. ## QL area From 14d178f81773c21c62d428af41607f45b5dd5882 Mon Sep 17 00:00:00 2001 From: Jon Janego Date: Wed, 19 Mar 2025 16:56:37 -0500 Subject: [PATCH 3/5] Update query-metadata-style-guide.md minor tag changes to align with existing tags --- docs/query-metadata-style-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/query-metadata-style-guide.md b/docs/query-metadata-style-guide.md index a13ad200a9a3..b51f6fea5a0a 100644 --- a/docs/query-metadata-style-guide.md +++ b/docs/query-metadata-style-guide.md @@ -153,7 +153,7 @@ We'll use two "top-level" categories of quality queries, with sub-categories ben * `@tags maintainability`–for queries that detect patterns that make it harder for developers to make changes to the code. * `@tags readability`–for queries that detect confusing patterns that make it harder for developers to read the code. - * `@tags unused-code`-for queries that detect functions that are never used and other instances of unused code + * `@tags useless-code`-for queries that detect functions that are never used and other instances of unused code * `@tags complexity`-for queries that detect patterns in the code that lead to unnecesary complexity such as unclear control flow, or high cyclomatic complexity From f698d0a060b01adec18a57c2e9bfe1d952bf76e0 Mon Sep 17 00:00:00 2001 From: Jon Janego Date: Wed, 19 Mar 2025 16:58:30 -0500 Subject: [PATCH 4/5] Update query-metadata-style-guide.md --- docs/query-metadata-style-guide.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/query-metadata-style-guide.md b/docs/query-metadata-style-guide.md index b51f6fea5a0a..4c3f5b8814fe 100644 --- a/docs/query-metadata-style-guide.md +++ b/docs/query-metadata-style-guide.md @@ -121,7 +121,7 @@ Alert queries (`@kind problem` or `path-problem`) support two further properties ## Query tags `@tags` -The `@tags` property is used to define the high level category of problem that the query relates to. Each alert query should belong to one of the following two top-level categories, with additional sub-categories: +The `@tags` property is used to define the high level category of problem that the query relates to. Each alert query should belong to one of the following two top-level categories, with additional sub-categories: ### High level category `@tags` * `@tags security`–for queries that detect security weaknesses. See below for further information. @@ -173,6 +173,8 @@ There are also more specific `@tags` that can be added. See, the following pages * [JavaScript queries](https://codeql.github.com/codeql-query-help/javascript/) * [Python queries](https://codeql.github.com/codeql-query-help/python/) +> [!NOTE] +> There is a limit of 10 tags per query! ### Severities From 00bb9056c2a36563f85a1f7c154657a289ddc3b0 Mon Sep 17 00:00:00 2001 From: Jon Janego Date: Tue, 1 Apr 2025 15:42:36 -0500 Subject: [PATCH 5/5] Update query-metadata-style-guide.md --- docs/query-metadata-style-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/query-metadata-style-guide.md b/docs/query-metadata-style-guide.md index 4c3f5b8814fe..e45d88ffab56 100644 --- a/docs/query-metadata-style-guide.md +++ b/docs/query-metadata-style-guide.md @@ -180,7 +180,7 @@ There are also more specific `@tags` that can be added. See, the following pages Maintainers are expected to add a `@security-severity` tag to security relevant queries that will be run on Code Scanning. There is a documented internal process for generating these `@security-severity` values. -TODO: should we have a severity value for quality queries? +We will use the `problem.severity` attribute to handle the severity for quality-related queries. ### Metric/summary `@tags`