C++: Initial telemetry queries #17892

calumgrant · 2024-11-01T16:44:22Z

This adds new internal telemetry queries, so should not need a changenote.

It loosely follows the examples in https://github.com/github/codeql/tree/main/java/ql/src/Telemetry and https://github.com/github/codeql/tree/main/csharp/ql/src/Telemetry.

As a new idea, Metrics.qll uses QL classes a bit more heavily, instead of binary predicates, but obviously they boil down to the same thing.

We don't (yet) emit a diagnostic warning about low quality databases.

Note that qltest can't test for missing includes, since these lead to a catastrophic error by default.

Fixes https://github.com/github/codeql-c-team/issues/2471

Pull Request checklist

All query authors

~~[ ] A change note is added if necessary. See the documentation in this repository.~~
~~[ ] All new queries have appropriate .qhelp. See the documentation in this repository.~~
QL tests are added if necessary. See Testing custom queries in the GitHub documentation.
New and changed queries have correct query metadata. See the documentation in this repository.

Internal query authors only

~~[ ] Autofixes generated based on these changes are valid, only needed if this PR makes significant changes to .ql, .qll, or .qhelp files. See the documentation (internal access required).~~
Changes are validated at scale (internal access required).
~~[ ] Adding a new query? Consider also adding the query to autofix.~~

jketema

Is there are specific reason we're not re-using some of the functionality already present in cpp/ql/src/Diagnostics?

cpp/ql/test/library-tests/extraction_errors/CompilerErrors.expected

calumgrant · 2024-11-04T10:18:52Z

Is there are specific reason we're not re-using some of the functionality already present in cpp/ql/src/Diagnostics?

The code in cpp/ql/src/Diagnostics isn't really that helpful, mainly because it's in the wrong format and has @kind diagnostic.

jketema · 2024-11-04T10:41:30Z

Is there are specific reason we're not re-using some of the functionality already present in cpp/ql/src/Diagnostics?

The code in cpp/ql/src/Diagnostics isn't really that helpful, mainly because it's in the wrong format and has @kind diagnostic.

Not even the libraries in that directory are of use?

jketema

Some questions about the used tags.

cpp/ql/src/Telemetry/Diagnostics.qll

calumgrant · 2024-11-04T11:18:38Z

Is there are specific reason we're not re-using some of the functionality already present in cpp/ql/src/Diagnostics?

The code in cpp/ql/src/Diagnostics isn't really that helpful, mainly because it's in the wrong format and has @kind diagnostic.

Not even the libraries in that directory are of use?

Well, telemetry is a bit simpler because it's a question of dumping counts and raw data, so we don't need to refine them in the same way.

cpp/ql/src/Telemetry/MissingIncludes.ql

cpp/ql/src/Telemetry/CompilerErrors.ql

jketema · 2024-11-11T10:53:20Z

cpp/ql/src/Telemetry/Diagnostics.qll

+  CannotOpenFile() { this.hasTag(["cannot_open_file", "cannot_open_file_reason"]) }
+
+  string getIncludedFile() {
+    result = this.getMessage().regexpCapture("cannot open source file '([^']+)'", 1)


Note that this regex does not what you want in the case of cannot_open_file_reason, i.e., you get the reason as part of the result string.

Yea, I've no real way of testing this as I've never seen the extractor produce this error message. I suppose a .* at the end of the regex might be reasonable?

Are the file names always quoted? This is non-obvious from the frontend source code.

Yes, I tested this on some real databases. But an integration test is clearly needed then. Qltest cannot test this situation due to the absence of SEMMLE_CPP_MISSING_INCLUDES_NOT_FATAL.

I think that having some integration test for this is the only thing that still needs addressing?

Here it is: https://github.com/github/semmle-code/pull/51788

cpp/ql/src/Telemetry/Metrics.qll

jketema · 2024-11-11T13:00:49Z

cpp/ql/src/Telemetry/Diagnostics.qll

+  CannotOpenFile() { this.hasTag(["cannot_open_file", "cannot_open_file_reason"]) }
+
+  string getIncludedFile() {
+    result = this.getMessage().regexpCapture("cannot open source file '([^']+)'", 1)


Are the file names always quoted? This is non-obvious from the frontend source code.

cpp/ql/src/Telemetry/Metrics.qll

cpp/ql/src/Telemetry/Diagnostics.qll

cpp/ql/test/library-tests/extraction_errors/MissingIncludes.qlref

cpp/ql/test/library-tests/extraction_errors/PrintAst.qlref

cpp/ql/src/Telemetry/Metrics.qll

…try" This reverts commit 1a68845, reversing changes made to 349518b.

github-actions bot added the C++ label Nov 1, 2024

calumgrant force-pushed the calumgrant/telemetry branch 2 times, most recently from 913056a to 8035a7c Compare November 1, 2024 17:08

calumgrant marked this pull request as ready for review November 4, 2024 09:28

calumgrant requested a review from a team as a code owner November 4, 2024 09:28

jketema reviewed Nov 4, 2024

View reviewed changes

cpp/ql/test/library-tests/extraction_errors/CompilerErrors.expected Outdated Show resolved Hide resolved

jketema reviewed Nov 4, 2024

View reviewed changes

cpp/ql/src/Telemetry/Diagnostics.qll Outdated Show resolved Hide resolved

cpp/ql/src/Telemetry/Diagnostics.qll Outdated Show resolved Hide resolved

cpp/ql/src/Telemetry/Diagnostics.qll Outdated Show resolved Hide resolved

calumgrant added the no-change-note-required This PR does not need a change note label Nov 4, 2024

calumgrant force-pushed the calumgrant/telemetry branch from 201f282 to ac692ee Compare November 4, 2024 14:04

jketema reviewed Nov 8, 2024

View reviewed changes

cpp/ql/src/Telemetry/MissingIncludes.ql Outdated Show resolved Hide resolved

cpp/ql/src/Telemetry/CompilerErrors.ql Outdated Show resolved Hide resolved

calumgrant force-pushed the calumgrant/telemetry branch from 4fc90ca to 8029eb2 Compare November 8, 2024 14:38

jketema reviewed Nov 11, 2024

View reviewed changes

calumgrant force-pushed the calumgrant/telemetry branch from a4a18be to 76cfb7b Compare November 11, 2024 17:10

calumgrant added 13 commits November 11, 2024 17:16

C++: Initial telemetry queries and tests

f6776a4

C++: Minor edits

a53e1de

C++: Minor edits

6c402ad

C++: Minor edits

8d2cef6

C++: Separate int and float metrics

9a81ce8

C++: Match more tags

067ecde

C++: Limit metrics to top 500

faeff39

C++: Remove unused class UndefinedIdentifier

e6f3512

C++: Limit number of errors/includes to 50

34ee947

C++: Various renamings

fb82d43

C++: Remove missing includes test

593dcb6

C++: Rename SourceFiles metric

2351328

C++: Don't use PrintAST in a test

317f43d

calumgrant force-pushed the calumgrant/telemetry branch from 76cfb7b to 317f43d Compare November 11, 2024 17:17

jketema reviewed Nov 12, 2024

View reviewed changes

cpp/ql/src/Telemetry/Metrics.qll Outdated Show resolved Hide resolved

C++: Some renaming

91b1cb8

jketema approved these changes Nov 12, 2024

View reviewed changes

calumgrant merged commit 1a68845 into main Nov 12, 2024
14 checks passed

calumgrant deleted the calumgrant/telemetry branch November 12, 2024 11:51

jketema added a commit to jketema/codeql that referenced this pull request Nov 28, 2024

Revert "Merge pull request github#17892 from github/calumgrant/teleme…

7e56d69

…try" This reverts commit 1a68845, reversing changes made to 349518b.

jketema added a commit to jketema/codeql that referenced this pull request Nov 28, 2024

Revert "Merge pull request github#17892 from github/calumgrant/teleme…

fa10981

…try" This reverts commit 1a68845, reversing changes made to 349518b.

calumgrant mentioned this pull request Nov 28, 2024

Revert "C++: Initial telemetry queries" #18152

Closed

C++: Initial telemetry queries #17892

C++: Initial telemetry queries #17892

Uh oh!

Conversation

calumgrant commented Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request checklist

All query authors

Internal query authors only

Uh oh!

jketema left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

calumgrant commented Nov 4, 2024

Uh oh!

jketema commented Nov 4, 2024

Uh oh!

jketema left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

calumgrant commented Nov 4, 2024

Uh oh!

Uh oh!

Uh oh!

jketema Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

calumgrant Nov 11, 2024

Choose a reason for hiding this comment

Uh oh!

jketema Nov 11, 2024

Choose a reason for hiding this comment

Uh oh!

calumgrant Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jketema Nov 12, 2024

Choose a reason for hiding this comment

Uh oh!

calumgrant Nov 12, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jketema Nov 11, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

calumgrant commented Nov 1, 2024 •

edited

Loading

jketema Nov 11, 2024 •

edited

Loading

calumgrant Nov 11, 2024 •

edited

Loading