C#: Generate data extension files #10777

michaelnebel · 2022-10-11T13:49:52Z

The purpose of this PR is to use data extensions for MaD models instead of inlining them in the Code QL source code for C#.

To that end we

Introduce a script that produces data extension (YAML) files for the existing hand-written models. The scripts can be used in the future, if we want to change the generated YAML for our existing hand-written models.
Introduce a new model generator scripts that produces a data extension YAML file instead of a Code QL source code file.
Introduce the relevant extensible predicates in ExternalFlow.qll.
Add hand-written models as data extensions and cleanup the Code QL source code for hand-written MaD models.
Add generated models as data extensions for .NET runtime (including negative models) and cleanup the Code QL source for generated and negative models.

Before making any changes the generated .NET runtime models were refreshed as a part of this PR.

github-actions · 2022-10-13T14:39:48Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-14T08:28:28Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-14T08:47:48Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-14T13:33:11Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-24T14:51:34Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-25T10:54:12Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-25T13:53:25Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-25T14:28:26Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-25T15:16:02Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-26T08:51:09Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-26T08:59:15Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-26T11:40:59Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-26T12:56:04Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-26T14:30:51Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-10-26T14:32:17Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

aeisenberg · 2022-10-26T15:45:42Z

cc: @cklin, In previous runs, the C# Language Tests job failed with OOME almost certainly due to 50k tuples from data extensions. We need to think of some ways of being more memory efficient when we read in the tuples.

github-actions · 2022-11-04T07:27:19Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

hvitved

Approach LGTM, and great work on converting all existing flow summaries from .qll to .yml.

csharp/ql/lib/ext/Dapper.model.yml

csharp/ql/lib/semmle/code/csharp/dataflow/ExternalFlow.qll

hvitved · 2022-11-08T10:16:02Z

csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowPrivate.qll

+  private import semmle.code.csharp.frameworks.system.threading.Tasks
+  private import semmle.code.csharp.frameworks.system.runtime.CompilerServices


This files only had code deleted, so why is this needed now and not before (perhaps it was)?

This is to ensure the bi-directional import for synthetic fields - this is implicitly handled by the bi-directional import in the Frameworks module in ExternalFlow.qll.
It turns out that the massive gordian knot that is the Frameworks module in ExternalFlow can't just be deleted (this needs to be a follow up, if we want to do that). I have document the this on: https://github.com/github/codeql-csharp-team/issues/298

Added an import for a similar reason in TaintTrackingPrivate.qll

csharp/ql/src/utils/modelconverter/project/Program.cs

github-actions · 2022-11-08T16:06:44Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

github-actions · 2022-11-09T13:58:37Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

csharp

Generated file changes for csharp

Changes to framework-coverage-csharp.rst:

-    System,"``System.*``, ``System``",4,12131,65,7
+    System,"``System.*``, ``System``",4,12142,65,7
-    Totals,,4,12694,397,7
+    Totals,,4,12705,397,7

Changes to framework-coverage-csharp.csv:

- System,65,4,12131,,8,8,9,,4,,33,3,1,3,10139,1992
+ System,65,4,12142,,8,8,9,,4,,33,3,1,3,10151,1991

Merge commit: 9c6875e

github-actions bot added the C# label Oct 11, 2022

michaelnebel force-pushed the csharp/generatedataextensions branch 5 times, most recently from 5e8692d to 734a98b Compare October 13, 2022 14:37

michaelnebel force-pushed the csharp/generatedataextensions branch from 734a98b to 56bb249 Compare October 14, 2022 08:24

michaelnebel force-pushed the csharp/generatedataextensions branch from c4872aa to 54bd84c Compare October 14, 2022 13:31

michaelnebel force-pushed the csharp/generatedataextensions branch from 54bd84c to b3f1979 Compare October 24, 2022 14:49

michaelnebel force-pushed the csharp/generatedataextensions branch from 8ce33c9 to 516d4e7 Compare October 26, 2022 08:46

michaelnebel force-pushed the csharp/generatedataextensions branch from 516d4e7 to accdd98 Compare October 26, 2022 08:56

michaelnebel force-pushed the csharp/generatedataextensions branch from dde65df to 7f773ac Compare October 26, 2022 12:51

michaelnebel force-pushed the csharp/generatedataextensions branch 2 times, most recently from 10db0c2 to b6f5888 Compare October 26, 2022 14:28

C#: Sync files.

3c8fb05

michaelnebel force-pushed the csharp/generatedataextensions branch from 0c9965e to 3c8fb05 Compare November 4, 2022 07:23

michaelnebel requested a review from hvitved November 4, 2022 07:43

hvitved reviewed Nov 8, 2022

View reviewed changes

michaelnebel added 4 commits November 8, 2022 16:01

C#: Remove [source|sink|summary|negativeSummary]Model/1 predicates.

f1b5ed1

C#: Remove blank lines in generated converted models.

d3a458e

C#: Update all data extensions files.

0b04505

C#: Make a dummy project as a part of running the conversion script.

d8889f2

michaelnebel requested a review from hvitved November 9, 2022 12:26

C#: Add change note.

57e31c1

github-actions bot added the documentation label Nov 9, 2022

michaelnebel marked this pull request as ready for review November 9, 2022 13:55

michaelnebel requested review from a team as code owners November 9, 2022 13:55

hvitved approved these changes Nov 10, 2022

View reviewed changes

michaelnebel merged commit 9c6875e into github:main Nov 10, 2022

michaelnebel deleted the csharp/generatedataextensions branch November 10, 2022 12:08

This was referenced Nov 23, 2022

Java: Use data extensions for MaD models. #11243

Merged

C#: ExternalFlow.qll cleanup. #11395

Merged

owen-mc added a commit to owen-mc/codeql that referenced this pull request Nov 29, 2022

Update shared library files for go to PR github#10777

dbf9d69

Merge commit: 9c6875e

owen-mc added a commit to owen-mc/codeql that referenced this pull request Nov 29, 2022

Update shared library files for go to PR github#10777

f426e22

Merge commit: 9c6875e

owen-mc added a commit to owen-mc/codeql that referenced this pull request Nov 29, 2022

Update shared library files for go to PR github#10777

7f16023

Merge commit: 9c6875e

owen-mc added a commit to owen-mc/codeql that referenced this pull request Nov 29, 2022

Update shared library files for go to PR github#10777

b63d089

Merge commit: 9c6875e

		private import semmle.code.csharp.frameworks.system.threading.Tasks
		private import semmle.code.csharp.frameworks.system.runtime.CompilerServices

C#: Generate data extension files #10777

C#: Generate data extension files #10777

Uh oh!

Conversation

michaelnebel commented Oct 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 13, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 14, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 14, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 14, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 24, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 25, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 25, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 25, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 25, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 26, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 26, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 26, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 26, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 26, 2022

csharp

Generated file changes for csharp

Uh oh!

github-actions bot commented Oct 26, 2022

csharp

Generated file changes for csharp

Uh oh!

aeisenberg commented Oct 26, 2022

Uh oh!

github-actions bot commented Nov 4, 2022

csharp

Generated file changes for csharp

Uh oh!

hvitved left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelnebel commented Oct 11, 2022 •

edited

Loading