Python dataflow: flow summaries restart #8781

yoff · 2022-04-20T09:35:32Z

This demonstrates that flow summaries can be based on API graphs and all our existing tests can pass 🎉

I have trimmed down the changes needed to get to this state, and given detailed explanations of the tricky transformations needed to avoid non-monotonic recursion.

For a moment, the commit history was beautiful.

Then main moved on and more changes needed to be made, now it is a bit longer and more hairy.
Also, there is a branch where it is even more hairy, cutting up some of the inheritance hierarchies in order to achieve
SourceParameterNode extends LocalSourceNode, but I suggest we leave that for now. SourceParameterNode is no longer something non-internal development should be concerned about.

Names have become a bite nicer and the impact on user code and on .expected-files has been minimised.

We might still do more test, showing off the capabilities, especially the use from MaD which is not demonstrated at the moment. However, simply having it compile and the basic tests pass seems like a worthy milestone at this point.

asgerf · 2022-04-20T11:11:35Z

I've only done a superficial glance at this, but I'd like to talk about the issue with getACall2. This is a fundamental issue when using API graphs and flow summaries together, and will likely affect all dynamic languages attempting to use flow summaries. I'd like to get a bit more clarity on what the problem is and what solution to use (ideally we'll all end up using the same solution).

The root of the problem is that the following cycle is created between call graph construction and API graphs:

Unless we want API graphs to be mutually recursive with type-tracking steps and call construction, we'll need to break the cycle somehow. As far as I can tell, you've broken the cycle like this:

This means anything not depending on API graphs can contribute to getACall2, and things depending on API graphs can use getACall. Type tracking steps will only be discovered for getACall2, which makes it more suitable for things like standard library modelling.

I believe Ruby's solution so far is to make type-tracking steps explicitly depend on non-library callables only. The downside here is that none of their standard library models are visible to type-tracking. (@hvitved @aibaars do you agree here?)

@yoff Since I'm unfamiliar with the Python libraries I'm not 100% sure my interpretation above is an accurate description of what you've done here. Could you please check if this is the case? If so, I think it's a fine solution provided we can come up with names that aren't confusing.

yoff · 2022-04-21T07:30:14Z

Force pushed because I could see that my amended commit message to the last commit had not been pushed.

hvitved · 2022-04-21T08:21:57Z

I believe Ruby's solution so far is to make type-tracking steps explicitly depend on non-library callables only. The downside here is that none of their standard library models are visible to type-tracking. (@hvitved @aibaars do you agree here?)

Correct (since flow summaries depend on API graphs, which depend on type tracking).

yoff · 2022-04-21T10:31:19Z

The diagram looks correct to me. Is the difference with Ruby just that we have not (yet?) written any standard library models using flow summaries? Or is there a more fundamental difference?

github-advanced-security

Found 8 vulnerabilities.

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll

python/ql/lib/semmle/python/frameworks/Stdlib.qll

python/ql/lib/semmle/python/dataflow/new/FlowSummary.qll

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowDispatchPointsTo.qll

python/ql/lib/semmle/python/dataflow/new/internal/FlowSummaryImpl.qll

+    /**
+     * A query predicate for outputting flow summaries in semi-colon separated format in QL tests.
+     * The syntax is: "namespace;type;overrides;name;signature;ext;inputspec;outputspec;kind",
+     * ext is hardcoded to empty.
+     */


python/ql/lib/semmle/python/dataflow/new/internal/DataFlowDispatchPointsTo.qll

python/ql/src/Security/CWE-020-ExternalAPIs/ExternalAPIs.qll

yoff · 2022-05-04T12:20:17Z

The failing language test belongs to https://github.com/github/semmle-code and cannot be fixed in this PR (I believe the fix is simply to update the test expectations).

tausbn

Thank you for this Herculean effort!

I think in terms of the implementation, this looks pretty solid. I've added a bunch of requests for clarification (and also some requests for renaming stuff). I'll be very happy to discuss this synchronously if that's easier. 🙂

tausbn · 2022-05-02T11:00:57Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPublic.qll

+  abstract predicate isParameterOf(DataFlowCallable c, int i);
+}
+
+class SourceParameterNode extends ParameterNode, CfgNode {


I took me a while to figure out what Source was referring to in this context (what with this being dataflow and all). I wonder if there is a better name for this, or is SourceParameterNode just the accepted term?

tausbn · 2022-05-02T11:02:55Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowDispatchPointsTo.qll

+  }
+}
+
+abstract class DataFlowSourceCall extends DataFlowCall, TDataFlowSourceCall {


Like my comment on SourceParameterNode, I feel like this is also potentially confusing. At the very least there should be QLDoc that explains what's going on.

tausbn · 2022-05-09T11:57:13Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll

@@ -282,6 +282,8 @@ module EssaFlow {
      nodeTo = TKwOverflowNode(call, callable) and
      nodeFrom.asCfgNode() = call.getNode().getKwargs().getAFlowNode()
    )
+    or
+    FlowSummaryImpl::Private::Steps::summaryLocalStep(nodeFrom, nodeTo, true)


Hmm... So we allow summaries to extend local flow steps (which means they underpin all of the dataflow machinery). Does this not lead to everything being recursively dependent?

No, there are now two local step relations. One for type trackers and one for the global data flow computation.

tausbn · 2022-05-09T12:01:46Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll

+  kind = kind and
+  exists(Call call, Name f, FunctionDef def |
+    f = call.getAnArg() and
+    def.getDefinedFunction().getName() = f.getId() and
+    // c.getCallableValue() = def.getDefinedFunction().getDefinition() and
+    c.getName() = f.getId()
+  )


This may have been fixed in a later commit, but as QL-for-QL helpfully points out, creation isn't restricted by this disjunct.

tausbn · 2022-05-09T12:08:10Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowDispatchPointsTo.qll

+   * Includes function calls, method calls, class calls and library calls.
+   * All these will be associated with a `CallNode`.
+   */
+  TNonSpecialCall(CallNode call) or


Something about NonSpecial... rubs me the wrong way. Could we use a positive formulation instead? I'm tempted to suggest CallNodeCall, though that is a bit of a mouthful. (However, it does precisely say that these are calls that happen at call nodes.)

Alternatively, since we have SpecialCall, perhaps its counterpart should be NormalCall?

tausbn · 2022-05-09T12:21:24Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowDispatchPointsTo.qll

+  /** Gets a callable value for this callable, if one exists. */
+  abstract CallableValue getCallableValue();
+
+  abstract CallNode getACall2();


This needs a better name. Even if it is only internal, it certainly doesn't convey what it's actually supposed to do.

tausbn · 2022-05-09T12:28:07Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowDispatchPointsTo.qll

@@ -497,59 +504,69 @@ class NonSpecialCall extends DataFlowSourceCall, TNonSpecialCall {
  override DataFlowCallable getEnclosingCallable() { result.getScope() = call.getNode().getScope() }
 }

+abstract class NonLibraryDataFlowSourceCall extends NonSpecialCall {


I think this name needs a bit of workshopping. Could we come up with a better description of "non-library"?

tausbn · 2022-05-09T12:35:52Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPublic.qll

+}
+
+class SourceParameterNode extends ParameterNode, CfgNode {
+  //, LocalSourceNode {


I'm still a bit ambivalent about these not being local sources, and I would like to understand this a bit more. Is this merely because of negative recursion issues?

Negative recursion is what directly prevents me from inheriting. But I can add that inheritance into the Ruby codebase which basically has the same structure, so I think the solution is to clear up the mutual recursive structure of our predicates, which I fear lacks a bit of discipline at the moment.

With all of the recent changes, is your comment still in effect?

Unfortunately, yes. I do feel we should be able to resolve this, but I had a quick go and it seems there are multiple cycles to untangle, so I would prefer to do this later..

Alright. Thanks for checking!

hvitved

A few drive-by comments. Is the logic actually hooked up yet? I fail to see references to e.g. any of the FlowSummaryImpl::Private::Steps predicates. See e.g. DataFlowPrivate.qll for ruby for how this is done.

hvitved · 2022-05-09T17:58:55Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPublic.qll

@@ -277,21 +286,26 @@ ExprNode exprNode(DataFlowExpr e) { result.getNode().getNode() = e }
 * The value of a parameter at function entry, viewed as a node in a data
 * flow graph.
 */
-class ParameterNode extends CfgNode, LocalSourceNode {
+abstract class ParameterNode extends Node {


Are you sure you want an abstract class in the public API?

Uh, probably not..

hvitved · 2022-05-09T17:59:14Z

python/ql/lib/semmle/python/dataflow/new/internal/FlowSummaryImplSpecific.qll

@@ -0,0 +1,175 @@
+/**
+ * Provides Ruby specific classes and predicates for defining flow summaries.


hvitved · 2022-05-09T18:01:42Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll

+    or
+    call = any(ClassValue c | not c.isAbsent()).getACall()
+    or
+    call = any(SpecialMethodCallNode special)


call instanceof SpecialMethodCallNode

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowDispatchPointsTo.qll

hvitved · 2022-05-09T18:06:15Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll

@@ -2,6 +2,7 @@ private import python
 private import DataFlowPublic


I think you are missing allowParameterReturnInSelf, see https://github.com/github/codeql/blob/main/ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowPrivate.qll#L940.

Thanks! Indeed we just have none().

yoff · 2022-05-10T11:40:11Z

A few drive-by comments. Is the logic actually hooked up yet? I fail to see references to e.g. any of the FlowSummaryImpl::Private::Steps predicates. See e.g. DataFlowPrivate.qll for ruby for how this is done.

They are mostly in DataFlowPrivate.qll and then one in DataFlowUtil.qll. I followed the structure from Ruby closely. The interface does not let you search for these terms inside collapsed files, which is a bit misleading.

hvitved · 2022-05-10T12:22:44Z

They are mostly in DataFlowPrivate.qll and then one in DataFlowUtil.qll. I followed the structure from Ruby closely. The interface does not let you search for these terms inside collapsed files, which is a bit misleading.

Ah, good, sorry for the false alarm. I guess I must have looked at a diff limited to a subset of commits perhaps.

- add flowsummaries shared files - register in indentical files - fix initial non-monotonic recursions - add DataFlowSourceCall - add resolvedCall - add SourceParameterNode failing tests: - 3/library-tests/with/test.ql

allowing more `OutNode`s (not restricting to `CallNode`s), gives more flow in the `classesCallGraph` test

also convert to inline tests

summary flow is excluded from the local flow relation used for typetracking, but included in the one used for global data flow.

- Branch predicates are made simple. In particular, they do not try to detect library calls. - All branches based on `CallNode`s are gathered into one. - That branch has been given a class `NonSpecialCall`, which is the new parent of call classes based on `CallNode`s. (Those classes now have more involved charpreds.) - A new such class, 'LambdaCall` has been split out from `FunctionCall` to allow the latter to replace its general `CallNode` field with a specific `FunctionValue` one. - `NonSpecialCall` is not an abstract class, but it has some abstract overrides. Therefor, it is not considered a resolved call in the test `UnresolvedCalls.qll`.

Two classes have been inserted into the hierarchies: - `NonLibraryDataFlowCallable` with a method `getACall2`. This method implements "get a call, not considering flow summaries". For `NonLibraryDataFlowCallable`s, `getACall` will defer to `getACall2`. While you could have a synthesised call to such a callable, it would not correspond to a `CallNode`. - `NonLibraryDataFlowSourceCall` with methods `getArg2` and `getCallable2`. These also refer to a call graph that does not consider flow summaries. `getArg2` is used to synthesise pre-update nodes for arguments. `getCallable2` is used in `connects` to compute argument passing. This is used to define data flow nodes for overflow arguments. `getACall2` ensures that `LibraryCallableValue::getACall` is not called when the charpred of `FunctionCall` is evaluated.

- sync summary files - format files - fix compilation

This can be used when one wants to consider a (source) parameter node as a local source.

github-advanced-security

Found 1 vulnerability.

python/ql/lib/semmle/python/frameworks/Django.qll

by making client code use the "new" class. Really, this part of the split class should have the old name, to minimise disruptions to clients. Same goes for the other split classes.

yoff · 2022-05-11T10:00:38Z

I think in terms of the implementation, this looks pretty solid. I've added a bunch of requests for clarification (and also some requests for renaming stuff). I'll be very happy to discuss this synchronously if that's easier. 🙂

Thank you for the review. I think both renaming and qldocs are warranted. Specifically, when classes are split (in order to only have one participate in a recursion that would otherwise be non-monotonic) the user-facing one should be retaining the old name so as to minimise breaking changes.

to avoid nodes in the extracted stdlib Was there a reason for this depth to be 1?

since map resolves to a class call also fix test expectation

This is not necessary as long as `LibraryCall` only includes unresolved calls.

RasmusWL

I think it would be super nice with a change to the result type of SummarizedCallable.getACall. (which seems like a fairly simple change, I can even do it myself if you want me to)

Besides that, I don't think I have requests for any other significant changes.

I put it in frameworks\Builtins.qll. It could also go in the StdLib folder, I guess. Or perhaps you have a different place that you find natural, feel free to suggest :-)

We have normally put both builtins and standard library modeling in the Stdlib.qll file. I guess that is not technically 100% correct. However, on the top of my head, it doesn't seem like we're gaining anything from splitting it into two, so I would suggest you move it into the existing file (but do let me know if I'm overlooking something 🤷). ((note: if you do want to keep the Builtins.qll file, you should add module structure like in all the other files in the frameworks folder)

RasmusWL · 2022-09-19T08:38:53Z

python/ql/lib/semmle/python/frameworks/Builtins.qll

+class ReversedSummary extends SummarizedCallable {
+  ReversedSummary() { this = "builtins.reversed" }
+
+  override CallNode getACall() { result = API::builtin("reversed").getACall().getNode() }


This line of code makes me wonder if we could change getACall for SummarizedCallable to have DataFlow::CallCfgNode as a result type, so this could just be written as result = API::builtin("reversed").getACall(). The extra .getNode() seems like it will be included every time we define a SummarizedCallable, so we might as well get away with it now?

It is slightly inconsistent for LibraryCallable to return a DatFlow::Node here, when the other callables return control flow nodes, but I tried it now, see if you like :-)

yes, I'm happy 👍 Much of it will be changed with the new call-graph anyway 🤷

python/ql/lib/semmle/python/dataflow/new/internal/FlowSummaryImplSpecific.qll

python/ql/lib/semmle/python/dataflow/new/FlowSummary.qll

python/ql/lib/semmle/python/dataflow/new/TestSummaries.qll

The other callables return control flow nodes, so it is slightly inconsistent for this to return a data flow node, but it does make models based on API graphs nicer.

python/ql/lib/semmle/python/Frameworks.qll

RasmusWL

Minor fixes needed... but almost 😅

python/ql/lib/semmle/python/Frameworks.qll

python/ql/lib/semmle/python/dataflow/new/internal/FlowSummaryImplSpecific.qll

Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>

RasmusWL

🤞 for the tests 😆

yoff · 2022-09-19T12:53:55Z

🤞 for the tests 😆

Indeed, thanks for the fix :-)

tausbn

A few minor comments here and there. Otherwise, I think this looks very good!

Regarding the overloading of "source" to mean "in source code", how about "extracted"? This, to me, seems fairly unambiguous...

tausbn · 2022-09-20T07:54:06Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll

-predicate lambdaCall(DataFlowCall call, LambdaCallKind kind, Node receiver) { none() }
+predicate lambdaCall(DataFlowCall call, LambdaCallKind kind, Node receiver) {
+  receiver = call.(SummaryCall).getReceiver() and
+  kind = kind


So, if I understand correctly, this is to allow kind to be any LambdaCallKind? If so, maybe exists(kind) is a more idiomatic way of indicating an intentional cartesian product?

That is correct (and LambdaCallKind is currently Unit).

tausbn · 2022-09-20T07:54:57Z

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPublic.qll

+}
+
+class SourceParameterNode extends ParameterNode, CfgNode {
+  //, LocalSourceNode {


With all of the recent changes, is your comment still in effect?

The precedence for the use of "source" to denote elements of source code is found in `EssaVariable::getSourceVariable` as well as in the Ruby code base. But it clashes with the many uses of source to mean "source of flow" found in the data flow library.

tausbn

github-actions bot added the Python label Apr 20, 2022

yoff force-pushed the python-dataflow/flow-summaries-from-scratch branch from 1430ab6 to b4aa941 Compare April 21, 2022 06:15

yoff force-pushed the python-dataflow/flow-summaries-from-scratch branch from b4aa941 to ebedc7e Compare April 28, 2022 11:16

github-advanced-security bot found potential problems Apr 28, 2022

View reviewed changes

hvitved mentioned this pull request May 2, 2022

Python: add MaD implementation #8883

Merged

tausbn requested changes May 9, 2022

View reviewed changes

hvitved reviewed May 9, 2022

View reviewed changes

yoff and others added 13 commits May 10, 2022 12:48

Python: Compiles and mostly pass tests

80175a9

- add flowsummaries shared files - register in indentical files - fix initial non-monotonic recursions - add DataFlowSourceCall - add resolvedCall - add SourceParameterNode failing tests: - 3/library-tests/with/test.ql

python: Add summary nodes

828db3a

allowing more `OutNode`s (not restricting to `CallNode`s), gives more flow in the `classesCallGraph` test

python: add summary flow steps

8c263b3

python: some summary flows

4024ce4

python: use new syntax for flow summaries

177dea5

also convert to inline tests

Python: local flow for type tracking

81ca479

summary flow is excluded from the local flow relation used for typetracking, but included in the one used for global data flow.

python: type tracking uses source nodes

d85844b

python: variaous fixes

da36341

- sync summary files - format files - fix compilation

python: add qldoc

28b239a

python: Add LocalSourceParameterNode

238c578

This can be used when one wants to consider a (source) parameter node as a local source.

python: summaries may allowParameterReturnInSelf

db008f1

github-advanced-security bot found potential problems May 10, 2022

View reviewed changes

python/ql/lib/semmle/python/frameworks/Django.qll Fixed Show fixed Hide fixed

python: fix compilation

f67be52

by making client code use the "new" class. Really, this part of the split class should have the old name, to minimise disruptions to clients. Same goes for the other split classes.

Python: --max-import-depth=0

56dcfc2

to avoid nodes in the extracted stdlib Was there a reason for this depth to be 1?

yoff force-pushed the python-dataflow/flow-summaries-from-scratch branch from ede5f3c to 56dcfc2 Compare September 12, 2022 21:34

yoff added 4 commits September 12, 2022 23:38

Python: remove ressurected file

2e9c60d

Python: fix QL alerts

03c2431

Python: format

c1ab661

Python: adjust expectations to new spelling

58cfac2

yoff marked this pull request as draft September 14, 2022 07:19

yoff added 2 commits September 14, 2022 10:28

Python: do not stake out too much territory

f83158f

Python: rename summary map -> list_map,

245baa5

since map resolves to a class call also fix test expectation

yoff marked this pull request as ready for review September 14, 2022 10:40

Python: undo change to --max-import-depth

33b508d

This is not necessary as long as `LibraryCall` only includes unresolved calls.

RasmusWL requested changes Sep 19, 2022

View reviewed changes

yoff added 4 commits September 19, 2022 14:02

Python: change type of LibraryCallable::getACall

37fb27f

The other callables return control flow nodes, so it is slightly inconsistent for this to return a data flow node, but it does make models based on API graphs nicer.

Python: comment out SummarizedCallableFromModel

da39c14

Python: expand comment on flow summaries

f560719

Python: move summary to Stdlib.qll

6377e6c

yoff requested a review from RasmusWL September 19, 2022 12:37

github-advanced-security bot found potential problems Sep 19, 2022

View reviewed changes

python/ql/lib/semmle/python/Frameworks.qll Fixed Show fixed Hide fixed

python/ql/lib/semmle/python/Frameworks.qll Fixed Show fixed Hide fixed

RasmusWL requested changes Sep 19, 2022

View reviewed changes

python/ql/lib/semmle/python/Frameworks.qll Outdated Show resolved Hide resolved

python/ql/lib/semmle/python/dataflow/new/internal/FlowSummaryImplSpecific.qll Outdated Show resolved Hide resolved

Apply suggestions from code review

f7cbcb2

Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>

yoff requested a review from RasmusWL September 19, 2022 12:52

RasmusWL previously approved these changes Sep 19, 2022

View reviewed changes

tausbn reviewed Sep 20, 2022

View reviewed changes

Python: more idiomatic cartesian product

9a7afa9

yoff dismissed RasmusWL’s stale review via 9a7afa9 September 20, 2022 10:48

yoff requested a review from tausbn September 20, 2022 11:29

tausbn approved these changes Sep 20, 2022

View reviewed changes

yoff merged commit ea74317 into github:main Sep 20, 2022

		@@ -0,0 +1,175 @@
		/**
		* Provides Ruby specific classes and predicates for defining flow summaries.

		@@ -2,6 +2,7 @@ private import python
		private import DataFlowPublic

Python dataflow: flow summaries restart #8781

Python dataflow: flow summaries restart #8781

Uh oh!

Conversation

yoff commented Apr 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asgerf commented Apr 20, 2022

Uh oh!

yoff commented Apr 21, 2022

Uh oh!

hvitved commented Apr 21, 2022

Uh oh!

yoff commented Apr 21, 2022

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Uh oh!

Uh oh!

Uh oh!

yoff commented May 4, 2022

Uh oh!

tausbn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hvitved left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yoff commented May 10, 2022

Uh oh!

hvitved commented May 10, 2022

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

yoff commented Apr 20, 2022 •

edited

Loading

yoff Sep 19, 2022 •

edited

Loading