Skip to content

Conversation

RasmusWL
Copy link
Member

@RasmusWL RasmusWL commented May 5, 2020

Fixes github/codeql-python-team#64,

Before this PR is ready to be merged, we need to solve 2 things:

  • 1. BoundMethodValue.getParameter(-1)
  • 2. getNamedArgumentForCall for builtins

1. BoundMethodValue.getParameter(-1)

Is it acceptable that BoundMethodValue.getParameter(-1) returns the NameNode for self?

Intuitively I just want to disallow it (since it is very surprising and strange), but I have this nagging feeling that we're removing the ability to find the self parameter and that this might be useful.

2. getNamedArgumentForCall for builtins

Why is getNamedArgumentForCall restricted to only work for non builtins?

instead of having just

call.getArgByName(name) = result

it has this:

call.getArgByName(name) = result and
exists(this.getParameterByName(name))

effectively this means that there are no results for <builtin function or method>.getNamedArgumentForCall(_, _).

But why? Looking in the archives, I could not find an explanation (I did not look further than when the ql repo was moved made public). I could come up with these ideas:

  • We had an assumption that builtin functions and methods never take keyword arguments.

    • This is wrong, since open takes keyword arguments, but is a BuiltinFunctionValue. (all other builtin functions I investigated did not take keyword arguments though).
  • Since we didn't have any mechanism to record the positional index of a keyword argument, we wouldn't be able to offer the translation between positional/keyword arguments. Since that is the only reason you would use getNamedArgumentForCall anyway, we chose to make it fail so users would not get any false assumptions.

  • We don't want to include arguments that are passed to a **kwargs.

    • in which case, we should probably change getArgumentForCall so it doesn't give results for *args

What to do now?

  1. I suggest we remove the exists(this.getParameterByName(name)) restriction.

    • For the call open(file="foo.txt") I think it's very unuseful (and surprising) that getNamedArgumentForCall(call, "file") doesn't give any result.
    • This will change our behavior in regards to **kwargs, which I'm happy about. With def foo(a, b, **kwargs), and foo(a=1, b=2, c=3) I think it's very unuseful that getNamedArgumentForCall(call, "c") will not have any result 😕
  2. Long term, I suggest we record the signature of builtin functions and methods, so we can properly support getArgumentForCall and getNamedArgumentForCall. My impression is that there are only a few builtin functions and methods that take keyword arguments, but I think it's disappointing that we don't properly support those 😐

RasmusWL added 15 commits May 4, 2020 20:49
Showing that
`call.getFunction().refersTo(func)` gives different results from
`call = func.getACall()`
Since other things than FunctionObject can be called ;)
That was not possible when using the old Object-API, but in Value-API getACall
is defined on all Values.
and rename the one for getArgumentForCall
These already have results for BoundMethodValue, although

1) it's a bit strange that `getParameter(-1)` has results
2) why does `Method(Function C.n, class C)` exists? this would only be relevant
if `n` was a classmethod, but it isn't. It's not a problem that it exsits per
se, but curious.
Also fixed a bug for BoundMethodValue, as highlighted in the expected diff 👍
So it matches the structure of getArgumentForCall -- call.getArgByName first!
@RasmusWL RasmusWL added the Python label May 5, 2020
@RasmusWL RasmusWL requested a review from tausbn May 5, 2020 08:27
@tausbn
Copy link
Contributor

tausbn commented May 5, 2020

Regarding your first question (having not looked at the actual code yet), I would be in favour of not having "magic" behaviour like the self parameter being at index -1.
If we need the self parameter, perhaps we should just have a getSelfParameter method?

@RasmusWL
Copy link
Member Author

RasmusWL commented May 5, 2020

Done 👍

Copy link
Contributor

@tausbn tausbn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really good. Especially the documentation.
I only have two minor suggestions.

Regarding named arguments for built-ins, I can only agree with your suggestions. Especially the **kwargs situation is a bit unfortunate currently.

@RasmusWL
Copy link
Member Author

Right, I implemented (1) from What to do now? (in PR description), and created internal tracking ticket for (2). So this should be ready to go 🚀

@RasmusWL RasmusWL marked this pull request as ready for review May 25, 2020 08:21
@RasmusWL RasmusWL requested a review from a team as a code owner May 25, 2020 08:21
@RasmusWL RasmusWL requested a review from tausbn May 25, 2020 08:21
Copy link
Contributor

@tausbn tausbn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!

@semmle-qlci semmle-qlci merged commit ac1a338 into github:master May 25, 2020
@RasmusWL RasmusWL deleted the python-add-BoundMethodValue-v2 branch May 25, 2020 11:01
RasmusWL added a commit to RasmusWL/codeql that referenced this pull request Sep 8, 2020
I looked through PRs between rc/1.24 and rc/1.25 and added missing change notes for:

- github#3314
- github#3302
- github#3212
- github#3453
- github#3407
- github#3563

```
git log --grep="Merge pull request" --format=oneline rc/1.24..rc/1.25 -- python/
```
luchua-bc pushed a commit to luchua-bc/ql that referenced this pull request Oct 16, 2020
I looked through PRs between rc/1.24 and rc/1.25 and added missing change notes for:

- github#3314
- github#3302
- github#3212
- github#3453
- github#3407
- github#3563

```
git log --grep="Merge pull request" --format=oneline rc/1.24..rc/1.25 -- python/
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants