python rules do not match list comprehension correctly

[rgraber]

Describe the bug
Using a python rule to match a list comprehension fails, in certain cases, to correctly match the full expression, resulting in extra ]s appearing after fixes are applied. Also, possibly a separate bug, repeated metavariables don't seem to work inside a list comprehension.

To Reproduce
https://semgrep.dev/s/0ykw

Expected behavior

1. I expect $Z = [ $X for $X in $Y] (python-replace-useless-comprehension-misses) to match thing = [ x for x in range(10) ]
2. I expect python-replace-useless-comprehension-hits to replace thing = [ x for x in range(10)] with thing = range(10) instead of thing = range(10)]

Screenshots
Worth nothing: it looks like the second matcher is matching everything except the last ], which is odd, but explains why the replacement has an extra ] at the end

What is the priority of the bug to you?

• P0: blocking your adoption of Semgrep or workflow
• P1: important to fix or quite annoying
• P2: regular bug that should get fixed

Environment
If not using semgrep.dev: are you running off docker, an official binary, a local build?

[r2c-demo]

This issue is synced in Linear at https://linear.app/r2c/issue/PA-664/python-rules-do-not-match-list-comprehension-correctly.

[rgraber]

Updated version of rule to demonstrate where we first encountered this: https://semgrep.dev/s/5D50

We have a method very_important_method() that was copied and pasted across several repositories. We want to update this method with the new standard across several repositories. However, whenever the current implementation of very_important_method ends with a list comprehension, somehow the very last square bracket doesn't get replaced, resulting in unparseable code.

[rgraber]

More updates: I'm unclear if this is the same issue but it seems similar even though it's not actually a list comprehension. A method that ends with a long compound boolean expression was not matched correctly: https://semgrep.dev/s/66G1

[aryx]

hmm you see @emjin with this last example I am more and more tempted to use ParenExpr there too

[ievans]

Thanks for the report @rgraber. I expect this fix will go out in 0.77 next week!