match on type of immediate #49
Assignees
Labels
enhancement
New feature or request
Comments
aryx
added a commit
to semgrep/pfff
that referenced
this issue
Feb 5, 2020
This will help semgrep/semgrep#49 Test plan: make test
aryx
added a commit
that referenced
this issue
Feb 5, 2020
This fixes issue #49. Test plan: test file included make test
aryx
added a commit
that referenced
this issue
Feb 5, 2020
* Typed metavar (part 2, sgrep part) for literals. This fixes issue #49. Test plan: test file included make test * * lib/typechecking_generic.ml: small diff, just to retrigger CI with latest pfff
|
@aryx I was unable to get this to work using the patterns in https://github.com/returntocorp/sgrep-rules/blob/develop/python/exceptions/exceptions.yaml#L7-L10; ie:
|
|
There are some ambiguities in the python grammar if you use $X : int anywhere in an expression, it's because K: V is also a key/value in a dictionary. Right now I've extended the python gramamr to allow X:T to appear only in arguments (the examples you had above). I can extend that to anywhere, but then you will have to use parenthesis around, as in foo(($X:int)). What do you prefer? |
DrewDennison
added a commit
that referenced
this issue
Feb 6, 2020
* rename and add black (#62) * rename and add black * blacken * exclude * Update README.md (#63) * add --error to exit 1 if there are findings (#64) * adapt to latest pfff, after AST generic modifications with the Golang integration (#69) test plan: make test also I've disabled the python regression testing because of hardcoded path to isaac dir * support for nested booleans (#67) Updated docs. Re-use patterns key to mean (group of ANDed expressions). Cleaned up some of the traversal logic Add support for nested boolean expression, e.g.: - pattern-either: - pattern: $X == $X - pattern: $X != $X - patterns: - pattern-inside: | def __init__(...): ... - pattern: self.$X == self.$X * * bin/main_sgrep.ml: use better Lang.files_of_dirs... when can. (#71) Should fix issue #66 Test plan: $ ./_build/default/bin/main_sgrep.exe -lang javascript -e foo tests/js/ * Support matching over decorators/attributes/annotations (#72) This solves issue #29 Test plan: file test included make test * Update python.yml (#75) * Adapt to latest pfff, and allow ... to match optional expression in return (#78) This should fix issue #73 and #52 Test plan: tests included $ make test * conform to r2c output with relative paths (#77) * make paths relative to cwd * cleanup rule id prefixes a bit * update expected test, re-enable (#79) * multilingual (#80) * group by languages for sgrep * add a test for multilingual lint * Update README.md (#82) close #81 * Ie/dockerfile paths (#83) * cleanup paths and re-written check IDs when running in docker * black * prefix stripping * revert Dockerfile * safer relative paths * black * update action names (#84) * Change readme to correct code equivalence (#86) * Better class matching (#85) * * lib/generic_vs_generic.ml: comments * * tests/python/dots_fields.py: support for dots in fields. This is related to issue #50 Test plan: test included. * * bin/main_sgrep.ml: add dumper actions to cli, -dump_pattern and -dump_ast * * lib/generic_vs_generic.ml: misc renaming, more consistent use of m_list__xxx * Dots for inheritance, and less-is-ok for inheritance Test plan: tests included make test * Typed metavar (part 2, sgrep part) for literals. (#88) * Typed metavar (part 2, sgrep part) for literals. This fixes issue #49. Test plan: test file included make test * * lib/typechecking_generic.ml: small diff, just to retrigger CI with latest pfff * Function pattern now match methods (and nested functions) (#89) This fixes issue #65 Test plan: tests included. * New --validate-pattern-stdin command for sgrep, to be used in CI (#97) This fixes issue #6 Test plan: echo '$X == $X' | ./_build/default/bin/main_sgrep.exe -lang python --validate-pattern-stdin pad@yrax:~/github/sgrep$ echo $? 0 pad@yrax:~/github/sgrep$ echo '$X == ' | ./_build/default/bin/main_sgrep.exe -lang python --validate-pattern-stdin pad@yrax:~/github/sgrep$ echo $? 1 * * lib/generic_vs_generic.ml: CEs on recent changes on ast_generic (#98) removal of Nop and additions of a few more expr option test plan: make test still works * tests/go/: adding more tests for sgrep for Go! (#99) * tests/go/: adding more tests for sgrep for Go! This should fix issue #96 Test plan: test files included * * tests/go/: more tests * * tests/go/TODO: a few TODOs * * lib/generic_vs_generic.ml: CEs on CaseEqualExpr * * tests/go/: remove optional ';' Test plan: make test also pad@yrax:~/github/sgrep$ ./_build/default/bin/main_sgrep.exe -lang go -e 'foobar()' -r2c ~/work/lang-go/ now works without any exceptions or noice on stdout * Add meetup invitation link (#100) Co-authored-by: Drew Dennison <dennison@mit.edu> Co-authored-by: Yoann Padioleau <yoann.padioleau@gmail.com> Co-authored-by: Pablo Estrada <pabloest@yahoo.com>
|
I think the approach you took is good! Thanks for the clarification |
DrewDennison
added a commit
that referenced
this issue
Feb 10, 2020
* rename and add black (#62) * rename and add black * blacken * exclude * Update README.md (#63) * add --error to exit 1 if there are findings (#64) * adapt to latest pfff, after AST generic modifications with the Golang integration (#69) test plan: make test also I've disabled the python regression testing because of hardcoded path to isaac dir * support for nested booleans (#67) Updated docs. Re-use patterns key to mean (group of ANDed expressions). Cleaned up some of the traversal logic Add support for nested boolean expression, e.g.: - pattern-either: - pattern: $X == $X - pattern: $X != $X - patterns: - pattern-inside: | def __init__(...): ... - pattern: self.$X == self.$X * * bin/main_sgrep.ml: use better Lang.files_of_dirs... when can. (#71) Should fix issue #66 Test plan: $ ./_build/default/bin/main_sgrep.exe -lang javascript -e foo tests/js/ * Support matching over decorators/attributes/annotations (#72) This solves issue #29 Test plan: file test included make test * Update python.yml (#75) * Adapt to latest pfff, and allow ... to match optional expression in return (#78) This should fix issue #73 and #52 Test plan: tests included $ make test * conform to r2c output with relative paths (#77) * make paths relative to cwd * cleanup rule id prefixes a bit * update expected test, re-enable (#79) * multilingual (#80) * group by languages for sgrep * add a test for multilingual lint * Update README.md (#82) close #81 * Ie/dockerfile paths (#83) * cleanup paths and re-written check IDs when running in docker * black * prefix stripping * revert Dockerfile * safer relative paths * black * update action names (#84) * Change readme to correct code equivalence (#86) * Better class matching (#85) * * lib/generic_vs_generic.ml: comments * * tests/python/dots_fields.py: support for dots in fields. This is related to issue #50 Test plan: test included. * * bin/main_sgrep.ml: add dumper actions to cli, -dump_pattern and -dump_ast * * lib/generic_vs_generic.ml: misc renaming, more consistent use of m_list__xxx * Dots for inheritance, and less-is-ok for inheritance Test plan: tests included make test * Typed metavar (part 2, sgrep part) for literals. (#88) * Typed metavar (part 2, sgrep part) for literals. This fixes issue #49. Test plan: test file included make test * * lib/typechecking_generic.ml: small diff, just to retrigger CI with latest pfff * Function pattern now match methods (and nested functions) (#89) This fixes issue #65 Test plan: tests included. * New --validate-pattern-stdin command for sgrep, to be used in CI (#97) This fixes issue #6 Test plan: echo '$X == $X' | ./_build/default/bin/main_sgrep.exe -lang python --validate-pattern-stdin pad@yrax:~/github/sgrep$ echo $? 0 pad@yrax:~/github/sgrep$ echo '$X == ' | ./_build/default/bin/main_sgrep.exe -lang python --validate-pattern-stdin pad@yrax:~/github/sgrep$ echo $? 1 * * lib/generic_vs_generic.ml: CEs on recent changes on ast_generic (#98) removal of Nop and additions of a few more expr option test plan: make test still works * tests/go/: adding more tests for sgrep for Go! (#99) * tests/go/: adding more tests for sgrep for Go! This should fix issue #96 Test plan: test files included * * tests/go/: more tests * * tests/go/TODO: a few TODOs * * lib/generic_vs_generic.ml: CEs on CaseEqualExpr * * tests/go/: remove optional ';' Test plan: make test also pad@yrax:~/github/sgrep$ ./_build/default/bin/main_sgrep.exe -lang go -e 'foobar()' -r2c ~/work/lang-go/ now works without any exceptions or noice on stdout * Add meetup invitation link (#100) * Update README.md (#93) * better handling of invalid keys (#94) * better handling of invalid keys * use validate patterns flag in sgrep (#102) * use validate patterns flag in sgrep * Update README.md (#107) * clean up paths, skip hidden dirs, and use r2c as the default config (#103) * clean up paths, skip hidden dirs, and use r2c as the default config * remove debugging try * add generate config arg * only replace if base path is given * blacken * change to kwarg * ignore when folder doesn't exist (#108) ebf * fix bad merge * Update README.md (#112) * Create sgrep-lint.yml (#111) * Create sgrep-lint.yml * Update sgrep-lint.yml * normal output (#113) * actually write and create file given with -o (#114) * use -o instead of | tee (#117) * use -o instead of | tee * remove newline * better runs with -e (#118) * Update sgrep-lint.yml (#119) * Regression tests showing the absence of FakeTok exn. (#121) This shows that issue #104 and #106 currently work with the develop branch. Test plan: file tests included, make test * Newline has a meaning in Go which automatically inserts semicolons with (#122) certain newlines. In sgrep we do not want those extra semicolons hence this diff. Test plan: make test * ask user to report sgrep core errors (#125) * ask user to report sgrep core errors * add pre-commit * trying out pre-commit * * tests/java/misc_assert.java: regression test (#127) This fixes issue #123 * make sgrep a pre-commit hook (#129) * add integrations.md (#130) * Update .pre-commit-hooks.yaml (#131) * Update .pre-commit-hooks.yaml * fmt * remove no-commit-to-branch breaks CI (#132) * Update sgrep-lint.yml (#133) * Update sgrep-lint.yml * add sgrep to our pre-commit hook (#134) * add sgrep to our pre-commit hook * This should fix issue #106 (#137) Test plan: in bento-core/ repo git checkout lomalley/sgrep-test ~/sgrep/.../main_sgrep.exe -verbose -rules_file .sgrep.yml bento/ no exn anymore * Bugfix/finding config (#135) * fix bug where config not found if path contains .. * --exclude-tests option, friendlier error messages * black Co-authored-by: Yoann Padioleau <yoann.padioleau@gmail.com> Co-authored-by: Isaac Evans <409041+ievans@users.noreply.github.com> Co-authored-by: Pablo Estrada <pabloest@yahoo.com>
DrewDennison
added a commit
that referenced
this issue
Feb 10, 2020
* rename and add black (#62) * rename and add black * blacken * exclude * Update README.md (#63) * add --error to exit 1 if there are findings (#64) * adapt to latest pfff, after AST generic modifications with the Golang integration (#69) test plan: make test also I've disabled the python regression testing because of hardcoded path to isaac dir * support for nested booleans (#67) Updated docs. Re-use patterns key to mean (group of ANDed expressions). Cleaned up some of the traversal logic Add support for nested boolean expression, e.g.: - pattern-either: - pattern: $X == $X - pattern: $X != $X - patterns: - pattern-inside: | def __init__(...): ... - pattern: self.$X == self.$X * * bin/main_sgrep.ml: use better Lang.files_of_dirs... when can. (#71) Should fix issue #66 Test plan: $ ./_build/default/bin/main_sgrep.exe -lang javascript -e foo tests/js/ * Support matching over decorators/attributes/annotations (#72) This solves issue #29 Test plan: file test included make test * Update python.yml (#75) * Adapt to latest pfff, and allow ... to match optional expression in return (#78) This should fix issue #73 and #52 Test plan: tests included $ make test * conform to r2c output with relative paths (#77) * make paths relative to cwd * cleanup rule id prefixes a bit * update expected test, re-enable (#79) * multilingual (#80) * group by languages for sgrep * add a test for multilingual lint * Update README.md (#82) close #81 * Ie/dockerfile paths (#83) * cleanup paths and re-written check IDs when running in docker * black * prefix stripping * revert Dockerfile * safer relative paths * black * update action names (#84) * Change readme to correct code equivalence (#86) * Better class matching (#85) * * lib/generic_vs_generic.ml: comments * * tests/python/dots_fields.py: support for dots in fields. This is related to issue #50 Test plan: test included. * * bin/main_sgrep.ml: add dumper actions to cli, -dump_pattern and -dump_ast * * lib/generic_vs_generic.ml: misc renaming, more consistent use of m_list__xxx * Dots for inheritance, and less-is-ok for inheritance Test plan: tests included make test * Typed metavar (part 2, sgrep part) for literals. (#88) * Typed metavar (part 2, sgrep part) for literals. This fixes issue #49. Test plan: test file included make test * * lib/typechecking_generic.ml: small diff, just to retrigger CI with latest pfff * Function pattern now match methods (and nested functions) (#89) This fixes issue #65 Test plan: tests included. * New --validate-pattern-stdin command for sgrep, to be used in CI (#97) This fixes issue #6 Test plan: echo '$X == $X' | ./_build/default/bin/main_sgrep.exe -lang python --validate-pattern-stdin pad@yrax:~/github/sgrep$ echo $? 0 pad@yrax:~/github/sgrep$ echo '$X == ' | ./_build/default/bin/main_sgrep.exe -lang python --validate-pattern-stdin pad@yrax:~/github/sgrep$ echo $? 1 * * lib/generic_vs_generic.ml: CEs on recent changes on ast_generic (#98) removal of Nop and additions of a few more expr option test plan: make test still works * tests/go/: adding more tests for sgrep for Go! (#99) * tests/go/: adding more tests for sgrep for Go! This should fix issue #96 Test plan: test files included * * tests/go/: more tests * * tests/go/TODO: a few TODOs * * lib/generic_vs_generic.ml: CEs on CaseEqualExpr * * tests/go/: remove optional ';' Test plan: make test also pad@yrax:~/github/sgrep$ ./_build/default/bin/main_sgrep.exe -lang go -e 'foobar()' -r2c ~/work/lang-go/ now works without any exceptions or noice on stdout * Add meetup invitation link (#100) * Update README.md (#93) * better handling of invalid keys (#94) * better handling of invalid keys * use validate patterns flag in sgrep (#102) * use validate patterns flag in sgrep * Update README.md (#107) * clean up paths, skip hidden dirs, and use r2c as the default config (#103) * clean up paths, skip hidden dirs, and use r2c as the default config * remove debugging try * add generate config arg * only replace if base path is given * blacken * change to kwarg * ignore when folder doesn't exist (#108) ebf * fix bad merge * Update README.md (#112) * Create sgrep-lint.yml (#111) * Create sgrep-lint.yml * Update sgrep-lint.yml * normal output (#113) * actually write and create file given with -o (#114) * use -o instead of | tee (#117) * use -o instead of | tee * remove newline * better runs with -e (#118) * Update sgrep-lint.yml (#119) * Regression tests showing the absence of FakeTok exn. (#121) This shows that issue #104 and #106 currently work with the develop branch. Test plan: file tests included, make test * Newline has a meaning in Go which automatically inserts semicolons with (#122) certain newlines. In sgrep we do not want those extra semicolons hence this diff. Test plan: make test * ask user to report sgrep core errors (#125) * ask user to report sgrep core errors * add pre-commit * trying out pre-commit * * tests/java/misc_assert.java: regression test (#127) This fixes issue #123 * make sgrep a pre-commit hook (#129) * add integrations.md (#130) * Update .pre-commit-hooks.yaml (#131) * Update .pre-commit-hooks.yaml * fmt * remove no-commit-to-branch breaks CI (#132) * Update sgrep-lint.yml (#133) * Update sgrep-lint.yml * add sgrep to our pre-commit hook (#134) * add sgrep to our pre-commit hook * This should fix issue #106 (#137) Test plan: in bento-core/ repo git checkout lomalley/sgrep-test ~/sgrep/.../main_sgrep.exe -verbose -rules_file .sgrep.yml bento/ no exn anymore * Bugfix/finding config (#135) * fix bug where config not found if path contains .. * --exclude-tests option, friendlier error messages * black * Fix consecutive statements matching (#141) * * lib/sgrep_generic.ml: do not stop at first match, behave like sgrep_lint_generic and try to find all matches. This avoid disrespancies between the behavior of sgrep -e and sgrep -rules_file. * * lib/generic_vs_generic.ml: add debugging information * * tests/python/misc_stmts1.py: support for multiple statements matching! This show that issue #92 is now fixed (require latest of pfff). * * bin/main_sgrep.ml: small diff just to retrigger CI * update multiline syntax to use the yaml | operator (#140) Co-authored-by: Drew Dennison <dennison@mit.edu> Co-authored-by: Yoann Padioleau <yoann.padioleau@gmail.com> Co-authored-by: Pablo Estrada <pabloest@yahoo.com>
DrewDennison
added a commit
that referenced
this issue
Feb 20, 2020
* rename and add black (#62) * rename and add black * blacken * exclude * Update README.md (#63) * add --error to exit 1 if there are findings (#64) * adapt to latest pfff, after AST generic modifications with the Golang integration (#69) test plan: make test also I've disabled the python regression testing because of hardcoded path to isaac dir * support for nested booleans (#67) Updated docs. Re-use patterns key to mean (group of ANDed expressions). Cleaned up some of the traversal logic Add support for nested boolean expression, e.g.: - pattern-either: - pattern: $X == $X - pattern: $X != $X - patterns: - pattern-inside: | def __init__(...): ... - pattern: self.$X == self.$X * * bin/main_sgrep.ml: use better Lang.files_of_dirs... when can. (#71) Should fix issue #66 Test plan: $ ./_build/default/bin/main_sgrep.exe -lang javascript -e foo tests/js/ * Support matching over decorators/attributes/annotations (#72) This solves issue #29 Test plan: file test included make test * Update python.yml (#75) * Adapt to latest pfff, and allow ... to match optional expression in return (#78) This should fix issue #73 and #52 Test plan: tests included $ make test * conform to r2c output with relative paths (#77) * make paths relative to cwd * cleanup rule id prefixes a bit * update expected test, re-enable (#79) * multilingual (#80) * group by languages for sgrep * add a test for multilingual lint * Update README.md (#82) close #81 * Ie/dockerfile paths (#83) * cleanup paths and re-written check IDs when running in docker * black * prefix stripping * revert Dockerfile * safer relative paths * black * update action names (#84) * Change readme to correct code equivalence (#86) * Better class matching (#85) * * lib/generic_vs_generic.ml: comments * * tests/python/dots_fields.py: support for dots in fields. This is related to issue #50 Test plan: test included. * * bin/main_sgrep.ml: add dumper actions to cli, -dump_pattern and -dump_ast * * lib/generic_vs_generic.ml: misc renaming, more consistent use of m_list__xxx * Dots for inheritance, and less-is-ok for inheritance Test plan: tests included make test * Typed metavar (part 2, sgrep part) for literals. (#88) * Typed metavar (part 2, sgrep part) for literals. This fixes issue #49. Test plan: test file included make test * * lib/typechecking_generic.ml: small diff, just to retrigger CI with latest pfff * Function pattern now match methods (and nested functions) (#89) This fixes issue #65 Test plan: tests included. * New --validate-pattern-stdin command for sgrep, to be used in CI (#97) This fixes issue #6 Test plan: echo '$X == $X' | ./_build/default/bin/main_sgrep.exe -lang python --validate-pattern-stdin pad@yrax:~/github/sgrep$ echo $? 0 pad@yrax:~/github/sgrep$ echo '$X == ' | ./_build/default/bin/main_sgrep.exe -lang python --validate-pattern-stdin pad@yrax:~/github/sgrep$ echo $? 1 * * lib/generic_vs_generic.ml: CEs on recent changes on ast_generic (#98) removal of Nop and additions of a few more expr option test plan: make test still works * tests/go/: adding more tests for sgrep for Go! (#99) * tests/go/: adding more tests for sgrep for Go! This should fix issue #96 Test plan: test files included * * tests/go/: more tests * * tests/go/TODO: a few TODOs * * lib/generic_vs_generic.ml: CEs on CaseEqualExpr * * tests/go/: remove optional ';' Test plan: make test also pad@yrax:~/github/sgrep$ ./_build/default/bin/main_sgrep.exe -lang go -e 'foobar()' -r2c ~/work/lang-go/ now works without any exceptions or noice on stdout * Add meetup invitation link (#100) * Update README.md (#93) * better handling of invalid keys (#94) * better handling of invalid keys * use validate patterns flag in sgrep (#102) * use validate patterns flag in sgrep * Update README.md (#107) * clean up paths, skip hidden dirs, and use r2c as the default config (#103) * clean up paths, skip hidden dirs, and use r2c as the default config * remove debugging try * add generate config arg * only replace if base path is given * blacken * change to kwarg * ignore when folder doesn't exist (#108) ebf * fix bad merge * Update README.md (#112) * Create sgrep-lint.yml (#111) * Create sgrep-lint.yml * Update sgrep-lint.yml * normal output (#113) * actually write and create file given with -o (#114) * use -o instead of | tee (#117) * use -o instead of | tee * remove newline * better runs with -e (#118) * Update sgrep-lint.yml (#119) * Regression tests showing the absence of FakeTok exn. (#121) This shows that issue #104 and #106 currently work with the develop branch. Test plan: file tests included, make test * Newline has a meaning in Go which automatically inserts semicolons with (#122) certain newlines. In sgrep we do not want those extra semicolons hence this diff. Test plan: make test * ask user to report sgrep core errors (#125) * ask user to report sgrep core errors * add pre-commit * trying out pre-commit * * tests/java/misc_assert.java: regression test (#127) This fixes issue #123 * make sgrep a pre-commit hook (#129) * add integrations.md (#130) * Update .pre-commit-hooks.yaml (#131) * Update .pre-commit-hooks.yaml * fmt * remove no-commit-to-branch breaks CI (#132) * Update sgrep-lint.yml (#133) * Update sgrep-lint.yml * add sgrep to our pre-commit hook (#134) * add sgrep to our pre-commit hook * This should fix issue #106 (#137) Test plan: in bento-core/ repo git checkout lomalley/sgrep-test ~/sgrep/.../main_sgrep.exe -verbose -rules_file .sgrep.yml bento/ no exn anymore * Bugfix/finding config (#135) * fix bug where config not found if path contains .. * --exclude-tests option, friendlier error messages * black * Fix consecutive statements matching (#141) * * lib/sgrep_generic.ml: do not stop at first match, behave like sgrep_lint_generic and try to find all matches. This avoid disrespancies between the behavior of sgrep -e and sgrep -rules_file. * * lib/generic_vs_generic.ml: add debugging information * * tests/python/misc_stmts1.py: support for multiple statements matching! This show that issue #92 is now fixed (require latest of pfff). * * bin/main_sgrep.ml: small diff just to retrigger CI * update multiline syntax to use the yaml | operator (#140) * Update sgrep.py * report parse errors and fail on them with --strict (closes #142) (#151) * report parse errors and fail on them with --strict (closes #142) * another YAML parse clarification, closes #143 and adds a test (#152) * another YAML parse clarification, closes #143 and adds a test * Update README.md (#150) * Update README.md Co-Authored-By: Pablo Estrada <pablo@r2c.dev> * * lib/generic_vs_generic.ml: adapt to changes in ast_generic.ml after (#158) php_to_generic.ml introduced a few new constructs in ast_generic.ml * Normalize some formatting / capitalization in docs/patterns.md * add a flag to skip pattern validation, which takes 0.05 seconds per rule (#156) * add a flag to skip pattern validation, which takes 0.05 seconds per rule * clean up builds and try to make work for forks * remove on push * use consistant on block * fix #155 (#167) * update to latest pfff * * lib/match_result.ml: better failure message when NoTokenLocation * Test files for issue #153 (#168) * test file for issue #157 (#169) * Add many new tests for ... in more places for Go, Js, and Java. (#170) This show issue #160 is fixed. * More prominent link to sgrep-rules repo (#165) Co-authored-by: Yoann Padioleau <yoann.padioleau@gmail.com> Co-authored-by: Isaac Evans <409041+ievans@users.noreply.github.com> Co-authored-by: Pablo Estrada <pabloest@yahoo.com>
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
should match
os.remove($X: float)->os.remove(1.0)os.remove($X: str)->os.remove("1.0")Etc., for the python primitive types seen as immediates
The text was updated successfully, but these errors were encountered: