Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* add integrations.md (#130) * Update .pre-commit-hooks.yaml (#131) * Update .pre-commit-hooks.yaml * fmt * remove no-commit-to-branch breaks CI (#132) * Update sgrep-lint.yml (#133) * Update sgrep-lint.yml * add sgrep to our pre-commit hook (#134) * add sgrep to our pre-commit hook * This should fix issue #106 (#137) Test plan: in bento-core/ repo git checkout lomalley/sgrep-test ~/sgrep/.../main_sgrep.exe -verbose -rules_file .sgrep.yml bento/ no exn anymore * Bugfix/finding config (#135) * fix bug where config not found if path contains .. * --exclude-tests option, friendlier error messages * black * Fix consecutive statements matching (#141) * * lib/sgrep_generic.ml: do not stop at first match, behave like sgrep_lint_generic and try to find all matches. This avoid disrespancies between the behavior of sgrep -e and sgrep -rules_file. * * lib/generic_vs_generic.ml: add debugging information * * tests/python/misc_stmts1.py: support for multiple statements matching! This show that issue #92 is now fixed (require latest of pfff). * * bin/main_sgrep.ml: small diff just to retrigger CI * update multiline syntax to use the yaml | operator (#140) * Update sgrep.py * report parse errors and fail on them with --strict (closes #142) (#151) * report parse errors and fail on them with --strict (closes #142) * another YAML parse clarification, closes #143 and adds a test (#152) * another YAML parse clarification, closes #143 and adds a test * Update README.md (#150) * Update README.md Co-Authored-By: Pablo Estrada <pablo@r2c.dev> * * lib/generic_vs_generic.ml: adapt to changes in ast_generic.ml after (#158) php_to_generic.ml introduced a few new constructs in ast_generic.ml * Normalize some formatting / capitalization in docs/patterns.md * add a flag to skip pattern validation, which takes 0.05 seconds per rule (#156) * add a flag to skip pattern validation, which takes 0.05 seconds per rule * clean up builds and try to make work for forks * remove on push * use consistant on block * fix #155 (#167) * update to latest pfff * * lib/match_result.ml: better failure message when NoTokenLocation * Test files for issue #153 (#168) * test file for issue #157 (#169) * Add many new tests for ... in more places for Go, Js, and Java. (#170) This show issue #160 is fixed. * More prominent link to sgrep-rules repo (#165) * * lib/generic_vs_generic.ml: CEs latest changes in pfff on AST generic (#181) * * tests/js/dots_params.js: support! (#183) Also this should show issue #178 is now fixed * ignore hidden files not just hidden dirs * a few fixes (#189) * * tests/go/dots_params.go: support! * * tests/go/misc_import.go: test file for issue #185 * * tests/python/misc_dots_stmts.py: test files for issue #186 * * tests/go/equivalence_naming_import.go: test file! (#190) show that it works for go for issue #162 * search full path not just directories * add a command to fail on code parse errors (#194) * add a command to fail on code parse errors * black * add config=r2c-develop * move docs to subfolder (#202) * move docs to subfolder Co-Authored-By: Pablo Estrada <pablo@r2c.dev> * revert moving the files to docs folder because it breaks Github convention and UI * revert moving the files to docs folder because it breaks Github convention and UI * highlight and color output matched range (#203) * colorama output * pattern-where-python (#198) * where-python pattern * complete port to SRange * flags.dangerously_allow_arbitrary_code_execution_from_rules * switich to a class for rule expression * avoid validating patterns that don't need to be sent to sgrep * clearer error messages for invalid YAML files (#215) * better error messages and tests, close issue #210 Co-authored-by: Clint Gibler <clintgibler@users.noreply.github.com> * clean up color formatter, better output for -e case (#216) * clean up formatter, better output for -e case * remove python as default language for patterns (#204) * remove python as default * sgrep main change default to unset * make readme explicit about language python * Fix matching parameter and parameter use. (#218) Description: the generic AST has the notion of an identifier (a string) and a name (a possibly qualified string). A parameter is always an identifier, but when it is used, it is converted in a name, which was preventing a equal match. This diff fixes that by adding the parameter as a Name in the environment, so it can be later matched against its use. This will fix issue #201 Test plan: test file included. * Test files for issue #195 (#219) Test plan: with latest pfff, make test now work. This show that issue #195 is fixed. * add tests for ... operator with go statements (#193) * add tests for ... operator with go statements * nit * newline example * Dumb modification, but now uses latest pfff that fixes the issue. Test plan: make test now work! Co-authored-by: Yoann Padioleau <yoann.padioleau@gmail.com> * Add sgrep.live to readme * refactor sgrep py (#221) * refactor sgrep.py * python workflow not relevant * no more requirements installation * cleanup imports, move config resolution to separate file (#222) Co-authored-by: Drew Dennison <dennison@mit.edu> * add extra paths to check and also don't break just print out a helpful warning message * update to use Isaac's suggestions * improve parsing error for invalid types in valid yaml * Update sgrep_lint/evaluation.py Co-Authored-By: Isaac Evans <409041+ievans@users.noreply.github.com> * bad yaml, not dict * Update evaluation.py * remove -r2c from ocaml sgrep core * * lib/generic_vs_generic.ml: collateral evolutions after the Name to Id vs IdQualified split in pfff/.../ast_generic.ml * Basic constant propagation for constants! Boom. Test plan: test file included * * tests/js/equivalence_constant_propagation.js: js example! * Add less_inheritance test for java, js, go (#237) * Add less_inheritance test for java, js, go * move tests to .TODO * address @aryx comments * Added link to sgrep.live Added it above the first paragraph as well as in the sgrep description so the word 'sgrep' links to sgrep.live * change the shape of evaluate expression to take a single expression (#233) * change the shape of evaluate expression to take a single expression * Update README.md Close issue #211 * Update README.md * * tests/GENERIC/equivalence_constant_propagation.sgrep: ulzii comment, better test case * fix integration test * update expected template too * update expected template too * Adding more js class tests * use ellipsis in class body * clean up generic * build sgrep lint separately (#254) * build sgrep lint separately * Ie/bugfix evaluation (#257) * eagerly intersect ranges for ANDed children (fixes bugs with where-python) * move test.py into sgrep lint (#256) * move test.py into sgrep lint * Add ellipsis into java class/enum/interface body * Test examples showing name resolution for locals is now working for Go (#253) This generalized the fix for issue #195 and should fix issue #243. Test plan: make test * Feature/generic import matching (#249) With semgrep/pfff#44, adds support for the following: code: from foo.bar import a1, a2 pattern: import foo.bar.a2 We use the conversion to generic_ast to normalize code: from foo.bar import a1, a2 into: from foo.bar import a1 from foo.bar import a2 * add golang multivar test (#224) * add three line ellipsis tests * even more minimal test case * move sgrep to subdirectory, pin pfff as git submodule (#258) * move sgrep to subdirectory, pin pfff as git submodule * update pfff to point at correct branch * update submodule docs * update pin * update pin * Update README with Sgrep language by feature table * better column names * Deep expression matching (#264) * Move tests/* test files that were left behind after the BIG move. Also fixed some merge/BIG-move issues where some renamed file persisted. * * sgrep/lib/generic_vs_generic.mli: new file! interfaces are useful, they also help as a side effect discover dead code * Experimemtal deep expression matching! This should help fix issue #171 and issue #255. I'm not sure I want deep expression matching by default without any additional syntax for conditionals, but this has been just requested so we can try and see if we get regressions or false positives. Test plan: test files included! * * sgrep_lint/tests/run-lint-tests.sh: disable sgrep-rules regression testing. It currently fails, and you can't run it twice locally. * * pfff: woohoo! it works! pin new version * make docker builds work (#265) * switch to https URL * Remove less-is-more featurre * remove -r2c and update paths in pre-commit * remove unused variable * update docker build * update docker-build name * build sgrep-lint in gh * trying to build sgrep-lint in GH * sudo * checkout * remove build-lint from circle * trying to build sgrep in an action * add runs-on * take 2 * take 3 * take 4 * take 5 * take 6 * take 7 * Dump ext of language * verbage * take 8 * take 9 * take 10 * try testing * hack to try to get tests working * try again * doh submodule * add path * remove all but super build and remove circle * add names to steps * mess with export * update build files * update build files * update to use https instead of ssh * update pfff * update builds to run on PRs we get from external forks * use just pull request to avoid the double builds * Update README.md * build on push to master and develop * Update README.md * use 'replace' strategy to fix unicode decode errors when reading sgrep output * allow test suite to ignore false positives with #todook * fix pfff pin * Test complex dataflow * Update README.md (#277) * Update README.md Co-authored-by: Isaac Evans <ine@mit.edu> * address renaming * Fix mypy not using strict options mypy by default does not run stricter checks. This turns on some more of the strict mypy options and fixes the codebase to be compliant with the new stricter settings. * fixup! Fix mypy not using strict options * fixup! Fix mypy not using strict options * Deep stmt matching! This should fix issue #229. It's very hacky but it works, and does not seem too slow. Test plan: Test files included * Test files showing support for import declaration pattern in sgrep. (#283) This fixes issue #250 Test plan: test files included cd sgrep; make test * Flexible and correct import matching (#284) * Flexible and correct import matching This fixes the import matching improvements introduced by Drew and Isaac by doing the flexible matching only if the pattern is simple. If the pattern is complex, for example searching for the use of 'import *' in Js or Python, it should not match any kind of import. This fixes issue #197 Test plan: Many tests included. * last fixes * Test files showing you can now use multiple stmts patterns in Go (#286) This fixes issue #281 Test plan: test file included * Test files showing sgrep can now parse lambda patterns (#288) This show issue #226 is now fixed Test plan: test files included * Test files showing issue #267 is fixed (#290) Test plan: test file included * update sgrep docker command to use env var for windows support * enable regression test suite * update pfff pin * reports stats on number of files processed, remove error filtering in sgrep-core * black * annotation * remove unused flags * fix bug * update tests to include error key * dump json (#303) * dump to json is working * add support to sgrep lint for dumping expressions and files * put to stdout not stderr * warn * annotation Co-authored-by: Isaac Evans <ine@mit.edu> * Remove useless return () (#304) Test plan: make test * Support for matching fields in any order Thx to Isaac for a first draft. #301 This should fix issue #298 and #241 Test plan: test files included. * misc * remove unused variable * fix tests * First draft of XML (JSX/XHP) matching (#307) * First draft of XML (JSX/XHP) matching This should fix issue #282 Test plan: test files included * It's ok to not have a Xml body in the pattern, less is ok Test plan: test files included * Match XML attribute in any order * Support for metavariables for attributes Test plan: test file included * update sgrep.opam and simplify builds * add back in the test.bc * Fix a typo in the readme file (#318) * remove hidden files not just folders (#310) * * sgrep/tests/python/less_typehint.py: support (#322) Test plan: test files included * Update to latest pfff (#331) Pfff requires now dypgen and menhir * Constant propagation and evaluation for strings (#332) This should fix issue #324 Test plan: test files included * Support 'var' matching also 'let' and 'const' (#333) This is especially useful for JS (maybe too JS-specific) Test plan: test files included * Fix python interpolated strs * update to handle multiple arguments * add colon for error * handle empty list * add more test * add the failing go test * apply only when all are strings * add false positive test * Update docker.yml * Update docker.yml * Update config.md (#321) * Update docs/config.md Co-Authored-By: Pablo Estrada <pablo@r2c.dev> * Add --exclude to sgrep Note that this just post-processes the output and I would like to pass this down to the sgrep core to save parsing time etc. * fix tests namespace * Call Naming_ast.resolve for constant propagation in every mode (#339) Description: I was calling Naming_ast.resolve only when calling sgrep with -e, and not via -rules_file. This is quick fix. The right fix is to factorize more code between sgrep_generic and sgrep_lint_generic to stop having slightly different behaviors between -e and -rules_file. Test plan: pad@yrax:~/github/sgrep/sgrep$ cat /tmp/file.yaml rules: - id: test-constants pattern: foo("...") message: "foo!" languages: [js] severity: ERROR pad@yrax:~/github/sgrep/sgrep$ yy -lang js -rules_file /tmp/file.yaml /tmp/equivalence_constant_propagation3.js + /home/pad/github/sgrep/sgrep/_build/default/bin/main_sgrep.exe -lang js -lang js -rules_file /tmp/file.yaml /tmp/equivalence_constant_propagation3.js { "matches": [ { "check_id": "test-constants", "path": "/tmp/equivalence_constant_propagation3.js", "start": { "line": 1, "col": 1, "offset": 0 }, "end": { "line": 1, "col": 14, "offset": 13 }, "extra": { "message": "foo!", "metavars": {} } }, { "check_id": "test-constants", "path": "/tmp/equivalence_constant_propagation3.js", "start": { "line": 4, "col": 1, "offset": 34 }, "end": { "line": 4, "col": 17, "offset": 50 }, "extra": { "message": "foo!", "metavars": {} } }, { "check_id": "test-constants", "path": "/tmp/equivalence_constant_propagation3.js", "start": { "line": 6, "col": 1, "offset": 53 }, "end": { "line": 6, "col": 16, "offset": 68 }, "extra": { "message": "foo!", "metavars": {} } } ], "errors": [], "stats": { "okfiles": 1, "errorfiles": 0 } } * Update bug_report.md * Add action to automatically build osx and ubuntu binary (#292) New pull requests will have a github action that builds binaries for OSX and ubuntu18.04 * Deep stmt for Try (#342) * * sgrep/bin/main_sgrep.ml: This fix issue #341, a stupid mistake (type-checking does not work when you use the same type for everything ...) Test plan: + /home/pad/github/sgrep/sgrep/_build/default/bin/main_sgrep.exe -lang py -f tests/python/deep_try.sgrep tests/python/deep_try.py does not raise anymore just pb exn. same for + /home/pad/github/sgrep/sgrep/_build/default/bin/main_sgrep.exe -lang py -e foo(...) tests/python/ * Support for deep stmt matching for Try blocks. Description: I forgot to recurse correctly on the Try case ... This fix issue #334 Test plan: test file included * * .github/ISSUE_TEMPLATE/bug_report.md: please pre-commit * make python output encoding be utf8 * Test files slowing class regression fix (#346) * Test files slowing class regression fix This show issue #260 is fixed Test plan: test file included * * pfff: update to latest pfff fixing sgrep-rules regressions * Resolving alias in attributes/decorators (#350) This fixes issue #294 Test plan: test files included * Add version string and command to print it out (#354) * Build on ubuntu:16.04 (#343) * Fix: sgrep-lint printing out number of config files/rules to debug (#355) sgrep-lint was printing out information that was not a finding during the run. Moved to print only when verbose is marked true. * Fix: don't print newline when there are no findings (#357) * Bump sgrep-lint to 0.4.9b5 * [Python] f-string with const propagation (#347) * F-string with const propagation * clarify test * complex f-string test * fix parsing issue * make ... match 0 or more * verbose comment * add v1 of autofix to sgrep (#358) * add v1 of autofix to sgrep * refactor and clean up the code a bit (#359) * Factorize sgrep_generic and sgrep_lint_generic (#360) * * sgrep/lib/sgrep_lint_generic.ml: delete! merged and factorized with sgrep_generic.ml * Finish factorization of sgrep_generic.ml and sgrep_lint_generic.ml Test plan: make test sgrep -e and -f still work. make install cd ../sgrep_lint make test * * sgrep/bin/main_sgrep.ml: remove PHP specific stuff * * sgrep/bin/main_sgrep.ml: cleanup * * sgrep/bin/main_sgrep.ml: more cleanup * User-defined equivalences in sgrep-core $X + $Y <==> $Y + $X (#362) * * sgrep/lib/equivalence.ml: start! type and parsing for user-defined equivalences This is one option to fix issue #329 Test plan: + /home/pad/github/sgrep/sgrep/_build/default/bin/main_sgrep.exe -lang py -dump_equivalences data/basic_equivalences.yml [("plus-commutative", Tag3 (Tag10 (Tag9 ([Tag1 (0); (("+", 3, 1, 3, "/tmp/tmp-7573-2815ef.py"))]), [(Tag7 (["$X"; (("$X", 0, 1, 0, "/tmp/tmp-7573-2815ef.py"))], ((0), (0), (0)))); (Tag7 (["$Y"; (("$Y", 5, 1, 5, "/tmp/tmp-7573-2815ef.py"))], ((0), (0), (0))))])), Tag3 (Tag10 (Tag9 ([Tag1 (0); (("+", 4, 1, 4, "/tmp/tmp-7 ... * * sgrep/bin/main_sgrep.ml: passing the equivalences to the engine * * sgrep/tests/EQUIV/test.py: test file! * Big step towards user-defined equivalences. In this diff, we apply a list of equivalences to the sgrep pattern by visiting the pattern, looking for a match, and substituting the matched pattern with the right hand part of the equivalence. This helps fix issue #329 Test plan: home/pad/github/sgrep/sgrep/_build/default/bin/main_sgrep.exe -lang py -equivalences data/basic_equivalences.yml -e 1+2 tests/EQUIV/ /home/pad/github/sgrep/sgrep/tests/EQUIV/test.py:5 return 2+1 The next diff needs to add DisjExpr to AST generic so we look for the original sgrep pattern Or the equivalent one. * Using DisjExpr and DisjStmt to apply code equivalances to the sgrep pattern This finally fixes issue #329 (at least one way to fix it) Test plan: pad@yrax:~/github/sgrep/sgrep$ cat data/basic_equivalences.yml equivalences: # Basic equivalences - id: plus-commutative pattern: $X + $Y <==> $Y + $X languages: [python, javascript, c, go, java] - id: eq-to-no-eq pattern: $X == $X ==> $X != $X languages: [python, javascript, c, go, java] pad@yrax:~/github/sgrep/sgrep$ yy -equivalences data/basic_equivalences.yml -e '1+2' tests/EQUIV/ + /home/pad/github/sgrep/sgrep/_build/default/bin/main_sgrep.exe -lang py -equivalences data/basic_equivalences.yml -e 1+2 tests/EQUIV/ /home/pad/github/sgrep/sgrep/tests/EQUIV/test.py:2 return 1+2 /home/pad/github/sgrep/sgrep/tests/EQUIV/test.py:5 return 2+1 make test also still works * Use latest pfff so destructuring assignments are not transpiled and work (#369) * Use latest pfff so destructuring assignments are not transpiled and work this fixes issue #337 Test plan: test file included * Use new destructuring for unordered matching This shows issue #338 is now fixed Test plan: test file included * Support for metavar on keyword argument. (#370) This fixes issue #367 Test plan: test file included * Factorize code in engine for managing ... in types, args, and extend it to arrays (#373) * * sgrep/lib/matching_generic.ml: new file, move stuff from generic_vs_generic * * sgrep/lib/matching_generic.ml: factorize code with m_list_with_dots generic matcher * * sgrep/lib/matching_generic.ml: bugfix, fix regression * Allow ... in arrays. This fix issue #361, and now use the generic helper introuced in the previous diff. Test plan: test files included * Create an initial brew tap (#376) This creates a functional (but pretty janky) brew tap based on the current state of affairs. Lots more downstream work to knock out: - Automatically build binaries on tags & add to release - Fixup the tap to use formulae dependencies instead of copying all these libs - fix the linker path to actually look for libraries in the right place * Update README.md * Update README.md * add coreutils as a dependency for tap (#378) * Create an initial brew tap This creates a functional (but pretty janky) brew tap based on the current state of affairs. Lots more downstream work to knock out: - Automatically build binaries on tags & add to release - Fixup the tap to use formulae dependencies instead of copying all these libs - fix the linker path to actually look for libraries in the right place * Update sgrep-r2c.rb * Update sgrep-r2c.rb * Update HomebrewFormula/sgrep-r2c.rb Co-Authored-By: Isaac Evans <409041+ievans@users.noreply.github.com> Co-authored-by: Isaac Evans <409041+ievans@users.noreply.github.com> * Update README.md * Correct subprocess example in README (#379) * Correct subprocess example in README subprocess.open doesn't exist, use Popen * Update subprocess at the bottom too * Build artifacts on merge (#366) * Release 0.4.9 (#387) * Wrap error entries to be compatible with r2c output (#388) * Wrap error entries to be compatible with r2c output Co-authored-by: Drew Dennison <drew@returntocorp.com> Co-authored-by: Yoann Padioleau <pad@returntocorp.com> Co-authored-by: Isaac Evans <409041+ievans@users.noreply.github.com> Co-authored-by: Clint Gibler <clint AT somewhere dot com> Co-authored-by: Pablo Estrada <pablo@r2c.dev> Co-authored-by: Clint Gibler <clintgibler@users.noreply.github.com> Co-authored-by: Yoann Padioleau <yoann.padioleau@gmail.com> Co-authored-by: Drew Dennison <dennison@mit.edu> Co-authored-by: Isaac Evans <ine@mit.edu> Co-authored-by: Ulzii <ulzii@returntocorp.com> Co-authored-by: Dannibal <2009327+dan5382@users.noreply.github.com> Co-authored-by: Sylvestre Ledru <sledru@mozilla.com> Co-authored-by: Russell Cohen <russell.r.cohen@gmail.com> Co-authored-by: mschwager <mschwager@users.noreply.github.com>
- Loading branch information
14 people
committed
Apr 6, 2020
1 parent
9a07f88
commit 6c55267