Manticore 0.2.0 #1043
Merged
Manticore 0.2.0 #1043
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yan
added a commit
that referenced
this pull request
Sep 6, 2018
* Python 3; optimization / stylization pass * cleanup list() from automation tools * style; use dict comprehensions * style; use set literals * Experiment reporting the finding at a JUMPI (#949) * Experiment reporting the finding at a JUMPI * Fix taint. Detect returned overflowded data * Fix contract names in benchmark * Move default plugin registration * merge * rm make_evm (#978) * Yolo dev evm fix address concretization (#1002) * DAO detector + bugfixes * The actual benchmark tests * The actual benchmark tests * CC * Experiment reporting the finding at a JUMPI * Fix taint. Detect returned overflowded data * DAO -> Reentrancy * DAO -> reentrancy, C -> Benchmark * DAO -> reentrancy, C -> Benchmark * Allow function names to have numbers * Fix contract names in benchmark * Fix contract names in benchmark * Move default plugin registration * Better regexp * Fix minimal_bytecode example * Fix Array Slice and test * add tests * correct other bug * implement bytesM * BROKEN partial progress * need bytearray here * rm cmt * add basic tests for bytesM and bytes symbolic * correct bytes symbolic test * Refactor, clean bytesM handling * Add initial symbolic 'bytes' handling * refactor tests * Unify symbolic/concrete bytes handling in bytesM/bytes * Rm import * Rm debug assert * cc * Visitor/migrate/simplify fixes to make the seth refactor pass * Fix concolic? * Fix concolic? * CC * bytesM fix * Fix address and caller concretization on symb tx * Fix account policy refactor * CC * cleanup * numbers.Integral * super() * remove/update deprecated * Report test coverage to CodeClimate (#1004) This PR enables the reporting of test coverage of all the test jobs (`eth` and `tests`) to CodeClimate. This uses S3 to temporarily store results between jobs and later upload them to CC. Fixes #1000 * codeclimate * codeclimate - bump similar-code thresh; false positive * Fix CC coverage (#1007) This fix does two things: 1. Ignores non-manticore files from the coverage report to limit what can fail. 2. Changes how travis runs s3 sync on completion. (Fixes #1006) * Report test coverage to CodeClimate (#1004) This PR enables the reporting of test coverage of all the test jobs (`eth` and `tests`) to CodeClimate. This uses S3 to temporarily store results between jobs and later upload them to CC. Fixes #1000 * Fix CC coverage (#1007) This fix does two things: 1. Ignores non-manticore files from the coverage report to limit what can fail. 2. Changes how travis runs s3 sync on completion. (Fixes #1006) * re-enable and fix eth regression 808 (#1011) * cleanup examples (#1010) * resolves #1008 (#1014) * Addresses performance issues; * reimplement caching for `arithmetic_simplifier` and `constant_folder` * optimize `ArithmeticSimplifier.visit_ArraySelect` * File mode fix - resolves #1018 * Fixes closed file serialization (#955) Fixes #954 * Add unit test for 954 (#1022) * Change how we query for version (#1023) Fixes #1021 This also should decrease how many times we invoke z3. (The instance used to query version should stick around) * Use capstone 3.0.5 and no longer rc2 (#1026) * binja cleanup * fixes docker - resolves #991 * Dev yolo retvalthing (#1001) * DAO detector + bugfixes * The actual benchmark tests * The actual benchmark tests * CC * Experiment reporting the finding at a JUMPI * Fix taint. Detect returned overflowded data * DAO -> Reentrancy * DAO -> reentrancy, C -> Benchmark * DAO -> reentrancy, C -> Benchmark * Allow function names to have numbers * Fix contract names in benchmark * Fix contract names in benchmark * Move default plugin registration * Better regexp * Fix minimal_bytecode example * Fix Array Slice and test * add tests * correct other bug * implement bytesM * BROKEN partial progress * need bytearray here * rm cmt * add basic tests for bytesM and bytes symbolic * correct bytes symbolic test * Refactor, clean bytesM handling * Add initial symbolic 'bytes' handling * refactor tests * Unify symbolic/concrete bytes handling in bytesM/bytes * Rm import * Rm debug assert * cc * Visitor/migrate/simplify fixes to make the seth refactor pass * Fix concolic? * Fix concolic? * CC * bytesM fix * Fix address and caller concretization on symb tx * Fix/refactor symbolic address/caller concretization * Fix caller concretization * Fix expression visiting * Fix account policy refactor * Accept numbers in function names abitypes * Simplify installation instructions to recommend install manticore only for the current user * Run some tests in parallel (#970) This PR splits the current test runner into three environments: 1. Linux examples 2. Ethereum tests 3. Remaining tests to faster complete each testing run. Ethereum tests include a number of integration tests that execute scripts to completion, which takes a while. We run them concurrently with other tests to save on execution time. The split is done by naming Ethereum tests differently (`eth_*.py` vs `test_*.py`) and updating what pattern unittest's `discover` uses. This change also updates the installation script and chooses to forego installing Keystone for EVM tests as it takes a while, and it adds a `setup.cfg` config file so that Nose finds the eth tests as well by default. * Be less verbose when testing * Fix slicing wrongly reference to proxyArray. Fix #912 * Only export human/external tx in the testcase (#972) * Make ManticoreEVM.make_symbolic_value size adjustable (#974) * Make size adjustable * Default to 256 * Dev evm yolo fix gas (#975) * Fix gas stipend on CALL and check dao * Add order dependence 1 * Going linter. Report/Detect that thing when code does not check returned value * cleaner example of fail * Update retval_crazy.sol * new solc for travis * CC * Remove duplicated ReentrancyDetector * POrt to py3 * POrt to py3 * P0rt to py3 * CC * Tests doc * CC * review changes * remove stray comment * missed one * resolves #992 (#1033) * resolves #992 * fix sys_write logger output (#1024) * fix sys_write logger output - resolves #1020 * write/writev/read fixes * openat((int32)dirfd, ...) resolves #940, syscall logging * disable E701, interferes with PEP484/526 * readme Ethereum update issue #1003 (#1034) * readme ethereum update issue #1003 * simplify * Update README.md * ignore resource warnings (e.g. unclosed files) (#1038) * Test manticore on MacOS (#1032) * Test manticore on MacOS like test_binaries.py for path to binary to test * MacOS compatibility achieved Replacement of /bin/ls in tests Use of basename in test_load_maps * Fix gast (#1039) * Readme updates (#1037) * add some more heft to the Ethereum section * no longer needed * Integrate requirements into installation * Update README.md * Update README.md * Update README.md * Update README.md * Duplicate commands for docker quick start * Rm --process-dependency-links note, moved into the faq on the wiki * Small tweaks * pedantic formatting * Emphasize new python requirement (#1041) * Emphasize new python requirement * Consistent formatting * Port remaining examples to py3 (#1042) * port use_def * port some scripts, cleanup * ported `scripts/gdb.py` - untested * misc * Manticore 0.2.0 (#1043) * Bump version * Initial changelog changes * Bump version in setup.py * Add skeleton and externals * Fill in 0.2.0 readme * Updates * Add logo to readme (#1046) * add logo to README * Fix missing profiling data (#1057) * fix missing profiling data - resolves #982 * unit test * Code cleanup and coverage (#1035) * dead code elimination, __init__ cleanup * `binary.Elf` bugfix, add `binary` package tests * Serialization cleanup (#1048) * refactor serialization / recursion limit handling * evm: aggressively check & migrate expressions into current ConstraintSet in case they are global/external (#1009) * Be mega forgiving on global expression usage - EVM * Refactor new_bitvector api * Fix neW_bool * CC * rename avoid_collisions collision * rename avoid_collisions collision * migrate on state.constraint too.. * Migration bugfixes * CC bugfixes * invalid assert removed * move rep code to method * reviewing the codes * CC * Change variable names * typo * Some mini docstrings and a unittest * Add migration integration testion * Keep fuzz-refactoring it * CC * Bugfixfixfixfix * CC * re refactor mig algorithm * better cleaner stronger. (reviewing) * CC * Small refactor and Fix strange strcmp test. * CC * re re refactor for readability * CC * rev * forgoten var * Fix for #1008 (#1063) * Fix for #1008 * add test for funcall output * Implements support for function overloading in ethereum (#1049) * implements `signature` kwarg for overloaded functions - resolves #810 * Fix typo mistake in multi-million word (#1073) * eth: add selfdestruct detector & misc bug fixes (#1068) * Don't keep selfdestruct states alive * Use avoid_collisions=True for internal uses of the .new_ methods * Better err msgs * Output pc in hex * Fix ignored workspace cli flag * hex pc * hex pc one last time * add selfdestruct detector * Add cli support * Add ok selfdestruct test * Add selfdestruct not ok - true positive * Add selfdestruct crazy - true negative * Reorganize plugin/detectors. Add LoopDepthLimiter plugin + cli flag * rename files * add another test * Add initial selfdestruct tests * Move integer overflow detector test into eth_detectors * cc * add missing import * add other missing import * Added --txnoether option to avoid sending ether to contracts (#1078) * added --txnoether option * Improved command line description * eth: add ether leak detector (#1077) * Add initial ether leak detector * Initial test * correct * Add another neg * rm stray print * initial tests refactoring + ether leak tests * finding name * initial refactor * clean comment * correct this test * update tests * Add fp comment * add other test * remove unnecessary payable function * make LoopDepthLimiter configurable * Use real pc * cc * Add other test * Add cli interface * Create readthedocs.yml (#1085) * Fix rtd (#1086) * test * wrong number * sorry Popen :( * mocking * x * clean * better explain this arcane stuff * don't need io * Improved printing of constructor call with decoded constructor arguments and transaction result (#1080) * added printing of decoded constructor arguments * Fixed test * Add --no-testcases flag (#1083) * Update the README (#1064) <!-- Reviewable:start --> This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/trailofbits/manticore/1064) <!-- Reviewable:end --> * Add detector for plain external call (#1087) * initial refactor etherleak to also do general external call * refactor * Update users * Update tests * Update * Fix tests * Don't use signed operator, check != 0 * Record constraint * Record constraint * Use did_evm_execute So we don't falsely report if the CALL were to fail * Revert "Use did_evm_execute" 96a84f2 * simplify deref logic * clean up derefs; rename backing array * reduce write calls * simplify * add name field * Detection of environmental and potentially manipulable instruction/data (#1096) * unittest * CC * import fix * typo * forgotten test * eth: new/alternative reentrancy detector (#1082) * initial second one * update * polish * correct * Correctly check gas * Record gas constraint and save in finding * simplify logic * Check if destination is a contract * Revert "Check if destination is a contract" 901be37 * better context key * Be lenient with Constants * Add new simpler/less input required reentrancy detector, use in the cli * Fix bad merge import * Fix import * Add final missing import * add an iter() interface to Memory * Add env instruction detector to cli (#1105) * Sha3 rework and performance enhancements (#1031) * DAO detector + bugfixes * The actual benchmark tests * The actual benchmark tests * CC * Experiment reporting the finding at a JUMPI * Fix taint. Detect returned overflowded data * DAO -> Reentrancy * DAO -> reentrancy, C -> Benchmark * DAO -> reentrancy, C -> Benchmark * Allow function names to have numbers * Fix contract names in benchmark * Fix contract names in benchmark * Move default plugin registration * Better regexp * Fix minimal_bytecode example * Fix Array Slice and test * add tests * correct other bug * implement bytesM * BROKEN partial progress * need bytearray here * rm cmt * add basic tests for bytesM and bytes symbolic * correct bytes symbolic test * Refactor, clean bytesM handling * Add initial symbolic 'bytes' handling * refactor tests * Unify symbolic/concrete bytes handling in bytesM/bytes * Rm import * Rm debug assert * cc * Visitor/migrate/simplify fixes to make the seth refactor pass * Fix concolic? * Fix concolic? * CC * bytesM fix * Fix address and caller concretization on symb tx * Fix/refactor symbolic address/caller concretization * Fix caller concretization * Fix expression visiting * Fix account policy refactor * Accept numbers in function names abitypes * Simplify installation instructions to recommend install manticore only for the current user * Run some tests in parallel (#970) This PR splits the current test runner into three environments: 1. Linux examples 2. Ethereum tests 3. Remaining tests to faster complete each testing run. Ethereum tests include a number of integration tests that execute scripts to completion, which takes a while. We run them concurrently with other tests to save on execution time. The split is done by naming Ethereum tests differently (`eth_*.py` vs `test_*.py`) and updating what pattern unittest's `discover` uses. This change also updates the installation script and chooses to forego installing Keystone for EVM tests as it takes a while, and it adds a `setup.cfg` config file so that Nose finds the eth tests as well by default. * Be less verbose when testing * Fix slicing wrongly reference to proxyArray. Fix #912 * Only export human/external tx in the testcase (#972) * Make ManticoreEVM.make_symbolic_value size adjustable (#974) * Make size adjustable * Default to 256 * Dev evm yolo fix gas (#975) * Fix gas stipend on CALL and check dao * Add order dependence 1 * Going linter. Report/Detect that thing when code does not check returned value * cleaner example of fail * Update retval_crazy.sol * new solc for travis * CC * Remove duplicated ReentrancyDetector * POrt to py3 * POrt to py3 * P0rt to py3 * CC * Be mega forgiving on global expression usage - EVM * Tests doc * Refactor new_bitvector api * function id to binary * Fix neW_bool * CC * rename avoid_collisions collision * rename avoid_collisions collision * migrate on state.constraint too.. * Migration bugfixes * CC bugfixes * invalid assert removed * move rep code to method * unittets fixes and CC * CC * Refactor result_ref out in favor of change_last_result() * CC * reviewing the codes * CC * Change variable names * typo * Basic refactors and output enhancements * Some minid docstrings and a unittest * Some mini docstrings and a unittest * Add migration integration testion * Keep fuzz-refactoring it * CC * Bugfixfixfixfix * CC * re refactor mig algorithm * better cleaner stronger. (reviewing) * CC * Small refactor and Fix strange strcmp test. * CC * funtion selector abinary * bugfix.. waiting for migreation PR * convenient tx abi parsing func * convenient tx abi parsing func * convenient tx abi parsing func * convenient tx abi parsing func * re re refactor for readability * CC * rev * CC * forgoten var * CC * CC * review * typo * CC * review * Adding single example to sha3 trick when there are not know examples * CC * review * CC * Forgotten rollback * CC * Detect the odd delegatecall instruction (#1108) * DAO detector + bugfixes * The actual benchmark tests * The actual benchmark tests * CC * Experiment reporting the finding at a JUMPI * Fix taint. Detect returned overflowded data * DAO -> Reentrancy * DAO -> reentrancy, C -> Benchmark * DAO -> reentrancy, C -> Benchmark * Allow function names to have numbers * Fix contract names in benchmark * Fix contract names in benchmark * Move default plugin registration * Better regexp * Fix minimal_bytecode example * Fix Array Slice and test * add tests * correct other bug * implement bytesM * BROKEN partial progress * need bytearray here * rm cmt * add basic tests for bytesM and bytes symbolic * correct bytes symbolic test * Refactor, clean bytesM handling * Add initial symbolic 'bytes' handling * refactor tests * Unify symbolic/concrete bytes handling in bytesM/bytes * Rm import * Rm debug assert * cc * Visitor/migrate/simplify fixes to make the seth refactor pass * Fix concolic? * Fix concolic? * CC * bytesM fix * Fix address and caller concretization on symb tx * Fix/refactor symbolic address/caller concretization * Fix caller concretization * Fix expression visiting * Fix account policy refactor * Accept numbers in function names abitypes * Simplify installation instructions to recommend install manticore only for the current user * Run some tests in parallel (#970) This PR splits the current test runner into three environments: 1. Linux examples 2. Ethereum tests 3. Remaining tests to faster complete each testing run. Ethereum tests include a number of integration tests that execute scripts to completion, which takes a while. We run them concurrently with other tests to save on execution time. The split is done by naming Ethereum tests differently (`eth_*.py` vs `test_*.py`) and updating what pattern unittest's `discover` uses. This change also updates the installation script and chooses to forego installing Keystone for EVM tests as it takes a while, and it adds a `setup.cfg` config file so that Nose finds the eth tests as well by default. * Be less verbose when testing * Fix slicing wrongly reference to proxyArray. Fix #912 * Only export human/external tx in the testcase (#972) * Make ManticoreEVM.make_symbolic_value size adjustable (#974) * Make size adjustable * Default to 256 * Dev evm yolo fix gas (#975) * Fix gas stipend on CALL and check dao * Add order dependence 1 * Going linter. Report/Detect that thing when code does not check returned value * cleaner example of fail * Update retval_crazy.sol * new solc for travis * CC * Remove duplicated ReentrancyDetector * POrt to py3 * POrt to py3 * P0rt to py3 * CC * Be mega forgiving on global expression usage - EVM * Tests doc * Refactor new_bitvector api * function id to binary * Fix neW_bool * CC * rename avoid_collisions collision * rename avoid_collisions collision * migrate on state.constraint too.. * Migration bugfixes * CC bugfixes * invalid assert removed * move rep code to method * unittets fixes and CC * CC * Refactor result_ref out in favor of change_last_result() * CC * reviewing the codes * CC * Change variable names * typo * Basic refactors and output enhancements * Some minid docstrings and a unittest * Some mini docstrings and a unittest * Add migration integration testion * Keep fuzz-refactoring it * CC * Bugfixfixfixfix * CC * re refactor mig algorithm * better cleaner stronger. (reviewing) * CC * Small refactor and Fix strange strcmp test. * CC * funtion selector abinary * bugfix.. waiting for migreation PR * convenient tx abi parsing func * convenient tx abi parsing func * convenient tx abi parsing func * convenient tx abi parsing func * re re refactor for readability * CC * rev * CC * forgoten var * CC * CC * Delete duplicated detector * WIP delegatecall. Lot of fixes. Recursion fix. * review * typo * CC * Sha3 random concre example when none. Concretize SIZE/OFFSET more. Initial calldata size management * remove debug print * Add check in constraints.add * review * Adding single example to sha3 trick when there are not know examples * CC * review * CC * about to merge sha3 * cleanups * make gas budget configurable by user * cleanups * cleanups * CC: * CC * Fix typo in tests * Fix import typo * pump some gas * typo in skipping slow/big test * skipping more test to make travis happy * skipping more test to make travis happy * debugging travis like a caveman * dbg * dbg * dbg * undbg * undbg * undbg * undbg * undbg and fixed * undbg and fixed * CC * Updates to README.md relating to Python 3 migration and sudo within virtualenv setup (#1109) * updated README.md to use python3 commands * updated README.md to add sudo to system pip3 install commands Line 186 uses a path to the specific pip3 binary (as per https://stackoverflow.com/questions/41429988/inside-virtual-env-sudo-pip-links-to-the-global-python-pip ). * Manticore 0.2.1 (#1106) * Bump version num * changelog skeleton * Switch to agpl * readme license update * update * gas * Add dc * Date update * Add last minute contributions * Release 0.2.1c (#1111) * Fix the versioning hell (#1112) * Fix version number / release (#1113) * Manticore 0.2.1-berlin (#1114) * Manticore 0.2.1-b (#1115) * manticore 0.2.1.1 (#1116) * start scan_mem refactor * Fix DecodeException message * Only import mapped memory * get a single value, instead of all, when decoding * clean up instruction decoding * simplify scan_mem * Changed instances of Concretice to Concretize (#1118) * Fix typo * rename a field * Attempt decoding with an arraystore simplifier * python3 update to linux example makefile (#1122) Ubuntu 18.04 doesn't alias python to python3 * Update decoding logic * Simplify [skip] importing read-only maps * Fix name serialization with anonmaps
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change is