Merge branch 'master' into ekilmer-dev

* master: (23 commits) Update make_VMTests.py (#1628) Update manticore.py (#1627) Symbolic function strategies. (#1609) Fix for returndatasize (#1612) Improve balance related exploration (#1615) Add __format__ to Account (#1613) Use `mypy` to check the `tests` dir; fix some typing errors (#1610) Add a native example program & test case for #1604 (#1606) Hotfix: add long_description_content_type Manticore 0.3.3 (#1598) Improve WASM Memory Ergonomics (#1589) Make sure std files are not None before assigning peer (#1604) Add Examples badge to CI Badges (#1603) Better iterative optimization for expressions (#1599) Fix Regression(s) (#1577) Add missing `python3-dev` dep to quickstart instructions (#1600) Add license to setup.py (#1597) Streamline State Serialization Interface (#1596) Simplify ci.yml (#1588) Fix gas check (#1587) ...
trailofbits · Mar 12, 2020 · a790f4f · a790f4f
2 parents a9501d7 + 2c774db
commit a790f4f
Show file tree

Hide file tree

Showing 43 changed files with 1,307 additions and 436 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -163,8 +163,8 @@ jobs:
             # but Manticore fails to explore the paths due to the lack of the 0x1f opcode support
             # see https://github.com/trailofbits/manticore/issues/1166
             # if [ "$(ls output/*tx -l | wc -l)" != "41" ]; then
-            if [ "$(ls output/*tx -l | wc -l)" != "13" ]; then
-                echo "Truffle test failed" `ls output/*tx -l | wc -l` "!= 13"
+            if [ "$(ls output/*tx -l | wc -l)" != "34" ]; then
+                echo "Truffle test failed" `ls output/*tx -l | wc -l` "!= 34"
                 return 1
             fi
             echo "Truffle test succeded"
@@ -203,7 +203,7 @@ jobs:
                 run_tests_from_dir $TEST_TYPE
                 RV=$?
 
-          echo "Running truffle test"
+                echo "Running truffle test"
                 install_truffle
                 run_truffle_tests
                 RV=$(($RV + $?))
@@ -224,34 +224,11 @@ jobs:
                 run_tests_from_dir $TEST_TYPE
                 RV=$?
                 ;;
-
             examples)
                 run_examples
                 ;;
-
-            all)
-                echo "Running all tests registered in travis_test.sh: examples, native, ethereum, ethereum_vm, other";
-
-                # Functions should return 0 on success and 1 on failure
-                RV=0
-                run_tests_from_dir native
-                RV=$(($RV + $?))
-                run_tests_from_dir ethereum
-                RV=$(($RV + $?))
-                make_vmtests; run_tests_from_dir ethereum_vm
-                RV=$(($RV + $?))
-                make_wasm_tests; run_tests_from_dir wasm
-                RV=$(($RV + $?))
-                make_wasm_sym_tests; run_tests_from_dir wasm_sym
-                RV=$(($RV + $?))
-                run_tests_from_dir other
-                RV=$(($RV + $?))
-                run_examples
-                RV=$(($RV + $?))
-                ;;
-
             *)
-                echo "Usage: $0 [examples|native|ethereum|ethereum_vm|other|all]"
+                echo "Unknown TEST_TYPE: '$TEST_TYPE'"
                 exit 3;
                 ;;
         esac

diff --git a/.gitignore b/.gitignore
@@ -96,3 +96,6 @@ venvpy27/
 
 # PyCharm files
 .idea/
+
+# mypy cache
+.mypy_cache/
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,33 @@
 # Change Log
 
-## [Unreleased](https://github.com/trailofbits/manticore/compare/0.3.2...HEAD)
+## [Unreleased](https://github.com/trailofbits/manticore/compare/0.3.3...HEAD)
+
+## 0.3.3 - 2020-01
+
+Thanks to our external contributors!
+
+ - [catenacyber](https://github.com/trailofbits/manticore/commits?author=catenacyber)
+
+### Ethereum
+* **[added API]** Flag to only generate alive states when finalizing Manticore [#1554](https://github.com/trailofbits/manticore/pull/1554)
+* Fix gas check [#1587](https://github.com/trailofbits/manticore/pull/1587)
+
+### Native
+* **[added API]** Add post-instruction hooks [#1579](https://github.com/trailofbits/manticore/pull/1579)
+* Fix issue with re-using stdio file descriptors after they'd been closed [#1604](https://github.com/trailofbits/manticore/pull/1604)
+
+### WASM
+* **[added API]** getattr-style calls for WASM functions [#1578](https://github.com/trailofbits/manticore/pull/1578)
+* **[changed API]** Pass state to function calls instead of constraint sets [#1578](https://github.com/trailofbits/manticore/pull/1578)
+* **[added API]** Added read/write helper methods to memory instances [#1589](https://github.com/trailofbits/manticore/pull/1589)
+
+### Other
+* **[added API]** Added streamlined state serialization interface [#1596](https://github.com/trailofbits/manticore/pull/1596)
+* Fixed Z3 version parsing [#1551](https://github.com/trailofbits/manticore/pull/1551)
+* Unique names for ArrayVars [#1552](https://github.com/trailofbits/manticore/pull/1552)
+* Improve pickling and multiprocessing compatibility [#1583](https://github.com/trailofbits/manticore/pull/1583)
+* Fix SMTLib visitor bug that broke the example tests [#1577](https://github.com/trailofbits/manticore/pull/1577)
+* Optimize MinMax SMTLib operations [#1599](https://github.com/trailofbits/manticore/pull/1599)
 
 ## 0.3.2 - 2019-11-11
 

diff --git a/README.md b/README.md
@@ -10,6 +10,7 @@
 [![PyPI version](https://badge.fury.io/py/manticore.svg)](https://badge.fury.io/py/manticore)
 [![Slack Status](https://empireslacking.herokuapp.com/badge.svg)](https://empireslacking.herokuapp.com)
 [![Documentation Status](https://readthedocs.org/projects/manticore/badge/?version=latest)](http://manticore.readthedocs.io/en/latest/?badge=latest)
+[![Example Status](https://img.shields.io/github/workflow/status/trailofbits/manticore-examples/CI/master)](https://github.com/trailofbits/manticore-examples/actions?query=workflow%3ACI)
 
 Manticore is a symbolic execution tool for analysis of smart contracts and binaries.
 
@@ -141,7 +142,7 @@ Install and try Manticore in a few shell commands:
 
 ```bash
 # Install system dependencies
-sudo apt-get update && sudo apt-get install python3 python3-pip -y
+sudo apt-get update && sudo apt-get install python3 python3-dev python3-pip -y
 
 # Install Manticore and its dependencies
 sudo pip3 install manticore[native]

diff --git a/docs/conf.py b/docs/conf.py
@@ -54,9 +54,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = "0.3.2"
+version = "0.3.3"
 # The full version, including alpha/beta/rc tags.
-release = "0.3.2"
+release = "0.3.3"
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/examples/linux/.gitignore b/examples/linux/.gitignore
@@ -0,0 +1,14 @@
+arguments
+basic
+crackme
+crackme.c
+fclose
+helloworld
+ibranch
+indexhell
+nostdlib
+sendmail
+simple_copy
+simpleassert
+sindex
+strncmp
diff --git a/examples/linux/Makefile b/examples/linux/Makefile
@@ -3,7 +3,20 @@ CFLAGS=-O3 -static
 NOSTDLIBFLAGS=-fno-builtin -static -nostdlib -fomit-frame-pointer -fno-stack-protector
 PYTHON=python3
 
-EXAMPLES=basic sindex  strncmp  arguments ibranch sendmail crackme indexhell  helloworld simple_copy simpleassert
+EXAMPLES=      \
+  arguments    \
+  basic        \
+  crackme      \
+  fclose       \
+  helloworld   \
+  ibranch      \
+  indexhell    \
+  sendmail     \
+  simpleassert \
+  simple_copy  \
+  sindex       \
+  strncmp      \
+
 OTHER_EXAMPLES=nostdlib
 
 all: $(EXAMPLES) $(OTHER_EXAMPLES)

diff --git a/examples/linux/fclose.c b/examples/linux/fclose.c
@@ -0,0 +1,31 @@
+// This example closes file descriptors 0, 1, and 2 (which correspond to stdin,
+// stdout, and stderr in most environments).
+//
+// This serves as a reduced testcase for what most of the programs in GNU
+// coreutils do.  See #1602 and #1604 on GitHub.
+
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+
+int main(int argc, char **argv) {
+    if (argc >= 2 && strcmp(argv[1], "--close") == 0) {
+        fprintf(stdout, "Closing file handles!\n");
+
+        int rc = 0;
+        if (fclose(stdin) != 0) {
+            rc += 1;
+        }
+        if (fclose(stdout) != 0) {
+            rc += 2;
+        }
+        if (fclose(stderr) != 0) {
+            rc += 4;
+        }
+
+        return rc;
+    } else {
+        fprintf(stdout, "Not doing anything.\n");
+        return 0;
+    }
+}
diff --git a/manticore/core/manticore.py b/manticore/core/manticore.py
@@ -60,9 +60,15 @@ def to_class(self):
     description="If True enables to run workers over the network UNIMPLEMENTED",
 )
 consts.add("procs", default=10, description="Number of parallel processes to spawn")
+
+proc_type = MProcessingType.multiprocessing
+if sys.platform != "linux":
+    logger.warning("Manticore is only supported on Linux. Proceed at your own risk!")
+    proc_type = MProcessingType.threading
+
 consts.add(
     "mprocessing",
-    default=MProcessingType.multiprocessing,
+    default=proc_type,
     description="single: No multiprocessing at all. Single process.\n threading: use threads\n multiprocessing: use forked processes",
 )
 consts.add(
@@ -305,6 +311,23 @@ def __init__(self, initial_state, workspace_url=None, policy="random", **kwargs)
     def __str__(self):
         return f"<{str(type(self))[8:-2]}| Alive States: {self.count_ready_states()}; Running States: {self.count_busy_states()} Terminated States: {self.count_terminated_states()} Killed States: {self.count_killed_states()} Started: {self._running.value} Killed: {self._killed.value}>"
 
+    @classmethod
+    def from_saved_state(cls, filename: str, *args, **kwargs):
+        """
+        Creates a Manticore object starting from a serialized state on the disk.
+
+        :param filename: File to load the state from
+        :param args: Arguments forwarded to the Manticore object
+        :param kwargs: Keyword args forwarded to the Manticore object
+        :return: An instance of a subclass of ManticoreBase with the given initial state
+        """
+        from ..utils.helpers import PickleSerializer
+
+        with open(filename, "rb") as fd:
+            deserialized = PickleSerializer().deserialize(fd)
+
+        return cls(deserialized, *args, **kwargs)
+
     def _fork(self, state, expression, policy="ALL", setstate=None):
         """
         Fork state on expression concretizations.
@@ -411,6 +434,7 @@ def _load(self, state_id):
         state._id = state_id
         state.manticore = self
         self.forward_events_from(state, True)
+        state.manticore = self
         self.stcache[state_id] = state
         return state
 
@@ -1040,14 +1064,18 @@ def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
 
 
+def raise_signal():
+    signal.signal(signal.SIGINT, signal.SIG_IGN)
+
+
 class ManticoreMultiprocessing(ManticoreBase):
     _worker_type = WorkerProcess
 
     def __init__(self, *args, **kwargs):
         # This is the global manager that will handle all shared memory access
         # See. https://docs.python.org/3/library/multiprocessing.html#multiprocessing.managers.SyncManager
         self._manager = SyncManager()
-        self._manager.start(lambda: signal.signal(signal.SIGINT, signal.SIG_IGN))
+        self._manager.start(raise_signal)
         # The main manticore lock. Acquire this for accessing shared objects
         # THINKME: we use the same lock to access states lists and shared contexts
         self._lock = self._manager.Condition()