Merge branch 'beta' into stable

* beta:
  Merge dev to beta (#506)
  Better exception msg (#300)
  handle window resize event and set width accordingly (#291)
  Remove pwndbg/linkmap.py and malloc.py (#303)
  Fix got command (#306)
  Fire isort so travis won't complain (#302)
  extend next_call to take optional symbol/target to break on (#290)
  got command: possibility to filter results (#284)
  ROPGadget: return when not installed (#283)
  Fix parsed commands (#282)
  added command 'got' to display status of the GOT table (#256)
  Clean up some unnecessary closures in pwndbg.commands (#278)
  Fix missing exception in pwndbg/exception.py (#277)
  Fix input issue after screen resize: pagination off (#276)
  Simplify command exception debugging and make stdio work correctly (#251)
  Add Command.repeat property for repeated commands (#272)
  support for multiple arenas (#262)
  Fix Python2 long types and inthook (#250)
  fixed GDB remote get command (#241)
  Add CONTRIBUTING and ISSUE_TEMPLATE (#238)

.github/CONTRIBUTING.md

@@ -0,0 +1,31 @@
+### Contributing
+
+Contributions to Pwndbg are always welcome! If you want to get more familiar with project idea/structure/whatever - [here are some developer notes](./DEVELOPING.md). If something is not clear, feel free to ask in a github issue!
+
+If you want to help, fork the project, hack your changes and create a pull request.
+
+If this is something big/new feature or a bug, consider creating an issue first.
+
+
+Some guides:
+* [Fork a project](https://help.github.com/articles/fork-a-repo/)
+* [Pull requests](https://help.github.com/articles/about-pull-requests/)
+
+### Versioning and releases
+
+* There are three branches: `stable`, `beta` and `dev`
+* Each developer works on his own fork
+* Only bug-fixes will be merged into either `stable` or `beta`
+* Every release, merges are cascaded `stable -> beta -> dev`
+* After merging, a new minor-point-release (`1.X`) is created for `stable`
+* Releases occur on predetermined schedule
+* Bugs _are not fixed_ on releases older than the current `stable` (i.e. `0.9` is never fixed)
+* Mid-cycle releases get a patch version bump (`1.1.X`) when bugs affecting `stable` or `beta` are found
+* Pull requests which fix bugs target the oldest branch they affect (e.g. `stable`).
+* There might be occassional cherry-picks if something is fixed in a later branch and we don't notice/forget that it should really target an earlier branch.
+* Documentation fixes, travis fixes, CHANGELOG/README fixes and other tiny fixes does not trigger a new point release.
+
+### Contact
+
+If you want to talk with other contributors and pwndbg users 
+join us at our irc channel: #pwndbg at freenode.

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,41 @@
+<!--
+Before reporting a new issue, make sure that we do not have any duplicates already open.
+If there is one it might be good to take part in the discussion there.
+
+Please make sure you have checked that the issue persists on LATEST pwndbg version.
+
+Below is a template for BUG REPORTS.
+Don't include it if this is a FEATURE REQUEST.
+-->
+
+
+### Description
+
+<!--
+Briefly describe the problem you are having in a few paragraphs.
+-->
+
+### Steps to reproduce
+
+<!--
+What do we have to do to reproduce the problem?
+If this is connected to particular C/asm code, 
+please provide the smallest C code that reproduces the issue.
+-->
+
+### My setup
+
+<!--
+Show us your gdb/python/pwndbg/OS/IDA Pro version (depending on your case).
+
+NOTE: We are currently supporting only Ubuntu installations.
+It is known that pwndbg is not fully working e.g. on Arch Linux (the heap stuff is not working there).
+If you would like to change this situation - help us improving pwndbg and supporting other distros!
+
+This can be displayed in pwndbg through `version` command.
+
+If it is somehow unavailable, use:
+* `show version` - for gdb
+* `py import sys; print(sys.version)` - for python
+* pwndbg version/git commit id
+-->

.gitignore

@@ -60,4 +60,8 @@ npm-debug.log
 .gdb_history
 
 # PyCharm project files
-.idea/
+.idea/
+
+# PyTest files
+.pytest_cache/
+tests/.pytest_cache/

.travis.yml

@@ -9,14 +9,14 @@ cache:
         - capstone
         - unicorn
 install:
-  - sudo apt-get -y install gdb
+  - sudo apt-get -y install gdb nasm
   - lsb_release -a
   - pip install -r requirements.txt
   - sudo ./setup.sh
 script:
-  - futurize --all-imports --stage1 --print-function --write --unicode-literals pwndbg
-  - git diff-index --quiet HEAD -- pwndbg
-  - isort --check-only --diff --recursive pwndbg
-  - nosetests ./tests/
+  - futurize --all-imports --stage1 --print-function --write --unicode-literals pwndbg tests
+  - git diff-index --quiet HEAD -- pwndbg tests
+  - isort --check-only --diff --recursive pwndbg tests
+  - PWNDBG_TRAVIS_TEST_RUN=1 ./tests.sh
   - python2.7 -m py_compile ida_script.py $(git ls-files 'pwndbg/*.py')
   - python3   -m py_compile               $(git ls-files 'pwndbg/*.py')

DEVELOPING.md

@@ -0,0 +1,39 @@
+# Random developer notes
+
+Feel free to update the list below!
+
+* If you want to play with pwndbg functions under GDB, you can always use GDB's `pi` which launches python interpreter or just `py <some python line>`.
+
+* If there is possibility, don't use `gdb.execute` as this requires us to parse the string and so on; there are some cases in which there is no other choice. Most of the time we try to wrap GDB's API to our own/easier API.
+
+* We have our own `pwndbg.config.Parameter` (which extends `gdb.Parameter`) - all of our parameters can be seen using `config` or `theme` commands. If we want to do something when user changes config/theme - we can do it defining a function and decorating it with `pwndbg.config.Trigger`.
+
+* The dashboard/display/context we are displaying is done by `pwndbg/commands/context.py` which is invoked through GDB's prompt hook (which we defined in `pwndbg/prompt.py` as `prompt_hook_on_stop`).
+
+* All commands should be defined in `pwndbg/commands` - most of them lie in seperate files but some files contains many of them (e.g. commands corresponding to windbg debugger - in `windbg.py` or some misc commands in `misc.py`). We would also want to make all of them to use `ArgparsedCommand` (instead of `Command` or `ParsedCommand` decorators).
+
+* We change a bit GDB settings - this can be seen in `pwndbg/__init__.py` - there are also imports for all pwndbg submodules
+
+* We have a wrapper for GDB's events in `pwndbg/events.py` - thx to that we can e.g. invoke something based upon some event
+
+* We have a caching mechanism (["memoization"](https://en.wikipedia.org/wiki/Memoization)) which we use through Python's decorators - those are defined in `pwndbg/memoize.py` - just check its usages
+
+* To block a function before the first prompt was displayed use the `pwndbg.decorators.only_after_first_prompt` decorator.
+
+* Memory accesses should be done through `pwndbg/memory.py` functions
+
+* Process properties can be retrieved thx to `pwndbg/proc.py` - e.g. using `pwndbg.proc.pid` will give us current process pid
+
+* We have an inthook to make it easier to work with Python 2 and gdb.Value objects - see the docstring in `pwndbg/inthook.py` . Specifically, it makes it so that you can call `int()` on a `gdb.Value` instance and get what you want. 
+
+* We have a wrapper for handling exceptions that are thrown by commands - defined in `pwndbg/exception.py` - current approach seems to work fine - by using `set exception-verbose on` - we get a stacktrace. If we want to debug stuff we can always do `set exception-debugger on`.
+
+* Some of pwndbg's functionality - e.g. memory fetching - require us to have an instance of proper `gdb.Type` - the problem with that is that there is no way to define our own types - we have to ask gdb if it detected particular type in this particular binary (that sucks). We do it in `pwndbg/typeinfo.py` and it works most of the time. The known bug with that is that it might not work properly for Golang binaries compiled with debugging symbols.
+
+* We would like to add proper tests for pwndbg - see tests framework PR if you want to help on that.
+
+# Testing
+
+Our tests are written using [pytest](https://docs.pytest.org/en/latest/). It uses some magic so that Python's `assert` can be used for asserting things in tests and it injects dependencies which are called fixtures, into test functions.
+
+The fixtures should be defined in [tests/conftest.py](tests/conftest.py). If you need help with writing tests, feel free to reach out on gitub issues/pr or on our irc channel on freenode.

README.md

@@ -1,24 +1,29 @@
-# pwndbg [![Build Status](https://travis-ci.org/pwndbg/pwndbg.svg?branch=master)](https://travis-ci.org/pwndbg/pwndbg) [![license](https://img.shields.io/github/license/mashape/apistatus.svg?maxAge=2592000)]()
+# pwndbg [![Build Status](https://travis-ci.org/pwndbg/pwndbg.svg?branch=dev)](https://travis-ci.org/pwndbg/pwndbg) [![license](https://img.shields.io/github/license/mashape/apistatus.svg?maxAge=2592000)](https://github.com/pwndbg/pwndbg/blob/dev/LICENSE.md) [![Py2&3](https://img.shields.io/badge/Python-2%20%26%203-green.svg)]() [![IRC](https://img.shields.io/badge/freenode-%23pwndbg-red.svg)](https://webchat.freenode.net/?channels=#pwndbg)
 
 `pwndbg` (/poʊndbæg/) is a GDB plug-in that makes debugging with GDB suck less, with a focus on features needed by low-level software developers, hardware hackers, reverse-engineers and exploit developers.
 
 It has a boatload of features, see [FEATURES.md](FEATURES.md).
 
 ## Why?
 
-Vanilla GDB is terrible to use for reverse engineering and exploit development. Typing `x/g30x $esp` is not fun, and does not  confer much information.  The year is 2016 and GDB still lacks a hexdump command.  GDB's syntax is arcane and difficult to approach.  Windbg users are completely lost when they occasionally need to bump into GDB.
+Vanilla GDB is terrible to use for reverse engineering and exploit development. Typing `x/g30x $esp` is not fun, and does not  confer much information.  The year is 2017 and GDB still lacks a hexdump command.  GDB's syntax is arcane and difficult to approach.  Windbg users are completely lost when they occasionally need to bump into GDB.
 
 ## What?
 
 Pwndbg is a Python module which is loaded directly into GDB, and provides a suite of utilities and crutches to hack around all of the cruft that is GDB and smooth out the rough edges.
 
-Many other projects from the past (e.g., [gdbinit][gdbinit], [PEDA][PEDA]) and present (e.g. [GEF][GEF]) exist to fill some these gaps.  Unfortunately, they're all either unmaintained, unmaintainable, or not well suited to easily navigating the code to hack in new features (respectively).
+Many other projects from the past (e.g., [gdbinit][gdbinit], [PEDA][PEDA]) and present (e.g. [GEF][GEF]) exist to fill some these gaps.  Each provides an excellent experience and great features -- but they're difficult to extend (some are unmaintained, and all are a single [100KB][gdbinit2], [200KB][peda.py], or [300KB][gef.py] file (respectively)).
 
 Pwndbg exists not only to replace all of its predecessors, but also to have a clean implementation that runs quickly and is resilient against all the weird corner cases that come up.
 
 [gdbinit]: https://github.com/gdbinit/Gdbinit
+[gdbinit2]: https://github.com/gdbinit/Gdbinit/blob/master/gdbinit
+
 [PEDA]: https://github.com/longld/peda
+[peda.py]: https://github.com/longld/peda/blob/master/peda.py
+
 [GEF]: https://github.com/hugsy/gef
+[gef.py]: https://github.com/hugsy/gef/blob/master/gef.py
 
 ## How?
 
@@ -40,7 +45,9 @@ For further info about features/functionalities, see [FEATURES](FEATURES.md).
 
 Most of Pwndbg was written by [Zach Riggle](https://twitter.com/ebeip90), with [many other contributors](https://github.com/pwndbg/pwndbg/graphs/contributors) offering up patches via Pull Requests.
 
+Want to help with development? Read [CONTRIBUTING](.github/CONTRIBUTING.md).
+
 ## Contact
 If you have any questions not worthy of a [bug report](https://github.com/pwndbg/pwndbg/issues), feel free to ping
-at [`ebeip90` on Freenode](irc://irc.freenode.net/pwndbg) and ask away.
+[`ebeip90` at #pwndbg on Freenode](irc://irc.freenode.net/pwndbg) and ask away.
 Click [here](https://kiwiirc.com/client/irc.freenode.net/pwndbg) to connect.

docs/requirements.txt

@@ -20,4 +20,3 @@ python-ptrace>=0.8
 six
 future
 unicorn>=1.0.0
-capstone

ida_script.py

@@ -6,6 +6,7 @@
 import threading
 import xmlrpclib
 from SimpleXMLRPCServer import SimpleXMLRPCServer
+from xml.sax.saxutils import escape
 
 import idaapi
 import idautils
@@ -20,13 +21,36 @@
 dt = datetime.datetime.now().isoformat().replace(':', '-')
 
 # Save the database so nothing gets lost.
-idc.SaveBase(idc.GetIdbPath() + '.' + dt)
+if idaapi.IDA_SDK_VERSION >= 700:
+    idaapi.save_database(idc.GetIdbPath() + '.' + dt)
+else:
+    idc.SaveBase(idc.GetIdbPath() + '.' + dt)
 
-xmlrpclib.Marshaller.dispatch[type(0L)] = lambda _, v, w: w("<value><i8>%d</i8></value>" % v)
-xmlrpclib.Marshaller.dispatch[type(0)] = lambda _, v, w: w("<value><i8>%d</i8></value>" % v)
+
+DEBUG_MARSHALLING = False
+
+def create_marshaller(use_format=None, just_to_str=False):
+    assert use_format or just_to_str, 'Either pass format to use or make it converting the value to str.'
+
+    def wrapper(_marshaller, value, appender):
+        if use_format:
+            marshalled = use_format % value
+        elif just_to_str:
+            marshalled = '<value><string>%s</string></value>' % escape(str(value))
+
+        if DEBUG_MARSHALLING:
+            print("Marshalled: '%s'" % marshalled)
+
+        appender(marshalled)
+
+    return wrapper
+
+xmlrpclib.Marshaller.dispatch[type(0L)] = create_marshaller("<value><i8>%d</i8></value>")
+xmlrpclib.Marshaller.dispatch[type(0)] = create_marshaller("<value><i8>%d</i8></value>")
+xmlrpclib.Marshaller.dispatch[idaapi.cfuncptr_t] = create_marshaller(just_to_str=True)
 
 host = '127.0.0.1'
-port = 8888
+port = 31337
 orig_LineA = idc.LineA
 
 
@@ -44,23 +68,30 @@ def LineA(*a, **kw):
 
 def wrap(f):
     def wrapper(*a, **kw):
-        try:
-            rv = []
-
-            def work():
-                rv.append(f(*a, **kw))
-
-            with mutex:
-                flags = idaapi.MFF_WRITE
-                if f == idc.SetColor:
-                    flags |= idaapi.MFF_NOWAIT
-                    rv.append(None)
-                idaapi.execute_sync(work, flags)
-            return rv[0]
-        except:
-            import traceback
-            traceback.print_exc()
-            raise
+        rv = []
+        error = []
+
+        def work():
+            try:
+                result = f(*a, **kw)
+                rv.append(result)
+            except Exception as e:
+                error.append(e)
+
+        with mutex:
+            flags = idaapi.MFF_WRITE
+            if f == idc.SetColor:
+                flags |= idaapi.MFF_NOWAIT
+                rv.append(None)
+            idaapi.execute_sync(work, flags)
+
+        if error:
+            msg = 'Failed on calling {}.{} with args: {}, kwargs: {}\nException: {}' \
+                .format(f.__module__, f.__name__, a, kw, str(error[0]))
+            print('[!!!] ERROR:', msg)
+            raise error[0]
+
+        return rv[0]
 
     return wrapper
 
@@ -71,15 +102,50 @@ def register_module(module):
             server.register_function(wrap(function), name)
 
 
+def decompile(addr):
+    """
+    Function that overwrites `idaapi.decompile` for xmlrpc so that instead
+    of throwing an exception on `idaapi.DecompilationFailure` it just returns `None`.
+    (so that we don't have to parse xmlrpc Fault's exception string on pwndbg side
+    as it differs between IDA versions).
+    """
+    try:
+        return idaapi.decompile(addr)
+    except idaapi.DecompilationFailure:
+        return None
+
+
+def versions():
+    """Returns IDA & Python versions"""
+    import sys
+    return {
+        'python': sys.version,
+        'ida': idaapi.get_kernel_version(),
+        'hexrays': idaapi.get_hexrays_version() if idaapi.init_hexrays_plugin() else None
+    }
+
+
 server = SimpleXMLRPCServer((host, port), logRequests=True, allow_none=True)
 register_module(idc)
 register_module(idautils)
 register_module(idaapi)
 server.register_function(lambda a: eval(a, globals(), locals()), 'eval')
+server.register_function(decompile)  # overwrites idaapi/ida_hexrays.decompie
+server.register_function(versions)
 server.register_introspection_functions()
 
-print('Ida Pro xmlrpc hosted on http://%s:%s' % (host, port))
+print('IDA Pro xmlrpc hosted on http://%s:%s' % (host, port))
+print('Call `shutdown()` to shutdown the IDA Pro xmlrpc server.')
 
 thread = threading.Thread(target=server.serve_forever)
 thread.daemon = True
 thread.start()
+
+
+def shutdown():
+    global server
+    global thread
+    server.shutdown()
+    server.server_close()
+    del server
+    del thread

profiling/.gitignore

@@ -0,0 +1,3 @@
+test
+stats
+stats.log

profiling/benchmark.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# Benchmark context command
+make test > /dev/null
+git log --abbrev-commit --pretty=oneline HEAD^..HEAD
+gdb ./test \
+  -ex "source ../gdbinit.py" \
+  -ex "b main" -ex "r" \
+  -ex "python import timeit; print('      1ST RUN:', timeit.repeat('pwndbg.commands.context.context()', repeat=1, number=1, globals=globals())[0])" \
+  -ex "si" \
+  -ex "python import timeit; print('      2ND RUN:', timeit.repeat('pwndbg.commands.context.context()', repeat=1, number=1, globals=globals())[0])" \
+  -ex "si" \
+  -ex "python import timeit; print('MULTIPLE RUNS:', timeit.repeat('pwndbg.commands.context.context()', repeat=1, number=10, globals=globals())[0] / 10)" \
+  -ex "quit" | grep 'RUNS*:'

profiling/profile.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+# Quick and dirty script to profile pwndbg using cProfile.
+make test > /dev/null
+git log --abbrev-commit --pretty=oneline HEAD^..HEAD
+# To profile first run, remove -ex "context".
+gdb ./test \
+  -ex "source ../gdbinit.py" \
+  -ex "b main" -ex "r" \
+  -ex "context" \
+  -ex "python import cProfile; cProfile.run('pwndbg.commands.context.context()', 'stats')" \
+  -ex "quit"
+
+python3 -c "
+import pstats
+p = pstats.Stats('stats')
+p.strip_dirs().sort_stats('tottime').print_stats(20)
+"
+[ -x /usr/local/bin/pyprof2calltree ] && command -v kcachegrind >/dev/null 2>&1 && /usr/local/bin/pyprof2calltree -k -i stats

profiling/test.c

@@ -0,0 +1,3 @@
+int main() {
+  while(1);
+}