pwnlib 2.2 rewrite of context, logging, tubes, various bugfixes

[zachriggle]

Major pwnlib overhaul to address outstanding issues.

It is highly recommended to look at the diff with whitespace disabled (add ?w=1 to URL or git diff -w).

Issues Closed

The following issues are closed by this pull request.

• process.connected_raw implemented incorrectly #159 process.connected_raw implemented incorrectly
• Tube recvline behavior inconsistent #151 Recvline regression
• Tubes repeat timeout #150 Tubes timeout
• pip2 not present on Debian #147 "pip2 not present on debian"
• Context default values and side-effects #141 Context default values and side-effects
• Context module usage #140 Context module usage
• Be able to turn logging on/off through environment #135 Logging re-write

Test Coverage

I have added a great deal of unit tests to pwntools, and started a regression testing framework.

Unit Test Coverage

One test fails without a custom-built binutils-mips.

$ make doctest
...
Doctest summary
===============
  591 tests
    1 failure in tests
    0 failures in setup code
    0 failures in cleanup code

Integration Testing

I've created a new repository at zachriggle/pwnlib-regression which performs regression testing against functionality. This uses a binary which is designed to run shellcode, leak offsets, and in general be useful for testing pwntools. bbgp and pork do not work. I have created a new branch for that repo with the changes necessary to work properly.

Additionally, I've verified that most of the pwnlib-write-ups exploits still work, as well as my collection of io.smashthestack.com exploits which I cannot publish. All of the published wargames exploits work.

Major Changes

Documentation

A good deal of documentation has been added and updated.

pwnlib.context

This has been made into a proper module. The global context is now just a globally-scoped object.

The mechanism used for defaults and threading have been updated, so that they're a bit cleaner.

Setting context.arch now sets other context values to sane defaults.

pwnlib.log

Uses the normal Python logging subsystem. Log levels for individual modules can now be set, while context.log_level provides a maximum verbose-ness for logging (same behavior as before). This is useful for when developing to see e.g. verbose ELF messages without printing everything a tube ever sees.

All animation still works exactly as it did. log.waitfor can now be used with the scoping with keyword.

The log_level module is removed.

pwnlib.asm

Large amount re-written to completely migrate to binutils, rather than relying on nasm for x86 and dealing with the differences. Also auto-detects the names of binaries, as the exact triplet of the installed version may differ. Since we're not relying on any ABIs when assembling shellcode, it doesn't matter if we use arm-linux-gnueabi or arm-none-none.

pwnlib.shellcraft

In keeping with the changes to pwnlib.asm, all of the i386 shellcode has been rewritten in gas -Mintel syntax.

pwnlib.gdb

Includes gdb.debug for debugging over SSH, as shown in #108.

pwnlib.data

All of the precompiled binutils have been removed. Users should use their package manager to install cross-architecture assemblers/disassemblers.

pwnlib.util.crc

Instead of performing text processing at-runtime to scrape constants, this information has been cached in pwnlib/util/crc/known.py.

pwnlib.util.packing

Rather than the same repeating copy/pasted/modified code, most of the routines are created dynamically at runtime. This should make maintenance easier, and less prone to copy-paste errors.

pwnlib.tubes

The internal buffering was pulled out into a separate Buffer class. The mechanism used to fix timeouts, _fix_timeout was pulled out into a context-aware structure. pwnlib.tubes.tube.tube now inherits from pwnlib.tubes.timeout.Timeout so that all tube objects can leverage this behavior.

[zachriggle]

I can confirm that f66c6b7 permits pwntools to assemble/disassemble code with no additional dependencies outside of those required by 'pwntools' itself, on Ubuntu 12.04 (for the installed architecture).