Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Command bins is not working properly #424

Closed
Eterna1 opened this issue Feb 26, 2018 · 9 comments

Comments

@Eterna1
Copy link

commented Feb 26, 2018

Description

Hello, this commit 6a640e8 has broken down bins command.

Steps to reproduce

Here is the simple test:

#include <stdio.h>

void foo() {}

int main() {
    char *a = malloc(0x16);
    malloc(0x16);
    free(a);
    foo();
} 

Set beakpoint on foo function and when it has been hit, type bins command.
Here is the result of the command after this commit: pastebin
Here is the result of the command before this commit:

pwndbg> bins
fastbins
0x20: 0x602000 ◂— 0x0
0x30: 0x0
0x40: 0x0
0x50: 0x0
0x60: 0x0
0x70: 0x0
0x80: 0x0
unsortedbin
all: 0x0
smallbins
empty
largebins
empty

My setup

I've been using 2 different setups so this is not the case probably

@Eterna1 Eterna1 changed the title commit 6a640e8 broke command bins commit 6a640e8 broke down command bins Feb 26, 2018

@disconnect3d disconnect3d changed the title commit 6a640e8 broke down command bins Command bins is not working properly Feb 26, 2018

@Eterna1

This comment has been minimized.

Copy link
Author

commented Feb 28, 2018

Can this commit be just reverted ?

@disconnect3d

This comment has been minimized.

Copy link
Member

commented Mar 1, 2018

@Eterna1 It could be, for a hotfix, if the issue happened on stable branch, but it appears only on dev (so, as a dev, one can revert it by themselves - if you really need a working dev version).

Lets aim for fixing this, as the idea of the changes is fine.

@disconnect3d

This comment has been minimized.

Copy link
Member

commented Mar 1, 2018

@AlexAltea would you be willing to help with that?

@AlexAltea

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2018

I'm not familiar with the bins command. What exactly is the issue here? According to your snippets it seems as if my commit caused smallbins and largebins to be non-empty and unsortedbin to contain a different value in all. I don't even grasp why would my patch change that behavior.

@disconnect3d If this is my commit's fault, then I'd definitely try to debug and fix it myself, starting from March 12th (I'm afraid that before that day I'll be too busy with RL-projects).

@disconnect3d

This comment has been minimized.

Copy link
Member

commented Mar 4, 2018

@AlexAltea I am not sure if that's ur fault but it seems like so :s

I really need to prepare myself a setup for working with heap stuff...

@mak

This comment has been minimized.

Copy link
Contributor

commented Mar 25, 2018

So the difference now is that chain.get pushes address before checking a hard_stop limit in the past was the other way around and since bins are initialized with their corresponding address loop was terminated returning empty list

if current semantics of chain.get is preferable then easy fix will be to check value in bins before resolving a chain, or add a flag to chain.get that wont append first address before the loop

@veritas501

This comment has been minimized.

Copy link

commented Mar 26, 2018

try this patch for get in pwndbg/chain.py

def get(address, limit=LIMIT, offset=0, hard_stop=None, hard_end=0):
    """
    Recursively dereferences an address. For bare metal, it will stop when the address is not in any of vmmap pages to avoid redundant dereference.

    Arguments:
        address(int): the first address to begin dereferencing
        limit(int): number of valid pointers
        offset(int): offset into the address to get the next pointer
        hard_stop(int): address to stop at
        hard_end: value to append when hard_stop is reached

    Returns:
        A list representing pointers of each ```address``` and reference
    """
    limit = int(limit)

    #result = [address]
    result = [] #patch 1
    for i in range(limit):
        # Don't follow cycles, except to stop at the second occurrence.
        if result.count(address) >= 2:
            break

        if hard_stop is not None and address == hard_stop:
            result.append(hard_end)
            break

        result.append(address)#patch 2
        try:
            address = address + offset

            # Avoid redundant dereferences in bare metal mode by checking
            # if address is in any of vmmap pages
            if not pwndbg.abi.linux and not pwndbg.vmmap.find(address):
                break

            address = int(pwndbg.memory.poi(pwndbg.typeinfo.ppvoid, address))
            address &= pwndbg.arch.ptrmask
            #result.append(address)  #patch 3
        except gdb.MemoryError:
            break

    return result
@disconnect3d

This comment has been minimized.

Copy link
Member

commented May 3, 2018

@veritas501 Thx for posting the patch! In the future please use pull requests for that :P (I have missed this one somehow :/).

I am going to check whether chain/telescope functionality works fine after those changes and merge it if it works.

disconnect3d added a commit to disconnect3d/pwndbg that referenced this issue May 3, 2018

Fixes bins command (pwndbg#424)
The problem was that after some of the recent changes to chain/get to prevent dereferencing too much addresses and having better display when dereferencing limit is 0 (used for bare metal debugging) the bins command displayed wrong results for everything except fastbins.

This was due to the fact we are adding the dereference start address to the list.

This fixes the `bins` command by adding `include_start=True` keyword argument to the `chain.get` function. The `bins` simply uses `include_start=False`.
@disconnect3d

This comment has been minimized.

Copy link
Member

commented May 3, 2018

@veritas501 After your proposed patch the set dereference-limit 0 -> context crashes.

Anyway, I have applied a fix proposed by @mak - by addding an include_start keyword argument to the chain.get. This lets us prevent adding the start address to the chain inside *bins commands.

The PR can be found here: #450

As the problem has been fixed, I am closing this issue :).

disconnect3d added a commit that referenced this issue May 3, 2018

Fixes bins command (#424) (#450)
The problem was that after some of the recent changes to chain/get to prevent dereferencing too much addresses and having better display when dereferencing limit is 0 (used for bare metal debugging) the bins command displayed wrong results for everything except fastbins.

This was due to the fact we are adding the dereference start address to the list.

This fixes the `bins` command by adding `include_start=True` keyword argument to the `chain.get` function. The `bins` simply uses `include_start=False`.

disconnect3d added a commit that referenced this issue Jul 29, 2018

Merge dev to beta (#506)
* Fixes `u` command `module object is not callable` (#311)

pwndbg> u 0x404030
'u': Starting at the specified address, disassemble
    N instructions (default 5).
Traceback (most recent call last):
  File "/home/dc/installed/pwndbg/pwndbg/commands/__init__.py", line 99, in __call__
    return self.function(*args, **kwargs)
  File "/home/dc/installed/pwndbg/pwndbg/commands/__init__.py", line 191, in _OnlyWhenRunning
    return function(*a, **kw)
  File "/home/dc/installed/pwndbg/pwndbg/commands/windbg.py", line 292, in u
    pwndbg.commands.nearpc(where, n)
TypeError: 'module' object is not callable

* refactored wrapper (#280)

* added command got to display status of the got table

Signed-off-by: degrigis <degrigis@gmail.com>

* return when checksec is not available and added decorator OnlyWhenRunning

Signed-off-by: degrigis <degrigis@gmail.com>

* removed duplicated code for pie and not pie binaries

Signed-off-by: degrigis <degrigis@gmail.com>

* inserted support function to get checksec output and performed all requirements check initially

Signed-off-by: degrigis <degrigis@gmail.com>

* corrected typo

Signed-off-by: degrigis <degrigis@gmail.com>

* reorganized the command got splitting the code in library routines and moved the checksec internal function in a separate module

Signed-off-by: degrigis <degrigis@gmail.com>

* handled exception directly inside functions and enhanced code

Signed-off-by: degrigis <degrigis@gmail.com>

* extracted only column in readelf output and enhanced exception handling

Signed-off-by: degrigis <degrigis@gmail.com>

* fix exception handling returning subprocess error

Signed-off-by: degrigis <degrigis@gmail.com>

* removed unused import and reordered

Signed-off-by: degrigis <degrigis@gmail.com>

* reordered imports

Signed-off-by: degrigis <degrigis@gmail.com>

* added wrappers module and refactored some code

Signed-off-by: degrigis <degrigis@gmail.com>

* removed not useful comment

Signed-off-by: degrigis <degrigis@gmail.com>

* removed unused import

Signed-off-by: degrigis <degrigis@gmail.com>

* moved comments in docstring

Signed-off-by: degrigis <degrigis@gmail.com>

* refactored code to use partial functions, simplified code

Signed-off-by: degrigis <degrigis@gmail.com>

* simplified a loc

Signed-off-by: degrigis <degrigis@gmail.com>

* capslock char fixed

Signed-off-by: degrigis <degrigis@gmail.com>

* removed unuseful pwndbg.arch.ptrsize check

Signed-off-by: degrigis <degrigis@gmail.com>

* refactored code and added the new module wrapper that contains every new wrapper module

Signed-off-by: degrigis <degrigis@gmail.com>

* used class style decorator for wrapper and improved code style

Signed-off-by: degrigis <degrigis@gmail.com>

* changed return with print for errors

Signed-off-by: degrigis <degrigis@gmail.com>

* removed prints debug and statically linked check moved at the top of the got function

Signed-off-by: degrigis <degrigis@gmail.com>

* refactored OnlyWithCommand decorator

Signed-off-by: degrigis <degrigis@gmail.com>

* wrappers are OnlyWithFile now

Signed-off-by: degrigis <degrigis@gmail.com>

* redirected stderr to stdout in subprocess.check_output and memoized the wrappers for readelf/file/checksec

Signed-off-by: degrigis <degrigis@gmail.com>

* reordered an import

Signed-off-by: degrigis <degrigis@gmail.com>

* removed pdb

Signed-off-by: degrigis <degrigis@gmail.com>

* fixed format string and removed desc from got command

Signed-off-by: degrigis <degrigis@gmail.com>

* consolidated decorators

Signed-off-by: degrigis <degrigis@gmail.com>

* merging

Signed-off-by: degrigis <degrigis@gmail.com>

* reordered import for travis

Signed-off-by: degrigis <degrigis@gmail.com>

* refactored some code

Signed-off-by: degrigis <degrigis@gmail.com>

* resolve travis complains

Signed-off-by: degrigis <degrigis@gmail.com>

* docstring for _extract_jumps

Signed-off-by: degrigis <degrigis@gmail.com>

* fixed isort

Signed-off-by: degrigis <degrigis@gmail.com>

* f*** isort

Signed-off-by: degrigis <degrigis@gmail.com>

* Enhance canary command

Canary command:
* Displays telescope result of places where canaries are located
* Moved to its own file (`pwndbg/commands/canary.py`)
* Moved to `ArgparsedCommand` (as discussed in #244)

* update for ida_script.py to handle ida 7.0 (#308)

* fix for ida 7.0

* using idaapi.save_database instead, change version cmp from == to >=

* Fix the current year (#319)

This triggered me

* checksec: cache output of command (#317)

* checksec: cache output of command

* checksec: use get_raw_out() for derived functions

* cp fixes from stable: malloc chunk names, remote target search bug (#323)

* Fix malloc chunk names (#318)

* heap: respect rename of malloc_chunk fields

newer glibc uses different names for the fields of malloc_chunk

* move value_from_type to typeinfo and rename to read_gdbvalue

* add comment about renaming of `[prev_]size`

* Workaround for gdb remote target search bug described in #321 (#322)

* Move vmmap to ArgparsedCommand; add sloppy_gdb_parse (#285)

* Migrate vmmap command to ArgparsedCommand

* vmmap command: better msg for no mappings

* WIP: vmmap

* Review fixes

* isort fix

* Add nextret command (#301)

* Py version check: use pwndbg.compat.py* instead of sys.version (#327)

* Dumpargs add --force to show all possible register arguments (#326)

* Added --all flag to dumpargs command

This gives possibility to dump all register argument even
if we failed to resolve arguments from metadatas.

* Display info when dumpargs not resolved call args

* Dumpargs: changed --all to --force

* Revert telescope changes as it fails when we are not on call instruction.

* Fix isort

* Fix IDA Pro decompiled code not being displayed (#328)

* Fix withHexrays decorator not returning wrapper function

* IDA xmlrpc: add cfuncptr_t marshaller & better errors

* IDA xmlrpc server: add shutdown() which can be used for dev

* Small refactor of context.py

* Fix context Hexrays decompiled code display

* Fix hard error when something else (not IDA) listens on IDA's port (#330)

* Fix hard error when something else (not IDA) listens on IDA's port

The default IDA port is 8888 and it can happen that some other program (such as
a jupyter notebook) is listening on that address. This made pwndbg unusable,
because it would crash trying to connect to IDA.

* add timeout to ida connect

* Fix exception if there is an indirect jump (#329)

This is a simple typo, but the error message that GDB gave was interesting:

Previously, if you stopped on an instruction that does an indirect jump, like
this:

```
jmp [ecx*4 + 0xdeadbeef]
```

then pwndbg would the following exception:

```
gdb.error: evaluation of this expression requires the program to have a function "malloc".
```

The reason is that the code used `memory_sz` and passed that to gdb.Value, thus
creating a string value. When casting the string to a pointer later, GDB tries
to allocate a string in the inferior which failed since malloc is not available.

The fix is, of course, to use the correct function (`memory`) that returns an
int and not a string.

* Fixes issue when we try to display context while selected thread is running #299 (#331) (#333)

* ArgparsedCommands: config, configfile, theme, themefile (#335)

* Fix Python<=2.7.6 "TypeError: Struct() argument 1 must be string, not unicode" (#336)

* Fix Python<=2.7.6 "TypeError: Struct() argument 1 must be string, not unicode"

Additional information is available here: http://python-future.org/stdlib_incompatibilities.html#struct-pack

* Completely remove libheap, as it is not ever referenced

* Expose IDA Pro commands, even when IDA is not available. (#337)

Closes #225

* Removed duplicate requirement (#339)

2 Lines stated "capstone"

* Closes #338: Fix 'This command is not documented' (#341)

* Reduce the number of times we check to see if running Android

* Do not populate the main exe symbols on Android, it's unnecessary

* Add nextproginstr command (#360)

* Add nextproginstr command

* Fix isort

* Update next.py

* Update next.py

* Update next.py

* Add possibility to prevent skipping repeating telescope values (#359)

* Merge stable to dev (#365)

* Fixes `u` command `module object is not callable` (#310)

pwndbg> u 0x404030
'u': Starting at the specified address, disassemble
    N instructions (default 5).
Traceback (most recent call last):
  File "/home/dc/installed/pwndbg/pwndbg/commands/__init__.py", line 99, in __call__
    return self.function(*args, **kwargs)
  File "/home/dc/installed/pwndbg/pwndbg/commands/__init__.py", line 191, in _OnlyWhenRunning
    return function(*a, **kw)
  File "/home/dc/installed/pwndbg/pwndbg/commands/windbg.py", line 292, in u
    pwndbg.commands.nearpc(where, n)
TypeError: 'module' object is not callable

* Fix malloc chunk names (#318)

* heap: respect rename of malloc_chunk fields

newer glibc uses different names for the fields of malloc_chunk

* move value_from_type to typeinfo and rename to read_gdbvalue

* add comment about renaming of `[prev_]size`

* Workaround for gdb remote target search bug described in #321 (#322)

* Fixes issue when we try to display context while selected thread is running #299 (#331)

* Fix tag_release (#348)

* Fix "dt" offsets which are sometimes floating-point (#355)

* Fixes #362 - broken entry command (#363)

* fix ds and da for gdb 7.11 (#364)

* fix ds and da for gdb 7.11

* add max argument to da and ds

* Support bare metal environment (#369)

* Add elf.find_elf_magic() and remove duplicate code

* Add pwndbg.abi.LinuxOnly decorator

* Support bare metal environment

Use @pwndbg.abi.LinuxOnly and pwndbg.abi.linux to disable
several util functions which search the memory to find
the AUXV, the ELF header, or the page bound.

* Add xinfo command for extended offset information (#376)

This commit adds a `xinfo` command that calculates the offset of a
specified address to other interesting locations within the address
space:

* In the most general case, simply the offset of the pointer into the
current mapping is displayed.
* If the address specified is a stack adress, the offsets to the top and
the bottom of the stack, as well as to the current stack pointer,
frame pointer and stack canary are displayed.
* If the address points into a memory mapped file, the command
additionally shows the offset to the beginning of the file in memory and
on disk.

* Fail on two commands with the same name (#372)

* More badges in README

Add "Python 2&3" and "freenode: #pwndbg" badges created with https://shields.io/

* Fix Python 2&3 badge in README

* Update README badges links

* Add dereference-limit and heap-dereference-limit parameters (#367)

* Add dereference-limit and heap-dereference-limit parameters

This allows setting the number of pointers dereferenced during 'telescope'
and in the register context.  Separately, the number of heap bins which
are dereferenced can be set.

* Cast LIMIT to an integer, and address off-by-one

* Adds $rebase(offset) function (#374)

Adds `$rebase(offset)` gdbfunction that can be used to set up a breakpoint
over an offset from program image base.

Also changed a bit the pwndbg banner displayed at startup.

* ArgparsedCommand: pass parser or description; move some cmds to ArgparsedC~ (#373)

* Fix upper stack boundary (#377)

* Fix upper_stack_boundary not working introduced in 31f468e

The `upper_stack_boundary` we returned wasn't matching the one from `vmmap`.

Previously we determined upper address by having a memory read failure.
Recent changes made it so we got a `None` instead of the address in such situation.

This adds a parameter to `find_elf_magic` which lets us get a result when gdb.MemoryError occurs.

* Small refactor: add missing newlines

* Add capstone, unicorn versions to version command (#379)

* Merge stable to dev (#381)

* Fixes `u` command `module object is not callable` (#310)

pwndbg> u 0x404030
'u': Starting at the specified address, disassemble
    N instructions (default 5).
Traceback (most recent call last):
  File "/home/dc/installed/pwndbg/pwndbg/commands/__init__.py", line 99, in __call__
    return self.function(*args, **kwargs)
  File "/home/dc/installed/pwndbg/pwndbg/commands/__init__.py", line 191, in _OnlyWhenRunning
    return function(*a, **kw)
  File "/home/dc/installed/pwndbg/pwndbg/commands/windbg.py", line 292, in u
    pwndbg.commands.nearpc(where, n)
TypeError: 'module' object is not callable

* Fix malloc chunk names (#318)

* heap: respect rename of malloc_chunk fields

newer glibc uses different names for the fields of malloc_chunk

* move value_from_type to typeinfo and rename to read_gdbvalue

* add comment about renaming of `[prev_]size`

* Workaround for gdb remote target search bug described in #321 (#322)

* Fixes issue when we try to display context while selected thread is running #299 (#331)

* Fix tag_release (#348)

* Fix "dt" offsets which are sometimes floating-point (#355)

* Fixes #362 - broken entry command (#363)

* Fix #373 for python2 env (#384)

Since the python2 use `from __future__ import unicode_literals`, so the string literals will be `unicode` type in python2.
Use `six.string_types` in `isinstance()` instead of using `str` type.

* Fix Endianess issue and Memory error on GDB (#386)

* Fix py2 import error (shlex.quotes vs pipes.quotes) (#389)

* Avoid enhancements when dereference limit is zero (#380)

* Avoid enhancements when dereference limit is zero

* Replace last element in chain with enhancements

* make everything themeable (#392)

* theme: make everything themable by avoiding explicite colors

This makes it posssible to theme everything logically grouped by
message types. This will also make it easier for future features
to keep a consistent way of coloring plus make every non-specific
coloring themeable automatically.

Direct explicit usage of colors should be avoided in future commits.

* theme: make banner fully customizable including positions

* fixup: fix wrong import during refactoring (#394)

* Fix inthook for enums in Python 3 (#393)

Fixes the problem that can be observed below:

```
pwndbg> py import re; flags = 1 | re.MULTILINE
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.6/enum.py", line 798, in __or__
    result = self.__class__(self._value_ | self.__class__(other)._value_)
  File "/usr/lib/python3.6/enum.py", line 291, in __call__
    return cls.__new__(cls, value)
  File "/usr/lib/python3.6/enum.py", line 533, in __new__
    return cls._missing_(value)
  File "/usr/lib/python3.6/enum.py", line 762, in _missing_
    new_member = cls._create_pseudo_member_(value)
  File "/usr/lib/python3.6/enum.py", line 788, in _create_pseudo_member_
    pseudo_member._name_ = None
AttributeError: 'int' object has no attribute '_name_'
```

* Implement asm&source syntax highlight (#390)

* Syntax-highlight: Add asm lexer in color/lexer.py

* Syntax-highlight: Add pygments to requirements.txt

* Syntax-highlight: Update lexer for supporting ARM

Support symbol, constant, comments

* Syntax-highlight: Enable asm syntax highlight

* Syntax-highlight: Add source highlight utils in commnads/context.py

* Syntax-highlight: Add disasm highlight utils in color/disasm.py

* Syntax-highlight: Implement Source code highlighting in commands/context.py

* Syntax-highlight: Add syntax_highlight() in color/syntax_highlight.py

* Fix texts

* Add color theme and prefix config for context code

* Add missed utf8 magic comment

* Fix isort

* context: bring back args section (default off) (#397)

This allows to use args section via the context-sections config
setting (default off).
Additionally introduce a nearpc-show-args config value making it
possible to disable showing it trice while using the args section.

* config: validate context-sections and show all available values (#396)

When setting an illegal value, fall back to the default sections.

* Make de-reference only works on known pages in bare metal mode and add commands to manually add pages (#385)

* Make chain.get() to check vmmap first in bare metal mode

Make chain.get() limit to de-reference within the known page in
bare metal mode.
Since the address are all valid when mmu is not enable and all
the value are valid physical address. It will be de-referenced
even these addresses are not used and actually, it is data in
the most of case. Ex. 0x1 often means the value 1, not the
address 0x1.

Also, for issue #371, some addresses may be the MMIO registers.
The read operation on these address will break the state.
It is better to limit the de-reference address range. This patch
will also fix it, hopefully.

* Add custom vmmap add/del API in vmmap.py

In some cases, ex. bare metal, the pages information can not be
detected automatically. Also, the most of pwndbg feature rely on
page information such as highlighting.
User may want to create page information manually and maintain it
by himself.

This commit add python APIs to manually add/del page information
and they are isolated.

* Fix stack page detection in bare metal mode

We can not detect the stack page size in bare metal mode by
1. finding the ELF location after the stack page
2. page fault

A simple workaround is returning the current $sp page
and assume it is the stack page.

* Add vmmap control command to add/del customized vmmap

In some cases, ex. bare metal, the pages information can not be
detected automatically. Also, the most of pwndbg feature rely on
page information such as highlighting.
User may want to create page information manually and maintain it
by himself.

I add few commands to make user can add/del pages and load page
information from ELF sections.

* Fix the command amount for auto test to pass CI

* Add warning message

* Fix descriptions

* Fix cache issue and use bisect in insert API

* Keep LinuxOnly in find_elf_magic

* remove XXX

* improve repeat functionality of commands (#395)

* hexdump: adjust shown offset from src while repeating

* nearpc: make command repeatable to show further instructions

The pc gets adjusted to the last instructions address making it
visually easy to follow where to continue reading the assembly.

This also forwards repeating of emulate() and pdisass()

* telescope: make command repeatable with adjusted offset from src

This also forwards stack() to be repeatable.

* [WIP] Stop highlight and prefix display when repeating nearpc command (#399)

Fix nearpc command repeat: highlight, prefix and instruction display

* color: make lrjust() work with multiple chars (#401)

This fixes the issue if ljust with multiple characters like
the banner separator char

* syntax: highlight code for chain format during enhance (#400)

* Fix missing enum in Python 2 (#403)

* Add developer notes (#405)

* Update DEVELOPING.md

* Better detection of extended-remote types

* Bring back possibility of empty context (#409)

* Change README about GEF / GDBINIT / PWNDBG (#413)

* Ban isort==4.3.0 (#417)

See timothycrosley/isort#652 for more information. tl;dr is `pip install isort==4.3.0` fails, which is what `pip install -Ur requirements.txt` will attempt to do.

Ban this specific version as it causes issues.

* display the frame pointer register (x29) in aarch64 context (#418)

* ensure length padding works with py2 by enforcing unicode awareness (#416)

This works around the issue of python2 not being unicode aware and
the config classes not properly returning instance of decoded raw
strings. This leads to length operations being performed on bytes
rather then logical characters.
We check for python2 and enfore decoding if not a text_type.

Fixes #412

* Profiling and performance improvements (#421)

* Add scripts for benchmarking and profiling pwndbg commands

* Fix performance issue in emulator.py

Register to unicorn enum lookup was really ineffective. Replaced with
parsing (consts) on initialization time, and only dict lookup on hot path.

* Fix performance issue in syntax_highlight.

Current code initialized pygments on each syntax_highlight(), which
apparently took some time.

* Minor performance improvements in syntax_highlight

* Memoize IDA availability.

Not sure it this is a valid solution, I have never used pwndbg with IDA.
However, we should not try to connect to ida on each context(), as this
takes 25% of current exec time.

* Explicitly source gdbinit in benchmark scripts.

* Refactor variable names in nearpc (#422)

* Try to connect to IDA on every debugger stop. (#423)

Add option to disable IDA integration completly.

* Fix Parameter config class (#404)

* Avoid to use 'type' as varialbe name

* Fix utf8 issue of Parameter.value in python2

* Fix Parameter member funcs

* Operator overwrite of Parameter

* Remove all workaround of Parameter

* Use regex to unwrap the string

* Remove impossible cases in commands/context.py after Parameter class update

* Revert "ensure length padding works with py2 by enforcing unicode awareness (#416)"

This reverts commit 8ecaa67.

* Fix #429 - osabi check for non-English GDB version (#430)

Detailed info is within the issue, but TLDR:

```
(gdb) show osabi
El actual SO ABI es «auto» (actualmente «GNU/Linux»).
El SO ABI predeterminado es «GNU/Linux».
```

* add basic rust support (#431)

When a rust binary is loaded gdb will not find the usual c types.

* Fixes #428 - pwndbg.memory.write encoding error (#432)

It seems that pwndbg.memory.write fix for Py2 introduced in 433bf23
wasn't tested properly on Py3.

In Py2 by default the `bytes` is just `str` and so doesn't accept the encoding argument.

Because of that a `from builtins import bytes` has been added.

Some more info on `builtins` module can be found here: http://python-future.org/imports.html#imports-of-builtins

* Fixes #427 - readelf parsing error on old readelf versions (#433)

Here is `readelf --program-headers <binary>` output for different readelf versions
(The `//` are commented lines; the output is truncated so it contains only useful data):

```
Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align

// GNU readelf (GNU Binutils for Debian) 2.25 (2014):
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x000000000001bad4 0x000000000001bad4  R E    200000

// GNU readelf (GNU Binutils) 2.29.1 (2017):
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x000000000001e050 0x000000000001e050  R E    0x200000
```

Our parser parsed the line after the one containing `LOAD` and expected that `Align` column value will be always prefixed with `0x`. As we can see this is not always the case...

* fix find_fake_chunk #410 (#435)

* Fixes #436 - memory write regression (#437)

* Fix regression made in #432

*This situation pushes me more and more to work on tests engine*

* Fix eX memory write on Python 2

As string literal is unicode, in Py2 the code below would fail if `bytestr` is just a `str`, due to `'0'` being unocide literal:

```
bytestr.rjust(size*2, '0')
```

* Tests framework (#375)

* Add prototype of unit tests for pwndbg

* Add test for pwndbg [filter]

* Fix isort, e2e tests, add pytest requirement

* Add comment about not handling exceptions for unittests

* Fixes after rebase

* Fix test_loads_without_crashing

* e2e tests: no colors & loading pwndbg tests

* Fix isort

* Add example of no file loaded test

* Move tests to unit_tests, add binary, add memory tests

* Isort fixes

* Move from e2e/unit tests to tests

* Add info about tests to DEVELOPING.md

* Fix tests

* review fixes

* commands filtering test: check for contents, not for equality

* Add tests launcher bash script

* Change tests launcher name from unittests to pytests

* Cleanup; better test file paths

* Add theme param to disable colors

* Better test_loads

* Skip some tests locally that can run on travis

* Fix test_loads according to travis

* Fix travis tests

* Don't check for IDA Pro if it is dissabled (#439)

* Improve behavior without IDA Pro (#442)

* Improve behavior without IDA Pro

* Fix import order

* Improved IDA Pro behaviour more

* Added only_after_first_prompt decorator

* Removed newline after import

* Added documentation

* Improved docstring

* Implement support for ptmalloc's tcache in heap/ (#420)

* Implement support for ptmalloc's tcache in heap/ (#387)

Glibc 2.26 added per-thread cache of free chunks. This implements
new "tcache" and "tcachebins" commands for displaying information about
this cache.

Note this works well only if pthread is linked in the debugged program.
Otherwise gdb cannot access thread-local variables, so it cannot find
address of tcache main struct. One can though find the address, ex. by stepping
through malloc code, and pass it to the new commands.

* Another round of review fixes.

* handle gracefully older libc, without tcache

* use aligned size for consistency with other bins

* Support hex data prefixed with 0x when using eX windbg command (#444)

When using `eX` commands and setting data to hex value prefixed with `0x`, we get an exception:

```
pwndbg> ed 0xffb21ae4 0x55616740
Traceback (most recent call last):
  File "/usr/lib/python3.6/encodings/hex_codec.py", line 19, in hex_decode
    return (binascii.a2b_hex(input), len(input))
binascii.Error: Non-hexadecimal digit found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/dc/installed/pwndbg/pwndbg/commands/__init__.py", line 109, in __call__
    return self.function(*args, **kwargs)
  File "/home/dc/installed/pwndbg/pwndbg/commands/__init__.py", line 200, in _OnlyWhenRunning
    return function(*a, **kw)
  File "/home/dc/installed/pwndbg/pwndbg/commands/windbg.py", line 141, in ed
    return eX(4, address, data)
  File "/home/dc/installed/pwndbg/pwndbg/commands/windbg.py", line 180, in eX
    data    = codecs.decode(bytestr, 'hex')
binascii.Error: decoding with 'hex' codec failed (Error: Non-hexadecimal digit found)
```

This commit fixes this problem so that if the data input has prefix, it is stripped.

* Adds stepret command and nextret docstring (#448)

So that now we can step-until-return-like-instruction like a boss! :)

* Add stepsyscall and rename next_syscall to nextsyscall (#447)

* Add stepsyscall (and stepsc) command

So that one can break at a syscall which is e.g. inside a call.

* Rename next_syscall into nextsyscall

* Display context on next/stepsyscall only if process is alive

* Fix bins command 'There is no member named tcache_bins' (#449)

Bins command fails on a libc that doesn't use tcache at all, e.g.:

```
GNU C Library (Ubuntu GLIBC 2.23-0ubuntu10) stable release version 2.23, by Roland McGrath et al.
```

Here is the output:

```
pwndbg> bins
Traceback (most recent call last):
  File "/root/pwndbg/pwndbg/commands/__init__.py", line 109, in __call__
    return self.function(*args, **kwargs)
  File "/root/pwndbg/pwndbg/commands/__init__.py", line 200, in _OnlyWhenRunning
    return function(*a, **kw)
  File "/root/pwndbg/pwndbg/commands/heap.py", line 255, in bins
    if pwndbg.heap.current.has_tcache():
  File "/root/pwndbg/pwndbg/heap/ptmalloc.py", line 47, in has_tcache
    return (self.mp and self.mp['tcache_bins'])
gdb.error: There is no member named tcache_bins.
```

This commit fixes this issue by checking whether `tcache_bins` field is present in the `malloc_par` structure.

* Fixes bins command (#424) (#450)

The problem was that after some of the recent changes to chain/get to prevent dereferencing too much addresses and having better display when dereferencing limit is 0 (used for bare metal debugging) the bins command displayed wrong results for everything except fastbins.

This was due to the fact we are adding the dereference start address to the list.

This fixes the `bins` command by adding `include_start=True` keyword argument to the `chain.get` function. The `bins` simply uses `include_start=False`.

* Fixes #391 - kills compat.py module (#452)

* Kill compat.py completely (#453)

* Fix IDA 7 unhandled DecompilationFailure (#455)

* version command: show IDA Pro versions (#456)

* version with IDA: proper hexrays detection (#457)

* Fix emulate command crash (#459)

After we added `repeat` functionality for some commands, the emulate stopped to work:
```
pwndbg> emulate
Traceback (most recent call last):
  File "/home/dc/installed/pwndbg/pwndbg/commands/__init__.py", line 109, in __call__
    return self.function(*args, **kwargs)
  File "/home/dc/installed/pwndbg/pwndbg/commands/__init__.py", line 200, in _OnlyWhenRunning
    return function(*a, **kw)
  File "/home/dc/installed/pwndbg/pwndbg/commands/nearpc.py", line 180, in emulate
    nearpc.repeat = emulate.repeat
AttributeError: 'bool' object has no attribute 'repeat'
```

This is due to the fact the command has the same name as argument which is a bool.

* Add filtering for config and theme commands (#458)

* Add filtering for config and theme commands

* Fix isort

* Change IDA xmlrpc default port (#462)

* Fixes #460 - getting SP reg on threaded apps (#463)

* Fixes #460 - getting SP reg on threaded apps

As the issue described: as we cache registers,
we might get their values wrong as we don't invalidate cache when thread is changed.

This leads to showing wrong context stack values in threaded apps.

This commit/PR adds a new memoization solution: `reset_on_prompt` which resets cache on `gdb.events.before_prompt` event.

* Fix isort

* Fix before_prompt event on old GDB versions (#464)

* Fix before_prompt event on old GDB versions

This adds an `EventWrapper` class which behaves similar to gdb events but lets us:
* check whether event is a real gdb event or not
* call event callbacks if it is not a real gdb event

* Better comment

* Fix pwndbg.disasm.near with disabled caching (#465)

Before this changes `context_disasm` produced different display based on memoization settings.

The bug can be seen below:

```
[dc@dc:pwndbg|dev $%]$ gdb ~/test/a.out
pwndbg: loaded 166 commands. Type pwndbg [filter] for a list.
pwndbg: created $rebase, $ida gdb functions (can be used with print/break)
Reading symbols from /home/dc/test/a.out...(no debugging symbols found)...done.
pwndbg> set context-sections disasm
Set which context sections are displayed (controls order) to 'disasm'
pwndbg> entry
Temporary breakpoint 1 at 0x400080

Temporary breakpoint 1, 0x0000000000400080 in _start ()
LEGEND: STACK | HEAP | CODE | DATA | RWX | RODATA
───────────────────────────────────────────────[ DISASM ]───────────────────────────────────────────────
 ► 0x400080 <_start>    jmp    _start <0x400080>
    ↓
 ► 0x400080 <_start>    jmp    _start <0x400080>

Breakpoint *0x400080
pwndbg> python import pwndbg; pwndbg.memoize.memoize.caching=False
LEGEND: STACK | HEAP | CODE | DATA | RWX | RODATA
───────────────────────────────────────────────[ DISASM ]───────────────────────────────────────────────
 ► 0x400080 <_start>    jmp    _start <0x400080>
    ↓
 ► 0x400080 <_start>    jmp    _start <0x400080>
    ↓
 ► 0x400080 <_start>    jmp    _start <0x400080>
    ↓
 ► 0x400080 <_start>    jmp    _start <0x400080>
    ↓
 ► 0x400080 <_start>    jmp    _start <0x400080>
    ↓
 ► 0x400080 <_start>    jmp    _start <0x400080>
    ↓
 ► 0x400080 <_start>    jmp    _start <0x400080>
    ↓
 ► 0x400080 <_start>    jmp    _start <0x400080>
    ↓
 ► 0x400080 <_start>    jmp    _start <0x400080>
    ↓
 ► 0x400080 <_start>    jmp    _start <0x400080>
    ↓
 ► 0x400080 <_start>    jmp    _start <0x400080>
Breakpoint *0x400080
pwndbg>
```

The tested binary can be reproduced with this assembly code:

```asm
global _start

_start:
jmp $
```

Compiled as `nasm -f elf64 code.asm && ld code.o`.

---

About the bug:

The check for multiple identical loops or rets is done using `set(insns[-3:])`. Before this hapens the `insns` is filled with the results of `one(address)`. This calls `get_one_instruction(address)` which is cached until `reset_on_cont`. As a result, when caching is enabled the `get_one_instruction` returns the same `capstone.CsInsn` instances for given address. When it is disabled, we return other instances which are identical.

The problem was that `set(insns[-3:])` creates a set based on `capstone.CsInsn` instances and not on the instruction addresses.

The fix changes this behavior so that we compare last 3 instruction addresses.

* Fix shell commands descriptions (#466)

Before that change all shell commands had a description of `None`.

This was because we used string formatting for a docstring and if one
does so, the string isn't a docstring anymore.

This also fixes `r2` command description.

* Use gdb.VERSION instead of parsing gdb.execute result (#467)

Yay, this is finally there:

```
>>> gdb.VERSION
'8.1.0.20180409-git'
```

* Fix NameError: global name 'abi' is not defined (#469)

* typeinfo: skip failed compile attempts (#470)

* typeinfo: skip failed compile attempts

This fixes the problem that single header files that are not compilable
do not abort the compilation loop.
Errors about the failure are printed anyway be check_output we just
avoid abortion.

* Remove shell=True in subprocess.check_output

* Improve dumpargs command edge cases (#471)

* Adds `-f` alias for `--force` option
* More informative docstrings

* Fix the up and down commands when provided with an argument. (#473)

* Command for calculating PIE offsets (#474)

* PIE command

* Kill compat in piebase command

* Improve piebase command

* Improve piebase command exe name introspection

* No longer rely on executeable segment for piebase

* Fix isort (#475)

* Fixes wrong pc/ip display in context introduced in 9fd5d35 (#477)

Before this PR we could get wrong RIP (like off by one) when single stepping through the code:

```
[...]

 RIP  0x555555559850 ◂— xor    ebp, ebp
───────────────────────[ DISASM ]──────────────────────
   0x555555559850    xor    ebp, ebp
 ► 0x555555559852    mov    r9, rdx <0x7ffff7de59a0>

[...]

pwndbg> i r rip
rip            0x555555559852	0x555555559852
```

The patch fixes the issue by reassigning GDB stop signal handler to getting register values.

* Fixes 476 - segfault handling when using rr project (#478)

* Fixes #476 - segfault handling when using rr project

* Fix isort

* bug fix: tcache bin (#482)

* Fix and enhance xinfo command (#480)

* Instead of unstable parsing of readelf output, use the elftools ELF wrapper for parsing PT_LOAD segments

* Fix #434 xinfo command doesn't show File(Disk) info on non-PIE binaries

Also remove some trailing whitespace

Also fix another bug in xinfo; now it can show the disk offset of all
mmap files, not just the primary executable

* New xinfo feature: Print containing ELF sections for file-backed addresses

* Only print header for ELF sections if at least 1 section contains the address

* Fix bug in section offset calculation when printing containing ELF sections

* Refactor ELF file parsing helpers for cleaner separtion of ELF metadata parsing and enrichment, and a specific use scenario (getting a list of segments/sections containing a given virtual addr). Also makes implementing caching parse results easier

Adjust xinfo command to these API changes

* Fix bug: Reference mem_end instead of file_end

* Don't use underscore variable names; change decorator to reset_on_objfile

* Update xinfo.py

* ptmalloc multiple heaps per non-main arena support, related fixes (#479)

* Multiple ptmalloc enhancements:

* Adds support for multiple heaps per arena for the `arenas` command.

* Names every heap objfile to enable proper coloring in vmmap - fixes 451.

* Refactors the `heap` command to address issue 443.

* Adds comment for HEAP_MAX_SIZE

* Refactors Arena and HeapInfo into classes

* Adds additional comment

* Objfile event dispatching fix (#486)

* Fixes objfile caching bug.

* Disables vmmap exploration when the target isn't alive.

* Resets the objfile cache to the proper type on exit. (#487)

* isort: fix import order to make travis pass (#490)

* Heap: allocator initialization check & global_max_fast bug fix (#485)

* Bug fix: global_max_fast symbol contains the actual value not the address

* heap: return from find_fake_fast if allocator is not initialized

* Bug fix: address method should return the symbol address if it's an intergral symbols

* Revert commit c35152d

* add OnlyWhenHeapIsInitialized decorator

* Update heap.py

* Refactors heap.get_region, adds special case for get_heap_boundaries. (#489)

Occasionally, the [heap] vm region and the actual start of the heap are
different, e.g. [heap] starts at 0x61f000 but mp_.sbrk_base is 0x620000.
Return an adjusted Page object if this is the case. Also changes the
callers of these functions where appropriate.

* Leak offset probing tool (#492)

* PIE command

* Kill compat in piebase command

* Improve piebase command

* Improve piebase command exe name introspection

* No longer rely on executeable segment for piebase

* Leak probing tool

* Fix description for probeleak

* Update probeleak.py

Changed `%x` to `0x%x` in edge case scenario print/reporting.

* Reorder imports

* Improve probeleak printing

* Fix isort (#493)

* Fixes #488: wrong regs display on threaded targets (#495)

Please see #488 (comment) for explanation.

* add vis_heap_chunks (#496)

* add vis_heap_chunks

* Add top_chunk suffix only when needed

* use ArgparsedCommand and pwndbg.arch.unpack + better formatting

* Minor improvements, fix isort

* Run each test in a separate GDB session (#498)

* it would be cool to have tests that run within GDB so that we don't have to parse GDB output and deal with weird problems
* we can't run all tests in one GDB session as `file x; entry; <some pwndbg command>; file y; entry; <some wndbg command>;` may have different results - it seems either us or GDB fails to cleanup everything properly

* Fix nearpc following jumps when used w/o emulation (#499)

* Tests launcher: show passed and failed count

* Build nearpc, emulate, u, pdisass test binaries

* Add tests for emulate, nearpc, pdisass, u

* Refactored disasm and emulator

* Fix nearpc following jumps w/o emulation

* Prevent tests from calling start_binary twice

* Add test for emulate_disasm_loop

* Fix isort

* Add nasm to travis install

* Add --eval-command quit to tests invocation

This should prevent travis from staying in gdb/stalled build when something fails in weird way (like a file is missing)
```
[+] Building 'emulate_disasm.o'
make: nasm: Command not found
make: *** [emulate_disasm.o] Error 127
gdbinit.py: No such file or directory.
pytests_collect.py: No such file or directory.
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
```

* Add test binaries

* Inform about `exception-debugger` on exceptions (#501)

Instead of hiding this feature just for devs who reads our dev guide or just knows that this exists lets make pwndbg development great again and show this command to the world!

* Fixes piebase and breakrva on remote debugging (#500)

Three things here:

1. This fixes `piebase` and `breakrva` commands - a bug with remote targets mentioned in #488 (comment).

2. It also adds a check if result address is still in the memory pages belonging to the given module. This works now as:

```
pwndbg> breakrva main
Offset 0x555555554601 rebased to module /home/dc/pwndbg_bug/a.out as 0xaaaaaaaa8601 is beyond module's memory pages:
    0x555555554000     0x555555555000 r-xp     1000 0      /home/dc/pwndbg_bug/a.out
    0x555555754000     0x555555755000 r--p     1000 0      /home/dc/pwndbg_bug/a.out
    0x555555755000     0x555555756000 rw-p     1000 1000   /home/dc/pwndbg_bug/a.out
```

3. It gives a better output for `piebase`:
```
pwndbg> piebase 1
Calculated VA from /home/dc/pwndbg_bug/a.out = 0x555555554001
```

---

To reproduce the fixed bug, launch any binary on a gdbserver:

```
gdbserver 127.0.0.1:4444 ./a.out
```

Then start a debugging session:

```
gdb -q -ex 'target remote 127.0.0.1:4444' ./a.out
```

and fire e.g. `breakrva 123`.

---

Below you can see the bug case and explanation why it occured:

```
pwndbg> breakrva 1
There are no mappings for specified address or module.
'breakrva': Break at RVA from PIE base.
Traceback (most recent call last):
  File "/home/dc/pwndbg/pwndbg/commands/__init__.py", line 109, in __call__
    return self.function(*args, **kwargs)
  File "/home/dc/pwndbg/pwndbg/commands/__init__.py", line 200, in _OnlyWhenRunning
    return function(*a, **kw)
  File "/home/dc/pwndbg/pwndbg/commands/pie.py", line 61, in breakrva
    spec = "*%#x" % (addr)
TypeError: %x format: an integer is required, not NoneType
```

So what is the issue here?

1. We have the same logic in both `piebase` and `breakrva` - if the user doesn't specify second argument - a module name - we retrieve it with `get_exe_name`:

```python
def breakrva(offset=None, module=None):
    offset = int(offset)
    if not module:
        module = get_exe_name()

    addr = translate_addr(offset, module)
    spec = "*%#x" % (addr)

    # [ ... - some more code, not important here ]
```

2. The `get_exe_name` returns just `pwndbg.auxv.get().get('AT_EXECFN', pwndbg.proc.exe)`. The difference is important here. On the case shown above the `pwndbg.auxv.get()['AT_EXECFN']` returns `./a.out` while `pwndbg.proc.exe` returns the full path: `/home/dc/pwndbg_bug/a.out`.
3. This `module` is then passed to `translate_addr` as can be seen on the code above.
4. The `translate_addr` tries to retrieve memory page (`Page` instance) which belongs to the module:
```python
def translate_addr(offset, module):
    mod_filter = lambda page: module in page.objfile
    pages = list(filter(mod_filter, pwndbg.vmmap.get()))

    if not pages:
        print('There are no mappings for specified address or module.')
        return

    # [ ... - some more code, not important here ]
```
5. The `translate_addr` returns `None` because the `page.objfile` for e.g. binary objfile returns its full path as can be seen below:
```
(Pdb) pwndbg.vmmap.get()[0].objfile
'/home/dc/pwndbg_bug/a.out'
```

6. Because we returned `None`, the `spec = "*%#x" % (addr)` string formatting for breakrva or `print(hex(addr))` for piebase fails.

* Fixes piebase and breakrva on remote debugging (#502)

Fixes the issue caught by ecx86 in:
#500 (comment)

The commands broke when we debugged a remote target which was
hosted on a remote gdbserver (NOT a local one).

This is because we used `pwndbg.proc.exe` (changed in previous commit)
which is a local path to the binary which was then used to filter out
memory pages belonging to the binary.

To fix the issue, the AUXV's AT_EXECFN is used first which was used
before previous commit but the returned path is now normalized
(as in previous version it didn't work because if it returned path './a.out'
it couldn't match it with binary's Page.objfile which was e.g. '/blabla/a.out').

* Bump version (#505)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.