Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

race condition with conditional breakpoints #1764

Open
mrexodia opened this issue Oct 15, 2017 · 8 comments
Labels
bug

Comments

@mrexodia
Copy link
Member

@mrexodia mrexodia commented Oct 15, 2017

  1. Place a breakpoint on a function that is called from multiple different threads at the same time
  2. Set the breakpoint condition to 0
  3. Run, the program will throw an EXCEPTION_BREAKPOINT or EXCEPTION_SINGLE_STEP
@mrexodia mrexodia added the bug label Oct 15, 2017
@torusrxxx

This comment has been minimized.

Copy link
Member

@torusrxxx torusrxxx commented Nov 5, 2017

The problem can be greatly reduced by setting process affinity(use single core).

@mrexodia

This comment has been minimized.

Copy link
Member Author

@mrexodia mrexodia commented Nov 18, 2017

Example code (put conditional breakpoint on increase() call, probably needs multiple cores):

#include <Windows.h>

static volatile int gCounter = 0;

extern "C" __declspec(dllexport) void increase()
{
    gCounter++;
}

static DWORD WINAPI ThreadProc(void*)
{
    while(true)
        increase();
    return 0;
}

int main()
{
    CreateThread(0, 0, ThreadProc, 0, 0, 0);
    CreateThread(0, 0, ThreadProc, 0, 0, 0);
    CreateThread(0, 0, ThreadProc, 0, 0, 0);
    ThreadProc(0);
}
@LFriede

This comment has been minimized.

Copy link
Contributor

@LFriede LFriede commented Dec 26, 2017

I don't know if this information is useful for you, but in case it is you know it now ;)
I encountered the same behaviour while trying to log heap frees. Like described in #1245 while bpx is on RtlHeapFree and "Break Condition" = 0 (Log Text was "RtlFreeHeap: {p:esp+c}" at this moment, the other fields are empty) the debugger begins to show EXCEPTION_BREAKPOINT to me.

The thing i noticed additionally is that sometimes it breaks with EXCEPTION_ACCESS_VIOLATION on EIP = RtlFreeHeap+1. Looking with cheat engine at the memory location there is the 0xCC in the right place and the stuff after 0xCC disassembles to a call dword ptr[] to nowhere in my case which is the reason of the access violation.

Second thing is that sometimes it breaks with "Breakpoint not in list" at RtlFreeHeap-1.

Hope this helps =)

@mrexodia

This comment has been minimized.

Copy link
Member Author

@mrexodia mrexodia commented May 12, 2019

The only possible fix for this issue (that I can see) is to redirect EIP/RIP to a separately allocated section in the debuggee.

The requirement for this is to correctly relocate instructions, which requires quite a significant effort.

It would look like this:

  • When writing the 0xCC on the original location, first rewrite the instruction to a code cave, followed by an absolute jump to the instruction after the original one. For many instructions this just means copying, but for example a call instruction (or any other instruction related to EIP/RIP for that matter) needs some sort of emulation to work. There is also the challenge of exceptions, because if the debugger observes an exception thrown in this code cave it needs to patch up the CONTEXT and do other things to be transparent to the debuggee.
  • Once the breakpoint is hit and resumed, change EIP/RIP to the code cave without removing the 0xCC bytes.

This kind of functionality is difficult to implement completely transparently for the debuggee and is not dissimilar to what tools like PIN or DynamoRIO do. However, if it works it is also possible to emulate the instruction instead.

Another attempted solution is to suspend all other threads once a breakpoint is hit, but this does not affect debug events that are already queued, so suspending a thread will (in theory) stop it, but in practice there could already be debug events for those threads in the queue, giving very confusing results.

@thatreguy

This comment has been minimized.

Copy link

@thatreguy thatreguy commented May 12, 2019

I've seen .bpsync to work painlessly

@mrexodia

This comment has been minimized.

Copy link
Member Author

@mrexodia mrexodia commented May 12, 2019

Interesting... I will try to make a standalone example to reproduce the issues I found, because I'm fairly sure calling NtSuspendThread on all other threads didn't work.

@Mattiwatti

This comment has been minimized.

Copy link
Contributor

@Mattiwatti Mattiwatti commented May 12, 2019

Note that freezing a thread != suspending it. I'm very curious how .bpsync works because there is no public API (user or kernel) to freeze a thread.

@flarejune

This comment has been minimized.

Copy link

@flarejune flarejune commented May 14, 2019

You can try this way,like InlineHook's Trampoline,Example:

005521CD | A1 78563412 | mov eax,dword ptr [12345678]
005521D2 | 40 | inc eax
005521D3 | A3 78563412 | mov dword ptr [12345678],eax

Now set the breakpoint at 0x005521CD,the debugger calls the "VirtualAllocEx" API,Within 2GB of the module memory address,Allocate a temporary memory,
copy "A1 78563412| mov eax,dword ptr [12345678]" to temporary memory(need to fix relative address),write to temporary memory(call WriteProcessMemory API):

0600000 | A1 xxxxxxxx | mov eax,dword ptr [12345678]
0600005 | E9 xxxxxxxx | jmp 0x005521D2

When the trigger breakpoint at 0x005521CD, do not restore the "0xCC" command of 0x005521CD,When the thread wants to resume running, set RIP = 0066000 (call SetThreadContext API) and continue running.

This is the only way to resolve thread concurrency

@mrexodia

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.