Memory Leak, and never ending disassembe.

[Nekketsu]

Y try to disassemble 64 bit version of notepad.exe. Depending the length of the buffer being disassembled, it never ends disassembling, and the memory consumed doesn't stop growing:

using Gee.External.Capstone;
using PeNet;
using System;
using System.Linq;

namespace PortableExecutable
{
    class Program
    {
        static void Main(string[] args)
        {
            var pe = new PeFile(@"C:\Windows\Notepad.exe");

            var entrypoint = pe.ImageNtHeaders.OptionalHeader.AddressOfEntryPoint;
            var entrypointAddress = entrypoint + pe.ImageNtHeaders.OptionalHeader.ImageBase;
            var offset = entrypoint - (pe.ImageNtHeaders.OptionalHeader.BaseOfCode - pe.ImageNtHeaders.OptionalHeader.SizeOfHeaders);

            var binaryCode = pe.Buff.Skip((int)offset).Take(100).ToArray();

            var disassembler = CapstoneDisassembler.CreateX86Disassembler(Gee.External.Capstone.X86.X86DisassembleMode.Bit64);
            foreach (var instruction in disassembler.Disassemble(binaryCode, (long)entrypointAddress))
            {
                Console.WriteLine($"{instruction.Address} {string.Join(string.Empty, instruction.Bytes.Select(b => b.ToString("x2"))).PadRight(16)}\t{instruction.Mnemonic}\t{instruction.Operand}");
            }
        }
    }
}

If I change the line:
var binaryCode = pe.Buff.Skip((int)offset).Take(100).ToArray();
To:
var binaryCode = pe.Buff.Skip((int)offset).Take(99).ToArray();
Or even 104 bytes .Take(104) it works perfectly.

The equivalent example in python works perfectly:

import pefile
from capstone import *

pe = pefile.PE(r"C:\Windows\Notepad.exe")
entrypoint = pe.OPTIONAL_HEADER.AddressOfEntryPoint
entrypoint_address = entrypoint+pe.OPTIONAL_HEADER.ImageBase
binary_code = pe.get_memory_mapped_image()[entrypoint:entrypoint+100]
disassembler = Cs(CS_ARCH_X86, CS_MODE_64)
for instruction in disassembler.disasm(binary_code, entrypoint_address):
    print("%s\t%s" %(instruction.mnemonic, instruction.op_str))

[9ee1]

Hello. That's interesting that it enters an endless loop. Some questions to help me out:

1. Is this a .NET Core or a .NET Framework project, or does it happen on either?
2. Are you compiling your .NET assembly as X86 or X64?
3. Are you using the packaged capstone.dll or are did you replace it with your own?

[9ee1]

So, I reproduced your example on a .NET Core X64 application and I confirm the bug.

The internal disassemble loop never terminates starting with offset 98 in the buffer. What's weird is the internal call to Capstone's cs_disasm_iter() function succeeds but it never updates the address of the internal pointer. The condition to exit the disassemble loop depends on that internal pointer being updated so that it can determine if the entire buffer has been disassembled or not. So it keeps trying to disassemble the same instruction over and over. I'll need to debug further to see why.

I don't have a Python environment handy. Is it possible for your to attach a text file with the output produced from the equivalent Python example? Just 1 instruction per line that prints the address, mnemonic, and operand string should be enough to help me out. Thanks!

[Nekketsu]

Thank you for the fast reply.

I am getting the issue also with a .NET Core X64 application.

To execute the same with Python I executed the following commands:

pip install pefile
pip install capstone

Then I ran the Python code that I included in the first post of the thread.

This is the output I get with Python which works fine:

sub rsp, 0x28
call 0x14001b564
add rsp, 0x28
jmp 0x14001ac68
int3
int3
int3
int3
int3
int3
mov qword ptr [rsp + 8], rbx
mov qword ptr [rsp + 0x10], rdi
push r14
sub rsp, 0xb0
and dword ptr [rsp + 0x20], 0
lea rcx, [rsp + 0x40]
call qword ptr [rip + 0x1c25]
nop
mov rax, qword ptr gs:[0x30]
mov rbx, qword ptr [rax + 8]
xor edi, edi
xor eax, eax
lock cmpxchg qword ptr [rip + 0xb132], rbx
je 0x14001acb2
cmp rax, rbx
jne 0x14001acc7
mov edi, 1

Thank you!

[9ee1]

Thanks. I'll hopefully have an update for you soon.

[9ee1]

OK. I found the source of the bug. The P/Invoke function signature for cs_disasm_iter() needed an explicit attribute to tell the .NET Marshaller to marshall its return type as a 1 byte boolean since by default it assumes its a 4 byte WinAPI BOOL.

After looking at the binary code, it became clear that there is no valid instruction to disassemble at offset 98 (the offset I noted earlier where it started looping forever). At first I thought it was a bug in Capstone itself since cs_disasm_iter() was indicating it was succeeding. But that couldn't be it because of the working Python example you posted.

Upon further debugging, I realized that it was the .NET Marshaller itself that was incorrectly marshalling the return value of cs_disasm_iter() as a boolean true because of the absence of the explicit attribute I noted above.

I'll push a fix and publish an updated NuGet package shortly. I'll close this ticket once I do that. Thank you for your report and patience.

[Nekketsu]

Thank you for the great job!

Looking forward to the updated NuGet package :)