Botty No Longer Detects D2R Window When Running on Host and D2R Running in VM

[codahq]

Botty Version
Current master as of 2022-02-17

Describe the bug
Botty can't detect the D2:R window. I reported this would be a problem. For reference here:
#453

It is related to that PR and other changes with python using the Win32 api instead of templates.

To Reproduce
Steps to reproduce the behavior:

1. Start botty.
2. It fails to detect the D2:R window.

Expected behavior
It detects the D2:R window.

Additional info
@AlekseyLesnoy, @VladimirMakaev, @mgleed I thought the agreed approach was that the Win32 api would be used first and if it failed it would default back to the old method. It is not defaulting back to the old method. Instead, it doesn't find the window at all and cannot start. I am only finding this now because my VM and other host machine are running compiled versions which I didn't update until today.

Potentially related commits or pulls
95c4431
#483

[VladimirMakaev]

@aeon0 decided to remove asset matching support in this commit to simplify the code and also removed monitor configuration requirements

8907a46

What is your use case for needing this? As I tried to control VM window from host based on template matching but wasn't able to as the mouse and keyboard wouldn't pass on the events correctly

[codahq]

if the goal of the pixel bot was to be undetectable a vm won't ever be able to detect a process, sdk, or anything on its host. that's my use case; to be extremely undetectable. the only option for detection would be forensics on mouse movement and character behavior.

mouse and keyboard won't passthrough unless you run python/botty from an elevated command prompt so open a command prompt/powershell as adminstrator.

[VladimirMakaev]

@codahq are you using Hyper-V window? I'm thinking of a fix that would allow to configure a window to search. E.g. by title or by process.

[Ezro]

@codahq Not trying to be that guy, but what's the advantage / point to sending mouse and keyboard inputs from your host into the VM?

In an ideal setup, imo, Botty should be running within the VM that way you can close it and let it do its own thing without overriding your host.

[VladimirMakaev]

@Ezro WinAPI events from our keyboard & mouse can be detected that are not coming from physical device, which can theoretically flag activity for botting. By using VM you'd channel the input through the virtual drivers that would be detected by the game as actual input devices. So the game would have 0 way to detect if any mouse or keyboard automation is used. I'm actually interested in the use case myself as it's a good way to develop and test changes on multiple accounts simultaneusly.

[codahq]

@codahq are you using Hyper-V window? I'm thinking of a fix that would allow to configure a window to search. E.g. by title or by process.

yes, i'm using hyper-v since it's the cheapest, most available way to setup gpu partitioning.

@codahq Not trying to be that guy, but what's the advantage / point to sending mouse and keyboard inputs from your host into the VM?

In an ideal setup, imo, Botty should be running within the VM that way you can close it and let it do its own thing without overriding your host.

i just answered this. my comment above i already explained this. in the future if the d2:r executable or b.net launcher implemented more detection features they would potentially be able to scan for active processes, executables or sdks, etc. on the system where d2r runs. if it's not running on the system because it's running on the host it won't be detected that way.

maybe your ideal setup is running everything in the same vm but it is not mine. i've been running botty for a long time with d2r in a vm and botty on the host. the commit 8907a46 as VladimirMakaev pointed out changed that.

[Ezro]

@Ezro WinAPI events from our keyboard & mouse can be detected that are not coming from physical device, which can theoretically flag activity for botting. By using VM you'd channel the input through the virtual drivers that would be detected by the game as actual input devices. So the game would have 0 way to detect if any mouse or keyboard automation is used. I'm actually interested in the use case myself as it's a good way to develop and test changes on multiple accounts simultaneusly.

@VladimirMakaev @codahq

Here's a simple keylogger to output the WinAPI SetWindowsHookExA (wrapped by keyboard lib):

import keyboard
import os

keyboard.add_hotkey('f12', lambda: os._exit(1))

def print_pressed_keys(e):
    print(f'{code for code in keyboard._pressed_events}\t{e}')
	
keyboard.hook(print_pressed_keys)
keyboard.wait()

From me running the keylogger and Botty, I can see that the keystrokes look identical:

# Botty
<generator object print_pressed_keys.<locals>.<genexpr> at 0x000001FDAC5D6200>  KeyboardEvent(g down)
<generator object print_pressed_keys.<locals>.<genexpr> at 0x000001FDAC5D6200>  KeyboardEvent(g up)
<generator object print_pressed_keys.<locals>.<genexpr> at 0x000001FDAC5D6200>  KeyboardEvent(f down)
<generator object print_pressed_keys.<locals>.<genexpr> at 0x000001FDAC5D6200>  KeyboardEvent(f up)
# Me
<generator object print_pressed_keys.<locals>.<genexpr> at 0x000001FDAC5D6200>  KeyboardEvent(f down)
<generator object print_pressed_keys.<locals>.<genexpr> at 0x000001FDAC5D6200>  KeyboardEvent(f up)
<generator object print_pressed_keys.<locals>.<genexpr> at 0x000001FDAC5D6200>  KeyboardEvent(g down)
<generator object print_pressed_keys.<locals>.<genexpr> at 0x000001FDAC5D6200>  KeyboardEvent(g up)

Do we have any evidence or way to prove that the inputs are detectable?

edit: Also, the opposite is also interesting: do we have any evidence to show / prove that inputs from the host into the VM are received as physics presses and not simulated?

[VladimirMakaev]

@Ezro boppreh/keyboard#21

[codahq]

i didn't mention inputs but i thought it was a very interesting additional detection method that could potentially be leveraged. i'm referring more to common cheat detection methods. like pb for example would basically act like a virus scanner and scan the hard drive. or vac scans memory and processes and uses signing code detection. if it's not on the system it's not going to be detected and it doesn't matter how innovative blizzard gets because it just won't be there to detect. the only option they would have for detection would be forensics. don't focus on just the inputs because while that's also an important aspect it's bigger than that.

[VladimirMakaev]

My personal usecase for this is 1 window with a paladin, another window with a sorc on different accounts. And I'm testing code on one or another without relogin

[Ezro]

Both of those makes a lot of sense. Thanks for the explanation

[VladimirMakaev]

@codahq Please test if PR is solving a problem for you. Also checkout how to configure settings for this

[codahq]

@codahq Please test if PR is solving a problem for you. Also checkout how to configure settings for this

@VladimirMakaev yeah, the pr fixes the issue of finding the window and gets me back up and running. i really appreciate the willingness here to provide something to accommodate this existing use case.

i wish i had collected some data on the compiled version of 0.6.4 before i switched to pulling branches because it feels like i'm having more issues related to performance afterwards. but i've also introduced a lot of variation that could be throwing that off. i switched from my fake lightning sorceress to a teleporting hammerdin. and i'm obviously coming from 0.6.4 to 0.6.7. is OCR running now along side template matching? or is having to using an offset or anything done for this PR potentially adding more processing time? i feel like every time it searches for a lot of templates (like chaos navigation) or does a lot of looting like the platforms in chaos i get more slowdowns than i did on the sorc. is that possible or am i just worrying too much?

[VladimirMakaev]

I'll close this issue as the problem is #resolved now and it's merged into master. I'd try to decompose the other concern you've raised and submit a specific issue with description / comparison. But in the nutshell a bunch of things changed between releases and it's been a long time so there could as well be a performance problem that we don't know about