Use TensorFlow to detect the Heros' hp bar

[ghost]

@Jire
I am looking into your overwatch repo now. I have commented on the issue (#36). I have some idea about using ML algorithm to calculate the position of the enemy characters in Overwatch and I need some help about it.

I went through ur code and it seems u used javaCV to capture the images and have already set a fixed color tolerance. Do u have any idea about how blizzard block health bar capturing in the new patch?

Is it because blizzard forbid the programmatically mouse movements? Or it is because the health bar cannot be detected anymore?

If it is the second one, I can use some TensorFlow algorithm to solve the issue. I can train some model to detect the hp bar in the game.

cc: @allpeople welcome to give any advice on this issue.

[Jire]

It fails for two reasons, Blizzard blocks gdi grab after a while (which I can fix), but a bigger issue is that the HP bar uses randomly generated shades of red that vary within every frame. Machine learning could he used to find the center or left corner, and I'd greatly appreciate any contribution or concept.

[ghost]

Thx for replying. In Progress.

[flengawy]

hello ..
i've downloaded your overwatch aim cheat but unfortunately it's not working , i got everything going fine and i started the game with the batch file but nothing happening when i start shooting , soo can you help me with this ?

[Exaphis]

@Jire @pIRATEhG Since this issue has been inactive for a while, I took it upon myself to explore this idea further. My results can be seen in this repo.

The main issues of using machine learning is the speed of detection as well as the GPU usage. The GPU usage stays around 40% for faster_rcnn_inception_v2, causing stuttering in game if I don't enable VSync. The usage for ssd_mobilenet_v1 was much more reasonable, at around 18%.

The faster_rcnn_inception_v2 model had impressive accuracy, but was slow. The ssd_mobilenet_v1 model was fairly fast but had a few false positives and false negatives. I'm sure you guys can make something out of this.

[Jire]

@Exaphis I thought of a simple center detection method which should be quite fast a few months ago, although I didn't implement it:

Count the number of consecutive horizontal pixels within the color range, then use the center pixel, which is the first horizontal pixel plus half the total spanning red pixels.

[v-b7]

@Jire I actually used the method you mentioned for another game and works fine. I'm guessing OW has different shades of red. If the intensity of red in each shade would fall in a specific interval detecting the hp bar shouldn't be complicated