Xbox Game Stream Support

[Max-ES]

As an Xbox user I think it would be pretty nice, if your software would support GTA Online on Xbox One.

There is an option to stream Xbox games to any device (also to a PC) and play them with a controller.

Maybe you could add a controller emulation to do the inputs instead of the current mouse input, so Xbox users could use it, too. I don't think that there are a lot more tasks to make it work.

Thank You!

[MarkusJx]

Well, yeah, that would be a cool feature, kind of. I guess.

Almost all the code required already exists in v1.2.1/unused.
So the part simulating an Xbox controller using VJoy would not be a huge problem.

But - hah, there's a catch, you see - I don't own a console myself, so I couldn't test that on my own, I'd need help for that step.
It does seem like there is not much to it, just let the program accept other "game window titles" and add controller support, but the past has shown, that this can't be the case, right?

So, yes, that is a thing I could work on, but, as I said, I'd need help in order to implement such a feature. If you wanna help, just tell me, that would be great,
But as I am (kind of) working on the next autobet release and I am somewhat fixed on what that version changes (plus I'm not willing to add an extra layer of complexity on top of the current changes) and I'm busy with some other stuff, there's no way I'd start working on that feature before beginning of March (or until v1.3.0 is released).

In the meantime, thank you for your suggestion, message me, if you want to help, if not, well, I'd have to find someone willing to help, which would result in this game stream feature being added later or maybe never, dunno.

[Max-ES]

If I find the time I could try to implement it, but i never worked on a c++ project. I will create a pull request if I manage to create some working code. :)

[MarkusJx]

Well, if you want to work on this, you can give it a try. But I don't expect you to do it all on your own. I'd just need someone to test the changes, to see if they actually work (surprise). I'll merge my devel branch tomorrow or so, that one is way better to work of, the code is cleaner, yeah. Whatever. I mean, maybe I can try to sqush in your suggestions into release v1.3.0, don't know yet. In hindsight, not adding game stream support would somewhat mess up the versioning scheme.

If I should take a look at that thing, I'd need the name of the streaming software (I'll assume it's microsofts xbox app) and maybe some pictures of how it actually looks when you are streaming the game to your pc. An alternative to hardcoding the streaming app's name into the native module would be adding a program selector, where the user can select a custom application which should be used to 'place the bets', so you could just select the (this is an assumption) xbox app instead of "GTAV.exe".

And, just throwing it out there, we'll have to work out the D-Pad or in general, button presses on the controller to:

• Place a bet
• Reset (change from the winnings screen to the betting screen, that one is quite important as it is used to reset when the game is stuck in the 'transaction pending' screen, so it should work without using the onscreen buttons [on pc, this is rightClick])

If I'll decide on putting this into v1.3.0, the earliest I can (seriously) start working on this is in 2 - 3 weeks.
So, if you wanna help and dont want to look into the coding stuff you could also work out (some of) those things and tell me something about the other stuff mentioned above.
I'll let you know, when I start working on this issue.

[Max-ES]

The app is called "Xbox Console Companion" - it's not the normal xbox app. In the resource monitor it's called "XboxApp.exe"

The button presses are as follows:

1. Main Screen -> D-Pad Right -> cursor on "place a bet"-button
2. A -> Single Race Horse Propabillity Screen
3. D-Pad Left -> cursor on last horse
4. (D-Pad Up -> cursor on second last horse)...
5. A -> Select Bet Size Screen opens on the left
6. D-Pad Right -> D-Pad Right -> cursor on "increase bet"-button
7. A x23 (this can be done really fast)-> bet is now 10,000 coins
8. D-Pad Down -> cursor on "place bet"-button
9. A -> Race starts
10. WAIT
11. D-Pad Down -> cursor on "play again"-button
12. A -> Main Screen

[MarkusJx]

Looks great, thank you very much.

Some small notes:

• For the app, I suspected the "Console Companion whatever" app to be the one to game stream, so we're on the same page there. Let's see what I can do with that.
• To place a max bet, you could also (I think) press the Y-Button, that one places 10.000 Chips with just one button press. That's a little bit faster.
• I'll have to work out the timing for the button presses, this will be fun.

Again, I'll let you know when I get started working on this, which will be at some point after next week or so.

[MarkusJx]

So it seems to be working...for now.

I've added an option to the ui to select a custom game executable. You'll have to select "ApplicationFrameHost.exe - Xbox Console Companion" for your custom application. So it is not actually "XboxApp.exe - Xbox Console Companion". This is so weird because the actual window is owned by "ApplicationFrameHost.exe" and not "XboxApp.exe" but the actual program logic is owned by "XboxApp.exe" which is sometimes detected as a window although it is not visible and has no size - and I'm starting to realize this already doesn't make any sense, so let's just agree that Microsoft's Window and API design is just stupid. Just select the right application.

There's also a new option to select a virtual game controller as an input device to the game. In order to use it, ScpVBus has to be installed, instructions on how to do that can be found here.
Also, there may be an issue that the cursor is not moving in-Game, in that case you'll just have to switch back to mouse controls and then back to controller controls to let the game (or whatever) realize that there is in fact a controller connected. At least this happens on the PC version of the game, which might also have something to do with the fact that Rockstar Games is extremely incompetent, but whatever. Additionally, there must be no controller connected to your PC (tested on the PC version of the game, your experience may differ as you are not running the actual game on your PC) in order for this controller control to work.

Some other notes:

• Yes, I know the button clicks are somewhat slow, but I needed to add somewhat long timeouts in order for this to work
• The xbox app size is not properly detected as windows reports back the window size including its drop shadows, but it might still work
• Maybe try using different window sizes or fullscreen for the xbox app if it's not working

If something is not working, please send me some logs and maybe some images generated by the full_debug option (see notes here).

In order to try out the program, just download the latest (successful) CI/CD Artifact from the Artifacts section.

If you have any other questions, just let me know and also tell me if the changes are working for you.

Have a good one,

[Max-ES]

Thank you very much for your work!
I tried it, but it wouldn't run properly. It just jumped between the betting screen and the main screen a few times and then stopped. This happened in full screen mode as well as in window mode.
Here are the logs:
autobet_debug.zip

[MarkusJx]

Ah, an interesting one.

First of all, the switching between the betting and main screen is a standard feature with version 1.3.0 to prevent the game from being stuck in the 'transaction pending'-thingy.

The actual problem is caused by some...let's call it inconsistency in the game:

Comparing these two images, with the first being taken from the PC version of the game and the second being a picture you sent, it is obvious that the UI elements in the console version of the game are placed a bit more to the right/center. I wonder why...

Anyway, I'll see what I can do with that.

[Max-ES]

Maybe that's because some TVs cut away a bit of the image (overscan), which is why there is an option to set the screen size in most games/consoles. So Rockstar wanted to make sure that one can see the important stuff (?).

[MarkusJx]

Interesting. So I've just extended the width of the single "odd images", therefore the numbers should be correctly read by the image detection algorithm. Works on PC.

Just check the CI/CD Downloads to get the new version. No need to uninstall the old one.

And let me know if it works

[Max-ES]

It still doesn't work:
autobet_debug.zip
The picture of the last probability is missing in the logs.

[MarkusJx]

Ah, yes, the good ol' german translation. If the game is set to english, your lowest odd will actually be 'evens', in the german version it is '1/1'... For some reason. I've already added odd translations and this finally confirms...that they don't work - I'll be right back.

[MarkusJx]

Well, that was embarrassing.

On the plus side, the odd translations do work now.
Although the program did crash once for whatever reason, but that's not the main problem here.

[Max-ES]

Ok, new problem:
It goes back to the main screen -> to betting screen -> hoovers over the 5th horse and waits for 36 seconds, thats it, no actual betting -> money earned all time decreases by 100$.

Looks like the inputs are done too fast?
autobet_debug.zip

[MarkusJx]

Yeah, let's just go with that. Should be fixed, the CI/CD build should be done in ~12 Minutes

[Max-ES]

Now the place-the-bet window opens on the right but that's it - the race is not being started. I think its still too fast.
autobet_debug.zip

[MarkusJx]

Just doubled the sleep time after a button press, tell me if that's enough

[Max-ES]

On my first try it worked for one single race, but when it got to the betting screen it stopped after the screen was opened. - Maybe it needs to wait a bit on the start of this "loop".
On all other tries it immediately stopped after clicking on the horse and opening the betting window:
too_fast_betting.zip
I guess it's still a bit too fast...

[MarkusJx]

Okay, let's try with a delay of 600ms. CD build should be done soon.

[Max-ES]

Now it runs one time but stops on the betting screen and this message appeares:

autobet_debug.zip

[MarkusJx]

The function to check whether the game is in foreground was not adapted for custom game applications, sorry, my bad

[Max-ES]

I tried the build "Adapt utils::isForeground for custom game applications autobet-build #163" but it still stops working after one race and shows the message above.

autobet_debug.zip

[MarkusJx]

This is weird but ok. I've removed the foreground check for custom game applications.
The program will from now on just log a warning if the game seems to not be the focused application and the selected game application is not the default one. This may be related to windows' strange window ownership concept for windows store apps.

[Max-ES]

Nice, now it ran great for about 8 minutes. After that it stopped on the main screen (not the betting screen) with the following error message:

I will activate the full debug mode and send you the zip, when i reproduced it.

[Max-ES]

Now there is the old problem (I guess - maybe not):
autobet_debug.zip
It goes back to the main screen, then to betting screen moves left one time to the 5th horse, clicks on it and "stops" without an error or warning at this stage:

Maybe this first left click from the cancel button to the horse should be followed by a longer waiting time because the cursor has to travel a relatively long distance. (longer than from one horse to another) which takes more time.
But maybe it's something else.

Even after a retry it stops exactly at this stage.

[Max-ES]

Another observation I made is, that the streaming quality setting should be at least medium. Otherwise the KNN doesn't recognize anything. :D

[MarkusJx]

And I just wrote an comment the length of the bible containing all sorts of theories...whatever.

So, I'll try creating some kind of custom sleep time for those movements? Maybe.
And we'll just remember to put that thing with the stream quality into the wiki, if this is ever going to work. I mean, I could train the model so it works for worse quality levels, but since you are streaming inside your home network, I don't see a reason to not crank up (who still cranks? Get it? I know, it's hilarious) the quality. Except if you are streaming over wifi with your microwave oven running 24/7 next to your router or if your windows decides to go into slowmo-lan-mode.

If it worked 6-7 times before, did it always stop when the horse was the fifth one? Or was it always a different one?
Just realizing, did it place a max bet on all runs before the failing one(s) or did it skip a bet (place $100 on a random horse) before and it worked fine?

[Max-ES]

:D I put it in low quality mode cause i thought it would maybe lower the latency but even in the very high setting the latency seems to be pretty low.

I meant it worked one time before and did six or seven rounds of betting. There it stopped on the fifth horse where we are still now.
I can't really remember if I ever saw it placing a 100$ bet.

I will start the current race manually and study the further behavior.

[MarkusJx]

Yeah, inside your network the latency should be non-existent, basically, except you are using wifi and your signal is really bad, who knows. Or microsoft uses some kind of online-encryption-validation thing like for example TeamViewer does (straight up just dies when using inside a local network and the internet connection drops), which may depend on your Internet speed. Or you are using some old 10BASE-T Ethernet shit, then, ok, yes, that may be a fair point.

Anyway, there are two possible problems causing this behaviour:

• The sleep time between the d-pad-right clicks is too low therefore the thing happens you suggested
• The *skipping strategy" does not work, if this only happens when a bet should be skipped
• Or both. Maybe. Dunno.

[Max-ES]

I think it's the second point you mentioned.
Just now it played about 5 rounds without problems and got stuck, when it should skip (I believe) as you can see:

autobet_debug.zip
(I had to delete the first 100 pictures because the file was too large)

[MarkusJx]

No skips before that point? Ok, then, I'll try to fix that. I can even try it myself on the PC version of the game, which handles controllers just awfully.

[Max-ES]

I cant remember seeing a skip.

[MarkusJx]

Since I currently don't have gta 5 installed and the epic games servers are not interested in providing proper download speeds, I just winged that one. May work, but probably won't.

[Max-ES]

I tried it and it did do one race and then went to the betting screen -> back to the main screen and stayed there.
autobet_debug.zip

So it was different than the previous behavior.

[Max-ES]

I retried and it got immediately stuck:
autobet_debug.zip
It went back to the main screen -> betting screen -> move left (landed on the cancel button) -> clicked the cancel button -> stuck on main screen

[MarkusJx]

That shouldn't happen. The game finally downloaded, so I'll try it myself, hang on

[Max-ES]

Yeah, got the same behavior on the second retry (still with the same horses).

[MarkusJx]

Ok, that was stupid, but it works now. At least for me it does.

[Max-ES]

Okay, so far it is running pretty well!

But one time it skipped even though there was a single horse with the highest chance to win:

autobet_debug.zip

Is that desired?

Another thought:
When there are two horses with the same probability e.g. 3/1 and the next lower probability is 9/1, wouldn't it make sense to bet on one of those two horses?

[Max-ES]

Do you save statistics (like [name of all horses, probability of all horses, outcome] - per race ) to a file to be able to analyse such things? That would be cool. :D

[MarkusJx]

Finally assigned myself, as if that matters...

And yes, the skip was intended, just take a look at the betting rules in the wiki, I think it is explained well enough in there. I always wanted to publish the statistics I (with the help of some other guy) have created, but that is some kind of a hassle with no reward (so if there's no pot 'o Gold at the end of the rainbow, I'm not interested, so the urban legend goes).

For the statistics thing, I also thought of that, it would be pretty cool, but again, that would be a huge pain in the ass for literally nothing. It would be just some amazing looking statistics and that's that. So I could add that to the list of long-term goals to add if I'll ever be really bored, to have something to do. But my time would be better spent converting the whole ui code to react code to make it look less garbage.

[Max-ES]

Okay, understandable. :D
Thank you very much for your great work!

[MarkusJx]

Does it work now?
If so, do we want to try lowering the sleep times between the button presses to speed up the betting process?
I mean, the mouse controls are way faster than the controller ones.

[Max-ES]

It does work.
Yes, we can try that. Maybe you could make the sleep times configurable through the UI?

[MarkusJx]

Ok, I've made some changes:

• The navigation strategy (mouse or controller) are now persisted across program restarts
• The same happens with the game application window name
• There are now two new settings in the ui:
  • An option for setting the time to sleep between button press/release actions (named "click sleep")
  • An option for setting the time to sleep after a button click (named "after click sleep")
• Those settings will also be persisted
• The game foreground detection should work now, but is still disabled for custom game applications
• The logging is less... "spammy"
• The double-clicks of the controller controls are now removed, the expected behavior created using these constructs may now be recreated by using the new sleep options
• The click sleep times are saved independently for the mouse and controller navigation strategies

This was all made possible by expanding end enhancing™ the settings.

A quick note on that last change: In order to change the settings for your desired navigation strategy you must first select the navigation strategy, the values in the specific text boxes may change to the preset (or previously set) values for this strategy and can then be edited. The settings are saved by pressing enter and the change will be confirmed with one of this snackbar notifications as usual.

Those changes seem to be working for me, let me know if that is also the case with your "streaming setup".

[Max-ES]

I set the click sleep and after click sleep to 0 and it worked.....

[MarkusJx]

Did you save using enter? Did you see the 'Settings saved' confirmation dialog? Because it didn't work when I tried that. It just...did not sleep. Surprise.

[Max-ES]

Ups, i didn't press enter, my bad. I figured out that a click sleep of 50 and an after click sleep of 350 works pretty well on my setup. (Xbox wired to the router, Notebook connected via WLAN)
With a bad connection this might be too fast but using that with a similar setup there shouldn't be any problems, I think.

[MarkusJx]

Well, that was not really your fault, the design of the whole thing was a bit misleading, I guess. From now on, if you unselect any of these setting textfields, the value will revert to the (actual) current setting and a message will pop up informing you about the change. Or the fact that nothing was changed.

[MarkusJx]

So is everything now? So I'll just set your values as the default values and close his issue on merging the branch, if that's ok with you.

[Max-ES]

Yes, I guess that is it.

[MarkusJx]

The changes are now merged into the master branch, so thank you for your patience and have fun with the new additions, I guess.
Let me know if you have any further suggestions to improve this tool.