Hardware mouse cursor

[Hezkore]

I know the flickering mouse cursor issue while moving has been talked about for years, and it's still an issue while playing on a higher end computer.
Playing on a lower end computer and the mouse cursor feels extremely sluggish when the frame rate jumps up and down.

All of these issues are all related to OpenTTD rendering the mouse cursor itself (software mouse), instead of using the native OS mouse cursor (hardware mouse).
Any inconsistency in frame rate or hitches by the game is felt in the mouse movement and is also visually disorienting.

I suggest a simple solution:

1. Add a new "Use hardware cursor" option to the "Game Options" menu.
2. If "Use hardware cursor" is enabled; do NOT call SDL_ShowCursor(0) and never render the in-game cursor.

The standard OS' mouse cursor will be rendered on top of the OpenTTD window and behave exactly like the user is accustomed to, without any flickering or hitches to speak of.

I realize that some OS' version of OpenTTD might use its own way of moving the mouse.
But I believe the Windows version of OpenTTD simply tracks the native Windows mouse cursor position.
Which means this solution will at least work on Windows. (and probably Linux and Mac as well)

[LordAro]

There's been some work in decoupling the cursor (and window) rendering from the main game loop - #6780 . Can you try that out and see if it solves your issue?

[Hezkore]

Sadly I'm in no position to compile OpenTTD myself, so I'm afraid I can't test this.
Though even with a cursor and interface using its own loop; the game will still suffer from some problems with the software mouse.

For example the mouse cursor scales with the interface.
So when using a double sized interface; the cursor will look atrocious.
Which happens to be one of the problems I have with it.
Example:

[nielsmh]

I'm not sure if Windows supports cursors larger than 32x32 pixels or some other limit. The limit might be too small to support the large animated cursors used.

[Hezkore]

Are any of the cursors actually needed though?

Alternatively you can render the software mouse when it's a non-standard cursor.
So the majority of the time the hardware mouse cursor is used, but for demolishing and placing things the software cursor kicks back in.

[Nik-mmzd]

I tried to compile OpenTTD with disabled hardware cursor disabling (commented this line) and default SDL cursor (i think it's default SDL cursor, never worked with SDL) really "faster" that software one
+1 for "Use hardware mouse" option

[PeterN]

The custom cursors often indicate what action will happen when you click. By using the default hardware cursor, this information will be lost.

Some platforms allow setting a custom hardware cursor, which would be ideal.

[Hezkore]

As I said before.
The hardware cursor could always be displayed while any additional pointer information could be drawn in-game.
For example; when demolishing things you could draw the bomb at the cursor while keeping the hardware cursor visible.

Here's how that would look VS. how it looks in-game for me right now:

The pointer itself does not need to be software rendered, it only causes issues.

[Nik-mmzd]

...but "software" bomb near hardware cursor will move slower that cursor and it'll be... >_<

[Hezkore]

Yeah, ideally you'd want to actually change the cursor itself, of course.
But that doesn't seem to be an option (due to size and animation?) so the next best thing is to at least have the cursor hardware. If the UI eventually runs using its own ticks, then things would get better.
Though it'll never be "hardware"-good.
And software information under the cursor is not as disorienting as a software cursor.

But yeah, either you run with everything in software and have a flickering, slow mouse cursor that acts differently depending on FPS, or you convert what you can to hardware and at least fix some issues.

And to be fair!
I've seen multiple triple-A games use the hardware-cursor-with-software-info approach.
In the end, I personally feel like it's a much better solution than pure software.

[PeterN]

I'm not saying it's a bad idea, by the way, just something to take into consideration.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the last two months.
If you believe the issue is still relevant, please test on the latest nightly and report back.
It will be closed if no further activity occurs within 7 days.
Thank you for your contributions.

[nielsmh]

This is still relevant IMHO.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the last two months.
If you believe the issue is still relevant, please test on the latest nightly and report back.
It will be closed if no further activity occurs within 7 days.
Thank you for your contributions.

[Hezkore]

So long hopes and dreams!

[Nik-mmzd]

Meh.

[LordAro]

Stupid stalebot

[Nik-mmzd]

@LordAro maybe you'll add "pinned" label to make Stale bot not mark this issue as stale?

[LordAro]

In theory yes, but we turned stalebot off a few months ago :)

[Nik-mmzd]

Ohh, missed that

[nielsmh]

I experimented with hardware/system cursor on Windows a little while ago, branch here: https://github.com/nielsmh/OpenTTD/tree/win32-syscursor

Two important points:

• Windows supports at most 48x48 pixel cursors, and some OTTD cursors are larger than that.
• Converting the cursors from sprites to Win32 cursors is not quite straightforward.

In my opinion this is not worth chasing further for Win32.

[Hezkore]

I experimented with hardware/system cursor on Windows a little while ago, branch here: https://github.com/nielsmh/OpenTTD/tree/win32-syscursor

Two important points:

* Windows supports at most 48x48 pixel cursors, and some OTTD cursors are larger than that.

* Converting the cursors from sprites to Win32 cursors is not quite straightforward.

In my opinion this is not worth chasing further for Win32.

100% hardware cursor might be out of the question, but see #7006 (comment) for a cross-platform way of handling mouse cursor information.

[KerekesDavid]

In my opinion either Hezkores solution, or at least a simple toggle off would be really welcome. I would gladly lose the bomb icon feedback to have a mouse pointer that feels responsive.

The current one on windows feels really jarring, especially if played in a window, switching between the software and hardware cursors is awful.

[techgeeknz]

What, realistically, needs to be done in order to fully decouple the GUI (and, in particular, the mouse pointer) from the GameLoop? When playing on Android TV (with BT keyboard and mouse), GUI responsiveness annoys me so much that, if I knew where to start refactoring, I might just take steps to make it happen.

[nielsmh]

The GUI can never be fully decoupled, there needs to at least occasionally be frames where the GUI and game state are in sync so the GUI can take a consistent view of the game state and send commands safely.

Presumably the idea would be to move the game simulation into a thread separate from the GUI so the simulation can continue in the background. This makes it unsafe for the GUI to access any of the game state since it will almost certainly by inconsistent. This means the entire game state needs to be wrapped in a mutex, and the GUI code needs a flag it can check for whether it's currently safe to access the game state. This flag could be set by the video driver loop, which will take the game state mutex occasionally, wrapping the GUI handling code, and allow the GUI to actually update from the game state.

Large parts of the GUI would then need to be reworked to contain a copy of all the game data it needs to draw, and landscape viewports would probably need to be double-buffered in some way.
When the player does an interaction with the GUI, that needs to send a command to the game state, the interaction would have to be buffered and only be executed when the game loop next has an interaction, and that could mean that e.g. buttons need to stay in a "down" state for a while until the command executes, and there may need to be some indication that a click with a tool on the map has been recognised but is waiting for the game loop.

This would be a massive change, very difficult to pull off correctly.

[LordAro]

You'd probably have more luck with a system/hardware cursor for SDL than for Win32, if you were interested in just doing that

[techgeeknz]

Thanks for the feedback, guys.

I realise that this would not be simple to pull off (if it were, someone would have done it by now); I'm currently having a look to see just how deep this particular rabbit hole goes. What I had in mind, though, was to at the very least decouple the mouse pointer and window chrome so they can update in realtime (to keep window management things like moving/resizing as responsive as possible); then having the window contents updated asynchronously following the call to GameLoop().

I can think of three things in VideoDriver::MainLoop() that can (potentially) take "too long" to execute in real-time:

1. The call to GameLoop()
2. The repainting of viewports
3. DrawSurfaceToScreen() (this seems to be where the Android port spends most of its time)

The idea is to decouple each of those and refactor DrawSurfaceToScreen so that mouse pointer and window chrome (or perhaps all window widgets with the exception of viewports) are updated on every video frame, the GameLoop() runs as near to real-time as possible (skipping GUI updates if necessary), and the viewports are updated as time permits (perhaps with a deadline of 100 ms).

I was thinking I could decouple the screen updates, such that MainLoop() can redraw the real-time interactive portions of the GUI (i.e. mouse pointer and window frames) to the back buffer (prepending its rectangles to the dirty list, so they take priority), UpdateWindows() can redraw the window client areas to the back buffer (appending rectangles to the dirty list), all while DrawSurfaceToScreen() copies as much as it can (within the real-time constraint) from the back buffer to the front buffer; triple-buffering if necessary to avoid conflicts between UpdateWindows() and DrawSurfaceToScreen(). The job of DrawSurfaceToScreen() would then become one of pumping dirty rectangles to the screen as quickly as possible, with the constraint that real-time updates happen first.

Handling input in real-time would then be the next challenge, of course.

[techgeeknz]

It may very well be that the path of least resistance is to reimplement the windowing subsystem as a full-on compositing window manager. If that's what it takes, and I can figure out a plan to achieve it, then I am prepared to do just that.

[ldpl]

@nielsmh you're overthinking it, in a multiplayer GUI is already effectively decoupled from the real (server) game state and it works just fine.

[techgeeknz]

@nielsmh you're overthinking it, in a multiplayer GUI is already effectively decoupled from the server game state and it works just fine.

That sounds promising. I'll have a look to see what multiplayer does differently.

[nielsmh]

@ldpl What? Each client is still simulating the entire game (except GS and AI scripts) so everyone has the same burden. The server and client are different processes on different computers (typically) hence there is no "shared state", but the client is still replicating the state of the server via computing the full simulation locally.
The other way to decouple the GUI would be to move all the game state out of global variables and into some structure that can be cloned such that the GUI works off a static copy of the game state. (This would likely need the help of some OS memory management or process management magic to be efficient at all.) But it doesn't change the fact that every client still needs to run the full simulation.

[techgeeknz]

You'd probably have more luck with a system/hardware cursor for SDL than for Win32, if you were interested in just doing that

Fixing the mouse cursor to update on every video frame would be a huge step in the right direction and would certainly be better than nothing; I would start with a patch that does just that.

[techgeeknz]

The other way to decouple the GUI would be to move all the game state out of global variables and into some structure that can be cloned such that the GUI works off a static copy of the game state.

I'm still new at this, but doesn't that then get into the domain of autosave and its associated (and noticable) pauses as it clones the game state for its own purposes?

I had briefly entertained the idea of rejigging the game state to be copy-on-write; but that seems like it may be too much effort for too little gain. Correct me if I'm wrong, but doesn't the game state change almost completely between each iteration of the game loop; meaning there would be very little to gain from a copy-on-write strategy vs. cloning the complete state?

[TrueBrain]

Although I fully understand the use-case for hardware mouse cursor, as mentioned above already, this is not trivial for OpenTTD. Other games, take Factorio for example, do use it, but their whole GUI is build up with this in mind. For example, you constantly see where the game thinks where your (hardware) mouse is when you have it on your main screen by showing the tile you are on. This kind of feedback is essential for any game with hardware mouse cursor, and as such, it would require some pretty hefty changes in OpenTTD to make happen. Which is a bit sad, as I can understand having a fluent mouse is important. But .. I do have some good news :D

Lately we have been working on our video drivers. A few things worth mentioning:

• the GUI now runs at 60fps (and can be changed into what-ever you want)
• OpenGL is supported now as backend driver
• we solved the main issue reported by the forum thread mentioned in the first post of this issue (this was a Windows-only issue)

All in all, our video-drivers now respond a lot quicker in most situations. The main issue that is not resolved yet is what happens when there isn't enough time to do everything .. something has to give of course. Currently this is both the game-tick and draw-tick, meaning the game starts to get slower and slower fps. This means that on such game, the mouse will feel more and more laggy again. But, it will be much more consistent in the lagginess, and not as the issue-started described very jumpy. So that should already be a huge improvement. In other words: when a game runs normally, we hit 60fps really consistent. When the game gets too busy, this gradually slows down.

Also, OpenGL driver allows OpenGL to draw the mouse, which is also a lot quicker / more fluent. We have some plans to extend this further, that even during stress the mouse is drawn at 60fps; but that is not yet in master.

In total, with the upcoming nightly (the one of 20210223), the reason to do hardware mouse cursor should be resolved, meaning the software mouse cursor should be quick enough to not be so annoying anymore. As I find it unlikely someone is going to spend a lot of time changing our UI to support hardware mouse cursor (but to be clear, I very much would welcome such Pull Request!), I am going to close this issue. We keep issues for, well, issues, and not for possible future additions, and I think our latest changes have sufficiently mitigated the original problem: a laggy mouse.

If this is not the case, please do let us know. We would love to improve further :D Tnx!