InputCommon: Dynamic Input Textures

[iwubcode]

Per game input profiles make it easy to play a variety of Wii games. However, due to the need to custom tailor input for most Wii games, it can be difficult to remember what input keys map to what. It can be even more confusing when you haven't played a game in a while! Some users have created texture packs that overwrite buttons in the UI with gamepad specific buttons but that can be frustrating because it forces a control scheme. Enter "Dynamic Input Textures"!

Dynamic Input Textures allow texture creators to replace areas of a texture using a bunch of color boxes. When the input keys are loaded, a custom texture gets generated with images representing the keys you have set!

Example video

Note: A big thank you to Jakey455 on the Dolphin forums for the XboxOne textures I used in my example

[lioncash]

Just a cursory review of the WIP code

[iwubcode]

Thank you very much @lioncash . I addressed those changes and fixed two of the outstanding items (updated the configuration in the first post). Still need to do some more testing and figure out how to get it to build on the buildserver!

[iwubcode]

I did a lot of testing this weekend. Games like Deadly Creatures, Go Vacation, etc use a bunch of smaller textures. These are easier to "map" but require more json descriptor files, whereas Nintendo's own games often use a single texture for all of the controller icons. One interesting thing to note for Spiderman Shattered Dimensions is all of the controller textures had a black background. This seems like a colorkey but even making my replacement textures have a black background did not show the correct transparency.

[Starmann]

This is one of the most creative pull requests I've seen in all my months of stalking Dolphin's pull requests for new features. I just have a few questions:

1. Have you tested it in Netplay? I'd imagine that could mess with things.

2. How would it handle the replacement of, say, the c-stick with arrow keys? What about a completely bizarre combination of keys that no sane person would choose?

3. Did you consider the possibility of this config being expanded for other dynamic forms of texture replacement when designing the format?

[stenzek]

Netplay shouldn't be an issue, as all of the replacement is being done in VRAM. It'd only be an issue if an EFB copy was taken with a replaced texture, EFB2RAM was enabled, and the game was reading the copy with the CPU and made some decision based on the content which caused it to desync. Basically, it's the same as custom textures here.

[Starmann]

@stenzek Yes but I'm wondering if the texture in question changes based on your controls in Netplay.

[stenzek]

The replacement is done locally. So I imagine it would be per-player.

[iwubcode]

@Starmann - let me try to answer some of your other questions.

How would it handle the replacement of, say, the c-stick with arrow keys?

Assuming it's a single c-stick texture, you could add 4 colors to represent each input direction. Then assuming the texture creator accounted for the arrow keys, it would just work, it might look a little weird though! Ideally, the game would have individual textures for each direction.

What about a completely bizarre combination of keys that no sane person would choose?

The texture creator has to define the button replacement textures. If the button is not accounted for, that portion of the texture will be pulled from the "fallback texture".

One thing I decided not to deal with, is trying to build a texture dynamically based on the expression provided. So button A & button B does not automatically create a texture with both keys in it. However, if the texture creator thinks an expression is likely to be used (or is requested) they can map that to an image.

Did you consider the possibility of this config being expanded for other dynamic forms of texture replacement when designing the format?

It's pretty hard wired to inputs at the moment. But a similar idea could be implemented if there was a good use-case. The thought did cross my mind.

[Miksel12]

Sounds like a fun idea but how are you going to manage mappings with multiple buttons? Eg 'Button 1' | 'Button 2'

[iwubcode]

@Miksel12

how are you going to manage mappings with multiple buttons? Eg 'Button 1' | 'Button 2'

Like mentioned above, if the expression is supported in the pack, then it will just work. However, in practice, I expect most texture creators will just map the normal buttons. That means if you want to map shake to 'Button 1' | 'Button 2' you will be out of luck. But if you want to map it to 'Button 3', it will work perfectly.

It's unfortunate but since expressions are evaluated at runtime, they don't fit well with the current implementation of Dynamic Input Textures which get generated when input configuration changes.

[iwubcode]

Updated now to support multiple textures in a single configuration. This can be useful if the textures are going to use the same host controls.

I also found out why Spiderman Shattered Dimensions has odd black around it. I believe there is another texture that causes the shadow to appear and that needs to be updated for the new icons.

I might test another game or two. But the next step will be figuring out how to reduce openCV to the bare minimum needed to build.

[iwubcode]

I moved away from opencv, so now all platforms build! With that, I think it's ok to open to user testing and general code review. There's room for optimization and more image manipulation features but I'll leave those for another PR unless there's a big push for something from reviewers.

[phire]

Yeah, replacing a 100mb+ dependency with ~500 lines of code is very much the right move.

[iwubcode]

Minor update, as I didn't fully understand the RebuildUserDirectories function when I originally looked at it. Now if the "Load" directory gets rebuilt, we will properly set the dynamic texture directory. I also rebased with latest master.

[iwubcode]

Thanks to some testing from @jordan-woyak , some bugs were discovered with the implementation for Vulkan/OpenGL and also in general when a button gets 'cleared'. The most recent push fixes those issues.

[iwubcode]

With #8628 merged, I now generate textures into a <full gameid>_Generated folder and place a gameids directory with the <gameid>.txt underneath it. This should allow users who happen to have a region free directory with normal textures (that doesn't take advantage of #8628 ) still be loaded (in addition to any dynamically generated textures).

EDIT: also cleaned up the documentation to reflect the ability to name the folder whatever the user likes, removed some unnecessary slashes, and squashed the commit about being able to name directories whatever you want.

[iwubcode]

@jordan-woyak - just curious if there were any other concerns preventing sign-off?

[iwubcode]

@leoetlino - thank you very much for reviewing. All comments have been addressed, please let me know if you see anything else

[iwubcode]

@leoetlino - thank you again for your feedback. All comments are addressed. Please let me know if you see any other concerns.

[jordan-woyak]

Code looks good.

[iwubcode]

I hated to update the code after it was signed off on but I wanted to make this change before the configuration format got set in stone and users were using it out in the wild.

I originally had the pack creator specify a "colorkey" texture and a fallback texture. The color key texture specified color regions and when combined with a mapping between emulated input-key and color, it provided all the necessary information to do the appropriate replacement. My original thought was this was fairly simple to deal with.

However, users found the json cumbersome to write and the colorkey image, while easy was still a pain as you had to figure out what colors weren't in the original image!

I created a tool to simplify writing the json but still the color mapping issue remained....

So, I removed the color specification in the json and replaced it with rectangle regions that mimic Dolphin's rect class.

This has a couple advantages:

• It's faster to process - Dolphin doesn't have to look for the regions by color, we can simply load the configuration up!
• It is more flexible - by using regions, we open up more possibilities.

The downside is, without the tool, it is cumbersome to define those regions. I think it's a fair trade-off but I can revert if needed.

---

I also added the ability to specify the generated folder name. It still defaults to <gameid>_Generated but gives some more flexibility.

---

@jordan-woyak - please take a look at the last commit and let me know if you think this is a good design decision. If so, I'll go ahead and squash the code so it's ready to be merged.

[jordan-woyak]

LGTM.

[iwubcode]

Thank you @jordan-woyak for reviewing! I squashed the commits, so this is now ready to go. I'm putting the finishing touches on the tool.