Add support for programmatic IR input

[adamviola]

I'd like to use Dolphin as a reinforcement learning environment, and the game I'm planning on working with requires Wii IR input, which appears unsupported.

Would it be possible to support IR input? The only alternative I can find is programmatically editing the TAS Input window.

[Felk]

Sure it's possible. I'm not sure what would be a good set of parameters to control the IR though. I see multiple possibilities:

• setting the 3D position + rotation of the remote
• setting the 2d screen positions + rotation + cursor size
• setting the raw IR data
• or maybe something else entirely

what would be preferable for your use case?

[adamviola]

Ideally, I would like to be able to instantly position the cursor at any location via x and y coordinates (similarly to the TAS input window). I assume this is setting the raw IR data? If not (and out of curiosity), can you point me to any docs or code that explain how that works?

[Felk]

Ideally, I would like to be able to instantly position the cursor at any location via x and y coordinates (similarly to the TAS input window). I assume this is setting the raw IR data?

Yeah, that's how the TAS input currently works, more or less. It uses the values to calculate 4 positions that are then sent from the emulated wiimote as the signal for "this is where I see the 4 infrared lights from the sensor bar". I have a few images in an old pull request of mine where I tried to fix the current weird offset: dolphin-emu#9018

I can probably do that quite easily. But the x and y values are arbitrary and don't resemble pixels on screen or anything. Would that be problematic? If yes I could try to reverse the calculation the Wii does to turn the IR signal into pixels on the screen.

[Felk]

for reference, the Wii TAS input does it like this (WiiTASInputWindow::GetValues):

    u16 y = m_ir_y_value->value();
    std::array<u16, 4> x;
    x[0] = m_ir_x_value->value();
    x[1] = x[0] + 100;
    x[2] = x[0] - 10;
    x[3] = x[1] + 10;

so basically just taking the x value from the GUI and calculating 4 points (x-10, y), (x, y), (x+100, y), (x+110, y) as the simulated IR lights

[adamviola]

This approach would definitely work for me! It would be great if the x and y values corresponded to pixels, but if it ends up being too difficult, no problem. Worst case scenario, I can just manually define which values of the arbitrary x and y approximately correspond to the edges and work with that.

[Felk]

I didn't bother with screen coordinates for now because that seems to require endless calibration. I don't know how I could make that any accurate, since there's a lot of smoothing going on. Instead, please have a look at the new function set_wii_ircamera_transform: https://github.com/Felk/dolphin/blob/master/python-stubs/dolphin/controller.pyi#L82

Here's a build with that change:
dolphin-python-scripting-with-wii-ircamera-transform.zip

[adamviola]

Awesome! The function looks exactly what I need, but I found some issues when trying it out.

The cursor does not show when I set the buttons of the controller before setting the IR camera transform. For example, this won't work.

buttons = controller.get_wii_buttons(0)
buttons['A'] = True
controller.set_wii_buttons(0, buttons)

controller.set_wii_ircamera_transform(0, 0, 0, -2)

Everything works fine if I run set_wii_ircamera_transform before set_wii_buttons.

Also, math functions don't work.

# Works, cursor is visible
controller.set_wii_ircamera_transform(0, 0, 0.0, -2) 

# Doesn't work, cursor is not visible
controller.set_wii_ircamera_transform(0, 0, math.sin(0), -2) 

[Felk]

Hmm, I've debugged a bit but couldn't figure out yet why it is behaving so strangely. I'll get back to you once I know more

[Felk]

The entire input manipulation code was refactored, mostly upstream, so there no longer is direct IR data manipulation. Instead, there's an API to set exact screen coordinates now. Can you take a look at the set_wiimote_pointer in the latest python stubs and try again? And do you require a prebuilt binary to test?

[adamviola]

Awesome, I can give it a go. Yeah, a build would be great

[Felk]

Here's a windows binary for the newest changes: https://github.com/Felk/dolphin/releases/tag/scripting-preview3

[adamviola]

Thanks! set_wiimote_pointer works great. The y-value is nicely calibrated such that -1/1 are the top/bottom of the screen. However, I found that I needed to scale the x-value a bit; it's more like -0.8/0.8 to reach the left/right edge (at least for the game I'm using). Thanks for your work!

Also, I've noticed an issue where the inputs are inconsistent when provided in an on_framedrawn callback. Looks like that issue is already well-documented here.

[Felk]

I'm glad it works nicely for you!

The y-value is nicely calibrated such that -1/1 are the top/bottom of the screen. However, I found that I needed to scale the x-value a bit; it's more like -0.8/0.8 to reach the left/right edge (at least for the game I'm using).

I'm interested in what game that is. In theory the x axis (left/right) should be pixel perfect, and the y axis may have a slight drift upwards or downwards depending on whether the sensor bar is configured to be on the top or bottom. I tested using Pokemon Battle Revolution

Also, I've noticed an issue where the inputs are inconsistent when provided in an on_framedrawn callback. Looks like that issue is already well-documented here.

Yep, I'll treat that as an issues with framedrawn rather than anything else for now