'Y' and 'R' button not detected by SDL applications

[terencode]

Version of xpadneo

v0.8.r515.g715c2e9

Describe the bug

When connecting an 8bitdo controller (tested with both the zero 2 and SN30 pro+) in X Input Mode over bluetooth,
the keys labeled 'Y' and 'R' are not registered in sdl controllermap and antimicrox.

Steps to Reproduce

1. Connect an 8bitdo zero 2 controller over bluetooth in X Input Mode (Start + X)
2. Run jstest/jstest-gtk and notice all buttons are correctly registered
3. Run antimicrox or sdl controllermap and notice the 'Y' and 'R' buttons do not do anything

Expected behavior

All the buttons should be detected.

System information

# uname -a
Linux terence-desktop 5.7.8-18-tkg-bmq #1 TKG SMP PREEMPT Fri, 10 Jul 2020 18:00:53 +0000 x86_64 GNU/Linux

# xxd -c20 -g1 /sys/module/hid_xpadneo/drivers/hid:xpadneo/0005:045E:*/report_descriptor | tee >(cksum)
00000000: 05 01 09 05 a1 01 85 01 09 01 a1 00 09 30 09 31 15 00 27 ff  .............0.1..'.
00000014: ff 00 00 95 02 75 10 81 02 c0 09 01 a1 00 09 33 09 34 15 00  .....u.........3.4..
00000028: 27 ff ff 00 00 95 02 75 10 81 02 c0 05 01 09 32 15 00 26 ff  '......u.......2..&.
0000003c: 03 95 01 75 0a 81 02 15 00 25 00 75 06 95 01 81 03 05 01 09  ...u.....%.u........
00000050: 35 15 00 26 ff 03 95 01 75 0a 81 02 15 00 25 00 75 06 95 01  5..&....u.....%.u...
00000064: 81 03 05 01 09 39 15 01 25 08 35 00 46 3b 01 66 14 00 75 04  .....9..%.5.F;.f..u.
00000078: 95 01 81 42 75 04 95 01 15 00 25 00 35 00 45 00 65 00 81 03  ...Bu.....%.5.E.e...
0000008c: 05 09 19 01 29 0a 15 00 25 01 75 01 95 0a 81 02 15 00 25 00  ....)...%.u.......%.
000000a0: 75 06 95 01 81 03 05 01 09 80 85 02 a1 00 09 85 15 00 25 01  u.................%.
000000b4: 95 01 75 01 81 02 15 00 25 00 75 07 95 01 81 03 c0 05 0f 09  ..u.....%.u.........
000000c8: 21 85 03 a1 02 09 97 15 00 25 01 75 04 95 01 91 02 15 00 25  !........%.u.......%
000000dc: 00 75 04 95 01 91 03 09 70 15 00 25 64 75 08 95 04 91 02 09  .u......p..%du......
000000f0: 50 66 01 10 55 0e 15 00 26 ff 00 75 08 95 01 91 02 09 a7 15  Pf..U...&..u........
00000104: 00 26 ff 00 75 08 95 01 91 02 65 00 55 00 09 7c 15 00 26 ff  .&..u.....e.U..|..&.
00000118: 00 75 08 95 01 91 02 c0 85 04 05 06 09 20 15 00 26 ff 00 75  .u........... ..&..u
0000012c: 08 95 01 81 02 c0                                            ......
3445511648 1458

Controller and Bluetooth information (for the zero2)

xpadneo-btmon.txt

xpadneo-dmesg.txt

[kakra]

[63032.114975] xpadneo: hello there!
[63040.284204] xpadneo 0005:045E:02E0.001D: unknown main item tag 0x0
[63040.284343] input: 8BitDo Zero 2 gamepad as /devices/pci0000:00/0000:00:01.3/0000:01:00.0/usb1/1-2/1-2:1.0/bluetooth/hci0/hci0:256/0005:045E:02E0.001D/input/input57
[63040.284451] xpadneo 0005:045E:02E0.001D: input,hidraw7: BLUETOOTH HID v9.03 Gamepad [8BitDo Zero 2 gamepad] on d4:d2:52:a9:4f:d3
[63492.176904] xpadneo: goodbye!

The log seems strange. Did you mix different xpadneo versions? hello there! should no longer be present in current versions.

[terencode]

Probably bad copy paste, I was trying to know if it's a regression but it doesn't seem to be the case.
Let me know if you want me to redo something with latest driver but I can certify it's the same with the latest version.

[kakra]

Did you manually make some SDL mappings in the past? E.g., look in $HOME/.local/share/gamecontrollerdb.local.txt. Also, do these buttons show up correctly in evtest? Also run env and look for SDL_* variables.

[terencode]

I unset the SDL_GAMECONTROLLERCONFIG and tried the programs again and same behaviour.

[kakra]

050000005e040000e002000001000000

This ID will also match your new controller and apply the same remapping to it. That may be the problem. During v0.8 we will add a remapping capability right into the driver in a way that it will detect the exact device. It will even work correctly with two identical models and still see them as different devices with different mappings.

You should clear the mappings from the file and env and retry. I can add a controller quirk to use Nintendo mapping for the buttons. So A,B,X,Y will match with your controller.

Do all 8Bitdo controllers use Nintendo mappings?

[terencode]

This ID will also match your new controller and apply the same remapping to it

Hmm that's weird I tried it again and while the ID for both the SN30pro+ and the zero2 are the same, they are now different from what they were before (030000005e040000e002000000006800,Xbox One S Controller).
Because the ID is different it shouldn't match right?

Your controller has X and Y swapped, and also A and B. Did you remap with the controllermap program for this reason?

I think so. I guess now that you implemented quirks for these devices it makes sense to fix this within the driver?

Do all 8Bitdo controllers use Nintendo mappings?

I would think so as they are first intended to be used on nintendo games.

[kakra]

SDL matches by ID only. The name is just what is presented to the application, you can rename your controller there if you'd want to. The first part 0300 is the bus number. 0500 is for Bluetooth. You probably connected by USB now. Actually, the order of bytes is swapped: 0300 should be read as 0003, 5e04 should be read as 045e (which is the PID for Microsoft).

If you confirm that all buttons work correctly (aside from being swapped) after you removed the mappings, I'll implement a quirk handler for that. After all, xpadneo should work OOTB without any mappings.

[terencode]

I rebooted after removing SDL_GAMECONTROLLERCONFIG and confirmed it with the absence of anything SDL related within env but the 'R', 'Y', left and right joystick click (only with SN30 pro+) are still not working in controllermap and antimicrox.

[terencode]

This is very strange... I probably need one of those controllers physically at hand to fix this. Or we could try some sort of remoting...

To set things up you can contact me on Telegram (@terencode),Matrix or Discord (Terence#2723) .

You could run with SDL_GAMECONTROLLER_... program-to-run to try if it makes a difference. That sets the environment for program-to-run just for this program while it is running.

Yep that's what I tried but no difference unfortunately.

Do you use the 8BitDo Bluetooth dongle by coincidence?

Nope (using intel Wireless-AC 9260).

lsof won't show you files that have already been closed again. You'd need to run strace -f sdl2-jstest to see the various files it opens and closes.

Thanks tried it but nothing stands out except confirmation it is not finding any gamecontrollerdb file: openat(AT_FDCWD, "gamecontrollerdb.txt", O_RDONLY) = -1 ENOENT (No such file or directory)

[kakra]

Nope (using intel Wireless-AC 9260).

I had some problems with this chipset lately - but on Windows, and for Wifi... It needed a driver update to work correctly. But whatever that problem is and where it is located - it should not affect this specific problem. But maybe worth keeping that in mind.

You'd need to run strace -f sdl2-jstest to see the various files it opens and closes.

Thanks tried it but nothing stands out except confirmation it is not finding any gamecontrollerdb file

Let's see if strace can tell us which event or other files from /dev/ it opens. Do you think you could trace that?

[kakra]

To set things up you can contact me on ... or Discord (Terence#2723) .

https://discord.gg/6RkrZzg

[terencode]

Let's see if strace can tell us which event or other files from /dev/ it opens. Do you think you could trace that?

here you go: sdl2-jstest-strace.txt

[kakra]

An strace of an SDL application would be useful, including udevadm monitor (start this before running strace on a second terminal) from the same testing session. Let's see how SDL matches up with the devices detected by udev. SDL should clearly not see bus 3 when the device is connected over Bluetooth. Or is another controller connected at the same time?

[terencode]

Or is another controller connected at the same time?

Only one controller is connected (zero 2).

udevadm.txt

controllermap-strace.txt

[kakra]

Hmm, I wonder if that's because it's opening js0 at the end. It's initially clearly opening the same event device as listet in udevadm, tho, and udevadm confirms it's seen on the Bluetooth bus. I'll compare with my own tests to see if it's doing the same here.

[kakra]

What does it say if you just run controllermap without arguments?

[terencode]

INFO: There are 1 joysticks attached
INFO: Joystick 0: Xbox One S Controller
INFO:        axes: 6
INFO:       balls: 0
INFO:        hats: 0
INFO:     buttons: 19
INFO: instance id: 0
INFO:        guid: 030000005e040000e002000000006800
INFO:     VID/PID: 0x045e/0x02e0
INFO: 

[kakra]

I'm not even sure how that could come from xpadneo because xpadneo exposes 11 buttons at most. Are you sure that this controller is handled by xpadneo? Is there something sitting in between that grabs the device from xpadneo and exposes it differently? Maybe a user space daemon, i.e. the Steam controller daemon that's coming with Steam? You could try turning that off in the Steam client.

See if anything has the device open: sudo lsof | grep "js0|event20" (and replace with the names your controller uses).

[terencode]

Are you sure that this controller is handled by xpadneo?

I would think so because 1) I set it to be in X input mode which emulates an Xbox 360 (https://support.8bitdo.com/faq/sn30-pro.html) and 2) xpadneo picks it up as shown in the different dmseg I posted: kernel: input: 8BitDo Zero 2 gamepad as /devices/pci0000:00/0000:00:01.3/0000:01:00.0/usb1/1-2/1-2:1.0/bluetooth/hci0/hci0:256/0005:045E:02E0.000C/input/input34

Maybe a user space daemon, i.e. the Steam controller daemon that's coming with Steam? You could try turning that off in the Steam client.

I had already thought it could conflict but it shouldn't be the case due to the fact steam isn't running.

See if anything has the device open: sudo lsof | grep "js0|event20" (and replace with the names your controller uses).

I used grep -E 'js0|event26' on the output of sudo lsof and I only see chromium using it (even when controllermap is running): lsof-js.txt

BTW: I joined the discord server.

[kakra]

0005:045E:02E0.

This is actually used by SDL to create the ID number, it would make 050000005e040000e0020000........ from it, but the dots are filled from the version field of the input device (usually the firmware version), in dmesg that's just a unique number built from a counter so that two identical models would have two different IDs in the kernel.

By looking at the SDL source code, it copies that straight from whatever udev provides. But then it does some magic: It looks if there's a default or xinput mapping available in the controller DB, and if it then cannot find the device, it tries to fall back to one of the two default mappings (it uses the xinput mapping if it finds a hint for Xbox controllers in the device name). But neither "xinput" nor "default" have the button map you're seeing. Also, your GUID is not in the SDL upstream controller DB.

Something remotely related only exists in the MacOS version of SDL (030000005e040000e0020000...) but that actually defines an xpadneo compatible mapping. And SDL doesn't match by device name as far as I can see, so it should not pick one of the 8BitDo mappings. So it seems your mapping entry is completely bogus and comes from somewhere else. This is also indicated by the name string which, when auto-detected, should say "8BitDo Zero 2 gamepad". Something is messed up in your system, something replaces the device. Or there's some obscure SDL environment variable set. Or something intercepts opening the device and emulates it as something else (which could explain the bus number, the Steam UI may do this but only while it is running: pgrep -a steam).

Could you run locate controllerdb or find / -type f -name "*controllerdb*"?

Which SDL version are you running?

Did you compile and install SDL yourself?

Could you run lddtree $(type -p controllermap)?

Does you environment set LD_PRELOAD or LD_LIBRARY_PATH?

In #228 (comment) you wrote that the controller ID changed, and I'm guessing that means from 0500... to 0300.... What happened to your system? What did you do?

Did you accidentally or on purpose modify a system-installed file of SDL?

Does your distribution patch SDL in a strange way? Or doesn't link to udev?

[kakra]

I think this may be antimicro... It is probably implemented as a uinput client: It needs to grab the controller and expose it as a new device. But then it should be on the virtual bus (bus 6). I wasn't able to reproduce such a behavior, the only device I was seeing was a virtual mouse. Maybe post $HOME/.config/antimicro/antimicro_settings.ini.

[terencode]

Which SDL version are you running? Did you compile and install SDL yourself? Does your distribution patch SDL in a strange way? Or doesn't link to udev? In #228 (comment) you wrote that the controller ID changed, and I'm guessing that means from 0500... to 0300.... What happened to your system? What did you do?

I didn't do much besides upgrading the system regularly, then noticing my gamepad had a different behaviour.
I'm using the one from the arch repo, SDL 2.0.12.

I found something very interesting though. Look at the the diff of the last packaged version: https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/sdl2&id=b77621a8fbe14a4e06ba640e2b687e8d6092d526

I reverted this and it's working as before!

With the revert, the guid is 030000005e040000e002000000006800 and all buttons are working correctly.
Without, it's 050000005e040000e002000003090000 and the previously mentioned buttons are not working.

Where do we go from here?

I still answered your other questions just in case:

Could you run locate controllerdb or find / -type f -name "controllerdb"?

I already tried that and many times, nothing relevant shows up (not used). locate.txt

Could you run lddtree $(type -p controllermap)?

lddtree: warning: controllermap: did not match any paths
lddtree: warning: is: did not match any paths
/usr/bin/controllermap (interpreter => /lib64/ld-linux-x86-64.so.2)
    libSDL2-2.0.so.0 => /usr/lib/libSDL2-2.0.so.0
        libm.so.6 => /usr/lib/libm.so.6
        libdl.so.2 => /usr/lib/libdl.so.2
        libpthread.so.0 => /usr/lib/libpthread.so.0
    libc.so.6 => /usr/lib/libc.so.6

Does you environment set LD_PRELOAD or LD_LIBRARY_PATH?

Did you accidentally or on purpose modify a system-installed file of SDL?

No.

[kakra]

I'm not sure what SDL is doing there exactly... But it seems bypassing any drivers, so xpadneo is not used. But this may also be related to b8ab951#diff-3cf66395a395dccd435d6973f8f96d68 which exposes the device as a hidraw device that SDL may then access directly when this patch is applied.

In udevadm monitor you should see the hidraw device name, you could chmod 0000 and see if SDL still behaves this way.

But I'll have to look at that patch more deeply. I don't think SDL should try to read devices directly without going through a driver.

[terencode]

In udevadm monitor you should see the hidraw device name, you could chmod 0000 and see if SDL still behaves this way.

If I chmod 0000 the hidraw device SDL behaves correctly.

[kakra]

So maybe I should revert that patch or we find some other way to exclude that from special handling in SDL... I'll have to dive deep into the patch first.

[kakra]

Please try setting SDL_JOYSTICK_HIDAPI=0 for starting SDL applications.

SDL 2.0.12 introduced a patch that talks to HID devices directly, bypassing drivers. It also doesn't pass the bus number, so it appears on bus 3 in SDL. This patch may be useful but it is quite bad for several reasons: It bypasses fixes that are implemented in drivers, such as that Xbox controllers need to throttle the rumble packets. Also, our fixes for buttons become messed up, the implementation cannot know that we fix the buttons in the HID reports. With this feature, it will not be possible to use some of the additional features we are going to add. IOW, it duplicates a lot of work and bugfixes: SDL would now have to solve the same problems we solved in xpadneo.

The new feature should be off by default, or only use the new implementation as a fallback if the device isn't supported by an OS driver. OTOH, SDL implements a user-space driver for any device regardless of the kernel driver support which isn't too bad. This is similar to how xow implements a user-space driver just that SDL bypasses the Linux HID layer that way.

I'm still on libsdl-2.0.10, let's see what we do about this when I'm upgraded, too. Maybe we need to revert the patch that allows direct raw HID access.

I'm not sure how this is meant to work correctly when SDL doesn't take the bus ID of the device into account because clearly Xbox controllers behave differently based on the bus number. Also, it should query the driver name and make that part of the controller mappings. And it should query and match the number of buttons it gets from the HID descriptor because only then it can know the button mappings to use.

If you disable the HID API of SDL, your controller should show up on bus 5 again and work correctly. Could you test?

[terencode]

Please try setting SDL_JOYSTICK_HIDAPI=0 for starting SDL applications.

Working. I now added it to my env :)

If you disable the HID API of SDL, your controller should show up on bus 5 again and work correctly. Could you test?

If you mean when using SDL_JOYSTICK_HIDAPI=0, I think it does show up on bus 5: 050000005e040000e002000003090000. Without, it's 030000005e040000e002000000006800.

It has the same effect as reverting to the unpatched SDL.

[kakra]

The unpatched SDL has a bug that prevented HID API from fully working with some versions of libusb, I believe.

So I think we'll just document the issue with SDL 2.0.12 and can then finally close this?

[terencode]

Alright.
Yes, after 91 92 messages we are good to close this.
Thank you very much for all your time spent diagnosing this :)

[kakra]

I'll reopen this to close it with a documentation update...

[terencode]

Oups, didn't get your comment about the documentation right, sorry.