New Bluetooth pairing script doesn't allow input, doesn't pair any devices

[timherr]

Following up on closed issue #215, the updated script does not work properly for me when running from the MiSTer UI. When running the script, it identifies three devices in my vicinity. For each device, it displays the "Enter device PIN or ENTER to skip:" prompt, but continues on to the next device with the message "ERROR: Authentication Failed" before I can enter anything. Therefore, it skips all devices and nothing connects.

When I run the btpair script from an SSH connection, it works as expected and allows time for me to input a value. I'm able to successfully connect to my wireless controller while skipping other devices.

This is running the 6/22 release.

@c0d3h4x0r

[sorgelig]

If some BT device requires to enter something from MiSTer side, then you can't pair it from OSD and it doesn't provide any inputs.
The only way to pair such devices is to use SSH/Console. This is what you did.

[timherr]

The previous version of the script successfully paired to my controller without using SSH, as long as it was the only device in the area. The update was supposed to gracefully handle multiple devices, but now breaks pairing.

For reference, this is a PS4 dual shock controller.

[sorgelig]

I've just tried to re-pair my DualShock 4 and didn't find any problem. But it's the only BT device around in pairing mode.

[timherr]

I turned off my other BT devices and ran the script with only the DS4 nearby. It only found the controller, but still displayed the same output: "Enter PIN" followed immediately by "Authentication Failed."

Does your DS4 require you to enter 0000 when you connect via SSH? Mine does, but the previous version of the script apparently handled that step automatically. So, I'd suggest that the updated version maintain that functionality.

[sorgelig]

my DS4 doesn't require any pin. Neither in OSD nor in USB console (terminal). And it never required before.
I didn't try SSH.

[sorgelig]

in SSH:

login as: root
Using keyboard-interactive authentication.
Password:
/root# btpair
Switch input devices
to pairing mode.
Searching...

Found: Wireless Controller
MAC: 1C:A0:B8:xx:xx:xx
Pairing...
Trusting...
Connecting...
Done.

/root#

[timherr]

Interesting. It seems like maybe there are two versions of the DS4 out there, and the updated script doesn't properly handle my type anymore.

Here's what I get via SSH if I don't provide a PIN:

/sbin# ./btpair
Switch input devices
to pairing mode.
Searching...

Found: Wireless Controller
MAC: 1C:96:5A:81:6F:E9
Pairing...
Enter device PIN
or <ENTER> to skip:
ERROR: Authentication Failed

And here's what I get if I do provide a PIN:

/sbin# ./btpair
Switch input devices
to pairing mode.
Searching...

Found: Wireless Controller
MAC: 1C:96:5A:81:6F:E9
Pairing...
Enter device PIN
or <ENTER> to skip: 0000
Trusting...
Connecting...
Done.

[timherr]

I just tried a second DS4 controller and got the same results as my first controller.

[sorgelig]

how is the previous linux release? you can simply replace linux.img

[timherr]

I tried the 6/18 Linux release with both the 6/18 and 6/22 Main releases and got the same behavior as described above.

I then tried the 4/29 Linux release with the 6/22 Main release and the controller pairs successfully through both the OSD and SSH, with no interaction on my part (as long as it is the only device in the vicinity, per #215):

/root# btpair
Switch input device
to pairing mode.
searching...

Got device: 1C:96:5A:81:6F:E9
Pairing...

Enter PIN 0000 on device

Release
New device (/org/bluez/1277/hci0/dev_1C_96_5A_81_6F_E9)
Add to trust list...
Connecting...
Done.

[sorgelig]

i don't call it really success as it still displays Enter PIN 0000 on device
I give 99% that your DualShock 4 is fake.
There are many clones of DS4.

[sorgelig]

besides fake DS4 i think MiSTer should try well known basic pins 0000 and 1234 before give up. Script needs to be updated.

[timherr]

Highly unlikely that it's a fake controller, as it's the one that came with my boxed retail PS4. Seems more likely that there are different revisions of the controller that behave differently.

Regardless, I agree - an updated script that handles basic PINs would be a good resolution.

[c0d3h4x0r]

i don't call it really success as it still displays Enter PIN 0000 on device

The old version of the script had a bug: when it was supposed to prompt the user to enter a PIN supplied by the device, it instead printed "Enter PIN 0000 on device", didn't prompt the user for input, and immediately returned a hard-coded 0000 as if that was what the user had typed. That buggy behavior just happened to work out nicely for timherr's DS4 controllers.

I give 99% that your DualShock 4 is fake.

I have two genuine SONY DS4 controllers (each one was included with a sealed new-in-box PS4 console I bought at retail, so they are definitely genuine), and when I try pairing either one with my MiSTer using the newbtpair, I am not prompted for a PIN code.

I wouldn't assume timherr's DS4 controllers are fakes. It's just as likely that they have a buggy firmware version or have somehow gotten into a bad state and just need to be reset (using the little pinhole reset button on the back side).

[sorgelig]

the issue is not in buggy/fake DS4.
Issue is that standard pin 0000 is not automatically entered as before.

The old version of the script had a bug: when it was supposed to prompt the user to enter a PIN supplied by the device, it instead printed "Enter PIN 0000 on device", didn't prompt the user for input, and immediately returned a hard-coded 0000 as if that was what the user had typed. That buggy behavior just happened to work out nicely for timherr's DS4 controllers.

what you call buggy actually in not buggy. It's intended behavior for pairing when you can't enter. "Enter PIN 0000" is actually for user to enter this pin on his BT keyboard to finish pairing. This is how BT keyboards are paired. It used only hardcoded 0000 pin which should be extended to 2 tries with 0000 and 1234 (for non-keyboard devices with hardcoded PINs) - these 2 PINs cover 99.9% devices using PIN to pair. I've thought your change included such behavior, but unfortunately now it's broken and such devices cannot be paired in OSD.

[c0d3h4x0r]

I see. I think it's possible for the Python code to detect if stdin is connected to the process or not, and take different actions in each case. That way it could automatically attempt 0000 (and maybe also 1234? not sure if the bluez API allows a second attempt) when run from the OSD, but prompt the user for a PIN when stdin is available.

[c0d3h4x0r]

I can try coding this up tomorrow after work, unless you get some other fix done before me.

[c0d3h4x0r]

Yes, Python has a built-in isatty() function that should work for this purpose.

[sorgelig]

i'll leave it to you. I'm not good at python.
As for attempts. May be can try to pair 2 times of the same device if PIN is requested and first (0000) is failed. Can use BT keyboard for test when first time PIN is entered wrong.

[c0d3h4x0r]

Good idea. I’ll see what I can come up with tomorrow.

[c0d3h4x0r]

Just FYI, I'm working on this, but hitting some challenges with self-testing it. My BT keyboard doesn't require a PIN to pair, but bluez keeps calling back into RequestPinCode to ask the user for the BT keyboard's PIN. And none of the PINs I try (automatically via my new retry code, or manually via a wired keyboard) are accepted. I'm probably overlooking something stupid and I'm just too tired to spot it.

[c0d3h4x0r]

Looks like it's some kind of real problem with my keyboard, because now whenever I try to pair it with my Windows PCs they also ask me to type in the PIN for the device. It doesn't have a PIN and has never required one before. 🤦‍♂️

[c0d3h4x0r]

Updated the keyboard's firmware, and now it's behaving sanely again. Glad I'm not actually losing my mind :)

[c0d3h4x0r]

I have three different Bluetooth keyboards, and two of them behave differently from the third one in terms of the pairing process. It seems that some BT keyboards don't require a PIN to pair, while others require you type type a PIN on the keyboard itself that should match the one that the MiSTer user is supposed to type into the MiSTer. Gotta love inconsistency! 🤦‍♂️ I'm revising the script and the messaging it gives to the user to handle both types.

[c0d3h4x0r]

Alright, here's the new version. It's two scripts that needed changing. timherr, can you please give these a try and report back? Unfortunately, there's just such a wide range of Bluetooth devices out there, and some of them don't seem to adhere to the intent of the spec/API, that the best we can do to avoid regressing things is do a bunch of manual testing with as many different Bluetooth devices as possible.

(EDITED: to remove lengthy now-pointless early revisions of the scripts)

[c0d3h4x0r]

Specifically, I have no BT devices that have a hard-coded PIN that must be entered on the MiSTer in order to pair, nor do I have any devices that need a simple yes/no confirmation/authorization, so there are some cases and codepaths in here I can't exercise completely.

[c0d3h4x0r]

Figured out I can make my laptop discoverable in such a way that MiSTer goes through the confirmation codepath. Looks like the new _confirm() method doesn't work quite right, so I'll need to either fix it or just revert back to using the _ask() method for the confirmation cases.

[c0d3h4x0r]

Fixed it; edited my earlier post to contain the fix. Please try now.

[timherr]

Awesome! I'll test tomorrow night and report back here.

[timherr]

Running the latest version of the Linux/Main/Menu releases, then updated btpair and pair-agent to the above code. Using the OSD, it finds my controller and gets to the "Trusting..." line and never completes. Via SSH, here's what I get:

/sbin# ./btpair
Switch input devices
to pairing mode.
Searching...

Found: Wireless Controller
MAC: 1C:96:5A:81:6F:E9
Pairing...
Trying PIN: 0000
Your keyboard may require
you to now type this PIN
and press ENTER to finish
pairing.
Trusting...
Traceback (most recent call last):
  File "/sbin/pair-agent", line 238, in <module>
    device.SetProperty("Trusted", True)
  File "/usr/lib/python3.5/site-packages/dbus/proxies.py", line 70, in __call__
  File "/usr/lib/python3.5/site-packages/dbus/proxies.py", line 145, in __call__
  File "/usr/lib/python3.5/site-packages/dbus/connection.py", line 651, in call_blocking
dbus.exceptions.DBusException: org.bluez.Error.Failed: No such file or directory

[c0d3h4x0r]

Thanks. That means the device object isn't valid. I'll review the code and see if I can figure out how that could happen.

[c0d3h4x0r]

Also, I realized that your strange DS4 pairing behavior probably meant you'd get the messaging about "Your keyboard", which obviously doesn't apply correctly to your case, so I'll revise that messaging more as well.

[c0d3h4x0r]

timherr, if you're willing to connect via a chat app like Facebook Messenger or Google Hangouts, I think that would enable more efficient back-and-forth while we iteratively work through this. If you're up for that, email me at [EDITED for privacy] and I'll send you contact info.

[timherr]

I lurk on the (heavily MiSTer focused) Classic Gaming Discord server here: https://discord.gg/UDu5ztY

Feel free to message me, I'll be around most of the evening (username: therr)

[c0d3h4x0r]

That works.

[c0d3h4x0r]

Update: timherr and I made some good progress on this tonight. I think it's finally done, but I'd like him to verify one last time tomorrow before I post the new scripts here.

[c0d3h4x0r]

Two great pieces of news:

1. We've arrived at a very solid set of changes to the scripts. timherr agrees that these scripts are looking good on his end, and that the remainder of his issues are specific to something with his hardware. I am posting them in a ZIP file below.
2. We've been able to narrow down timherr's weird DS4 pairing problems a lot. Somehow, the binary file linux/bluetooth at the root of his exFAT partition (which gets mounted as an ext filesystem at /var/lib/bluetooth to hold all the bluetooth pairing settings) keeps ending up with a corrupt-looking internal filesystem. His BT USB dongle is the kind that gets a different random MAC address on every startup, which means he gets a new folder created under /var/lib/bluetooth on most power-ons. And a lot of those folders end up with the ultra-suspicious "January 1, 1970" date (Linux zero-reference datecode), which suggests either an incompatibility between his MAC-rotating BT dongle and the kernel driver, or a bad/failing microSD card. (He already scanned his exFAT partition with chkdsk /F on a Windows PC, and it found no corruption, so it seems the contents of this particular binary file on the exFAT may not be reading/writing correctly.) We were able to confirm that by deleting that binary file from the root of the exFAT, he was finally able to pair his controllers again.

Summary of changes

pair-agent:

• Added a bunch of super-helpful new debug output, which can be enabled by setting an environment variable DEBUG_BLUETOOTH=1.
• Added automatic PIN/passkey entry, with automatic retries, for most commonly-used hard-coded values (0000, 1111, 1234, etc)
• Added far more thorough Python/DBus exception handling, so the script reports problems more helpfully and doesn't just abort with a Python exception stack.
• The script can now automatically detect if a device is an input device and early abort (skip) non-input devices.
• In support of the new debug output mechanism (which automatically prefaces any debug messages with the function name), I refactored the code into a more logical and maintainable set of standalone functions.
• New order of operations is: remove existing pairing for device, check if device is an input device, pair the input device, connect to it, and then mark it as trusted.
• Script now automatically detects when it is being run from the OSD or not (sys.stdin.isatty()), and only prompts for user input (as a last resort, after automatic PIN/passkey attempts) when the stdin is actually connected to a tty. Because of this logic, the script can now figure out the right capability value on its own instead of requiring the caller to pass it in, so that command-line option has been removed.
• Added a comment block at the top with handy URLs to the relevant dbus and bluez documentation (which were both a little tricky to track down).
• Added specific detection of the "Operation not permitted" error that was the telltale symptom of timherr's local filesystem problem, with clear messaging of investigation/workaround instructions to the user.

btscan:

• Replaced big TODO and commented-out code block with a simple hciconfig hc0 reset to forcibly reset the BT adapter before kicking off a scan for input devices. This cures intermittent reliability problems with pairing, in exchange for the very minor annoyance of already-connected Bluetooth devices disconnecting and powering down during the scan. (They reconnect right away whenever the user powers them back on after the reset.)

btpair:

• Removed passing the capability to pair-agent as it is no longer needed.

/etc/bluetooth/main.conf:

• Added and enabled the missing "fast connect" setting, which enables all my devices to find and reconnect to MiSTer in much less time now.

Download: issue-254-fix.zip

[sorgelig]

Thanks for constant improving the BT pairing.

[c0d3h4x0r]

Happy to help! I think I forgot to properly escape the backslash characters in the new messaging about the “Operation not permitted” error. I’ll check it and send one more tiny revision to fix it if I need to.

[sorgelig]

i'll leave the changes to you.

[c0d3h4x0r]

Posting revised ZIP file which refines the messaging and avoids the backslash-escaping problem.
ZIP file now includes examples of output from the new script to demonstrate the new features and user messaging.

Download: issue-254-fix.zip

[c0d3h4x0r]

One more quick thing I noticed: you know that "wonky" BT adapter state I mentioned in the btscan script? I just found the BT adapter can sometimes get stuck in that same bad state just due to normal connection/disconnection of Bluetooth devices, which can leave the BT adapter permanently in that state (even across soft restarts). I'm thinking it might be a good idea to add hciconfig hci0 reset into the startup sequence by way of /etc/inittab as well. That would mean Bluetooth devices always get disconnected on reboot, but then should immediately reconnect if powered back on by the user. The MiSTer process will be rebuilding its physical device mappings from scratch on every startup anyway, so I don't think that would be much of a usability problem. What do you think about this idea?

[sorgelig]

Official USB hub v2.x cuts off the power from all USB devices upon reset.
As for hci0 reset it's not so clear. I remember i've played with this command before - it provoked unresponsive state of BT dongle. Reboot procedure with many random commands for BT will just increase the chance of non-working BT.
You need to get several BT dongles for test. Also you need to get both CSR and Broadcom dongles. Then you can test all them to make sure your new idea won't break the things.

[c0d3h4x0r]

Okay, then I will forget about this extra idea.

[sorgelig]

While i have BT dongle in my MiSTer, i use RF keyboard (Logitech K400r) and RF gamepad (Logitech F710).. Guess why ;)
But sometimes i use my DualShock 4 over BT.

[c0d3h4x0r]

Were the latest changes included in the most recent MiSTer main release? If so, it should be safe to close this now.

[timherr]

I can confirm the latest version from here works far far better for me, so I support it being officially released: #254 (comment)

I still have to re-pair every time I boot the MiSTer because my adapter randomizes its MAC, but pairing works like a charm now, so that's trivially easy.

@c0d3h4x0r I sent you a DM on Discord last week with a possible edge case, but it's not too urgent.

[sorgelig]

This belongs to Linux. So it will be released with Linux. I plan to release soon

[sorgelig]

should be done already