Basic audio working. Many kinks.

[poconbhui]

This is a little nebulous. I pulled the audio processing out to another thread for performance. So there's effectively two event machines going on here over a number of threads. I could probably do with cleaning up some terminology. Suggestions?

I'm having some weird problems with Gtk where xcb is complaining about multiple threads. I'm also having an annoying issue where the write_report function isn't being called from a "main thread" for the audio, even though I'm pretty sure it's being called from a process in a way I've done before, so I can't put a timeout on the write. The write timeout is pretty important on my end for smooth audio.

I might actually start looking at pulling Gtk/gstreamer out now since they aren't playing nicely with threads.

[a1ien]

It appears that the volume is set in percentage.
0-100

[a1ien]

And I do not understand why there needs gtk. Maybe better use GObject.MainLoop. And gst.Bus for catch EOS and other signal from gstreamer

[poconbhui]

Me neither. It was in the tutorials I was following and it worked. I'll check those bits out.

[poconbhui]

GObject.MainLoop is working MUCH better.

[poconbhui]

I've finally got a pure pulseaudio / sbc implementation working in C. Getting the buffering right was a real pain. I'm looking into wrapping the necessary bits and pieces for python.

[parkerlreed]

@poconbhui Have a link to the C example?

[poconbhui]

@parkerlreed I've just pushed a working pulseaudio/sbc implementation. It's pretty rough and still has some debug print statements littered throughout.

Getting some of this stuff working nicely is a bit painful. Python threading and multiprocessing has a number of painful limitations so it works in Windows and Linux, which I don't really care about here. I started out trying to use the data management bits from those modules, but os.pipe does a muuuch better job than anything I could put together with them.

Another is that writing to the hidraw output on my system sometimes hangs, so I need some way of interrupting a system call. Python only allows signal/setitimer to be set on the main thread. I've managed to add a gross workaround here using the multiprocessing Pool. I've opted not to use a pure Process, since these sometimes die unexpectedly and a Pool reboots its worker thread automatically.

I think the audio timeout needs some tweaking as well. I'm not sure of the best way of getting frame time length, since the result from sbc_get_frame_duration seems to lie on my system.

I think SWIG might be overkill here too. I can probably make the pulseaudio class an opaque type and make new/start/stop/add_fd/remove_fd functions with C linkage and just wrap those with ctypes. I just used it because I've used it a number of times before. I should probably also wrap sbc.h instead of the sbc header parser I wrote from following the docs.

[poconbhui]

This should now be fully integrated with the controller's eventloop, so each controller is processing their own audio on their own thread.

I'm finding that sending other data to the controller, ie LED colour update on profile switch, causes some skipping in the audio. I think a different model of writing data to the controller might be better. I'm thinking of a constant write loop which will sync the ds4drv controller state to the controller, including LED, audio etc, instead of different events competing for bandwidth.

[poconbhui]

Ok, I think this is about it on my end as far as basic implementation goes. Could someone give this a go on their machine and tell me if it works well enough? I think the timeout might still not be tight enough.

I'm sure @Ape will have plenty to say about my style and a number of bits and pieces that may need some cleaning up.

[Ape]

Thanks for your effort, we are actually getting audio support!

I tried using this. It runs and creates a pulse audio sink just fine, but the audio I am hearing sounds like this (no matter what I try to play): https://x.ape3000.com/projects/ds4drv/audio.mp3

[poconbhui]

Sounds like an encoding issue. I've tried fixing it there. Let me know if that worked.

It all works on my machine, so I might have some difficulty guessing what's wrong from my end...

[parkerlreed]

I'll test on my end in just a second.

[parkerlreed]

Ok so I have ds4drv installed from your add-audio branch. Do I just launch ds4drv in --hidraw mode and it should work or is there an argument to pass?

[parkerlreed]

I just ran sudo ds4drv --hidraw (user doesn't have access to the dev entry) and I get two "Test d4drv sink" entries but neither of them plays audio to the controller.

Checked the hidraw permissions and saw they were rw for everybody. Tried without sudo and I get the same two sinks that don't output to controller. Also closing out of ds4drv via ^C doesn't clean up the sinks.

EDIT: Ok exiting cleans up one sink but not the other.

[parkerlreed]

Rebooted to start fresh. Running ds4drv as user created the single sink correctly but still no audio output. I am using the Bluez method of Share + PS to connect and get the hidraw device.

[parkerlreed]

... It works. I didn't realize it was only outputting to the headphone jack. :| I kept wondering why it wasn't going to the speaker, haha. Few pops here and there but overall sounds really good on a set of speakers. (Pops are probably just due to Bluetooth connection)

[poconbhui]

Sweet. Some options and tweaks still need to be added to handle audio like managing the headphone jack and volume.

Are the sinks cleaning up properly now? Are you sure you only have one instance of ds4drv with audio running?

I still need to add some stuff for reconnecting to pulseaudio if it's killed at some point.

[parkerlreed]

Yeah the double sinks just seemed to be an issue with pulse running as user and ds4drv as root. All running as user works and cleans up correctly.

[Ape]

The current version works really well. There are just some rare pops like parkerlreed said. Clean up seems to be working for me.

Is it possible to add another sink for the controller speaker? Or is just one stream at a time?

Does this work with multiple controllers at once?

[StalCaiRe]

@poconbhui i realized that this is in development atm ;)
i wrote my comment to help to develop this feature.
Since my python and linux skills in development are not so skilled i choose to inform you and not to write own crappy code.
I'll checkout the commit asap and try to figure out why the gcc compiler want to have the 'std=c++11' option which is not set

Also i must say this is an incredebile amazing work from you and i hope i can investigate and help to bring this to SteamOS and other OS

edit.: also sbc.h should be maybe packed with this version because SteamOS failed to find it easily
i fixed this by adding it manually tu usr/include/sbc

edit2: if we manage to make this fully working i will try to take the code to the windows version wich is more like my skills

[poconbhui]

Check out if Jessie has sbc and sbc-dev as separate packages. The -dev might be needed for building. It comes as the one sbc package on my distro. If they are split into package and package-dev, something like pulse-dev and boost-dev might be needed for building as well.

The compiler doesn't have -std=c++11 set because I didn't put it in setup.py. My version of gcc (6.1.1) has -std=gnu++14 by default, so I missed it. Whatever is on Jessie probably has an older standard as default. I've set -std=c++98, the oldest, reasonable one that seemed to work and stripped out the c++11 only headers. Should compile fine on Jessie now... hopefully.

See the discussion with @a1ien (#76) for more information on playing audio and a simpler example if you're looking to get it working on Windows.

[StalCaiRe]

are you still working with Python 2.7 ? or do i need to run it in Python3.5.x Environment.

[poconbhui]

I've been working with python 3 so this branch might have some python 3 only syntax in it. It should be python 2.7 compatible when it's done and cleaned up.

[StalCaiRe]

I saw this :) on debian i#m searching for a way to setup python3.5 since 3.4 also have errors in multithreading and TimeoutError is a Python3.5.2 definition.

[poconbhui]

Ah... oh dear... That one piece of multithreading is vital... but TimeoutError can be changed to anything. I'll have a look at running it with python 2.7 next chance I get.

[rektide]

Seems like really really good work here. What needs to happen for this to get merged in?

[poconbhui]

The audio isn't perfectly smooth. The fundamental problem is the threading model in python.

I think the threading model in python isn't performant enough to keep up with audio because of the global interpreter lock. One approach to fix it would be to run the audio stream and sending messages to the controller in a separate process using the multiprocessing module and have the main process pass messages about volume, led, rumble to that process via queues.

Adding it in nicely could need a pretty hefty change to how sending and receiving messages is handled. It's a bit gross and I haven't played with the controller in a while, so I've not touched it yet.

[mungewell]

Hi,
been playing with the audio stuff too and have a snippet of information which may help. The last 2 bytes (before the CRC) of the input report will give you information about audio playback

# hexdump -v -e '78/1 "%02x " "\n"' < /dev/hidraw0  | cut -d ' ' -f 70-
00 00 00 0b 04 45 0f d1 ec
00 00 00 0a 04 1d 9d d4 15
00 00 00 0a 04 d1 fb 0f c1
00 00 00 0a 04 bd e4 74 55
00 00 00 09 04 08 e4 fe 5c
00 00 00 09 04 97 df ac 26
00 00 00 03 04 a0 51 f0 70
00 00 00 02 04 ca 9c c4 98
00 00 00 02 04 bf 3a 03 cb
00 00 00 00 05 a6 3b f5 ef <- audio glitch here!
00 00 00 00 06 f7 25 17 f2
00 00 00 00 06 06 8e 5e ef
00 00 00 00 06 67 46 29 1b
00 00 00 00 06 9f a6 a0 d0
00 00 00 00 06 ea 68 38 a1
00 00 00 00 06 36 a3 7c 9a
.. .. .. .. .. ** ** ** ** CRC
.. .. .. .. .*             Audio Session (increases on buffer over/underflow)
.. .. .. **                Buffer level (0-3f, 0 empty)

With file playback you can throttle the data you send, with live stream you perhaps can deliberate delay a little to ensure that the buffer never empties.
Cheers,
Simon.

[mungewell]

Hi all,.
I got audio playing in both directions (ie. recording the MIC) to/from file. A modified version of 'play.py' is here:
https://dl.dropboxusercontent.com/u/34518077/playfile.py

Cheers,
Simon

[mateuscrash]

Hey guys im sorry im from Windows 7, i saw that you have been get progress in audio from ds4 on pc, do you have anything for windows ? thank you

[surprized]

yes, please

[parkerlreed]

@mungewell Is that latest script only for raw file playing? Would be cool to test input going into Pulse.

[JunPritsker]

Hey @poconbhui sorry for the dumb question, I've been reading all these threads and looking at your code out of curiosity, did you initially write the pulseaudio_sbc_stream in C++ to get it working and then port it to python so that it could be used within the ds4drv project? I'm just trying to figure out the general structure of your code and how it's used from the main project. Thanks for any help.

Edit: Second question, I have your ds4drv running but I get the error "AttributeError: 'DS4Controller' object has no attribute 'sbc_stream'" after I connect to my controller. Might be wonky since i'm running Ubuntu as a dual boot on my MacBook Pro, but everything seems to be working fine so far.

[JunPritsker]

@mungewell Would you mind explaining to me how I should go about using your python file? I'm noticing weird behavior. I have the correct device entered but all the script does is change my LED color to a kind of turqoise every time. I got the original script to play choppy audio so I know nothing's wrong with the controller. I followed the preparation instructions but I assume i'm passing parameters wrong. I tried giving a wav, au, and sbc, file all of which didn't work and i prepared the au file with either method. What am i missing?

[parkerlreed]

@JunPritsker From what I tested that script isn't for Pulseaduio or even playing a regular file. You have to pipe it a very specific format. Check the first handful of comments in that file if you haven't already (I honestly couldn't get it going myself either)

[JunPritsker]

@parkerlreed Oh ok gotcha, are you saying I should just pipe the output of that sox command to the execution of the python file? I used test.au which I generated with my wav file as the first parameter to the playfile and playfile produces output as if it's working except the only effect on the DS4 is the change in LED color like I said.

[poconbhui]

@JunPritsker It's written in C++ and wrapped with SWIG so it can be called from Python. I started trying to wrap the pulseaudio stuff for python using ctypes, but there was too much wrapping and faffing involved, so I wrote the C stuff in C (later in C++) and has SWIG automatically generate the necessary interfaces. It's a bit of a nuclear option with so few exported interfaces now, but there were many more in earlier development.

I can't say why you're getting the error you are because I haven't looked at the code in a good while. It still needs a bit of cleaning up and hardening. My controller is in a different country so I can't do much for now :(

[JunPritsker]

Ok gotcha, thanks for clarifying. I should've looked more into the uses of SWIG, I didn't know that it was used to wrap code for higher level use! Could you spare me one more explanation about all of the preparation of the audio before it can be streamed to the controller? I don't know much when it comes to audio but I do understand everything that's going on with the HID communication and whatnot. Any reading you could point me to would be fine too.

Also, Happy New Year!

[poconbhui]

Pulseaudio is a little gross because it's all based on callbacks. I eventually get the pulse stuff wrangled so it runs a the read_pulse_stream callback when it has some audio data available. In that callback, the audio data is read into a buffer, encoded into an SBC format and written to a file descriptor. The buffer is needed because SBC encodes in discrete chunks. SBC is a standard bluetooth audio encoding format, and what the DS4 uses.

The pulseaudio manager is run in a separate C++ thread (as opposed to a python thread) for performance reasons. The file descriptor mentioned is made on the python end and passed to the pulseaudio manager with the add_fd method. From python, the fd is monitored for activity. Upon data waiting, a new SBC frame is waiting, so it's read and passed to the controller manager. The controller manager then wraps it up into a bluetooth frame to be sent to the controller.

Information on the format for the bluetooth frames sent to the controller can be found at http://eleccelerator.com/wiki/index.php?title=DualShock_4 , more information on playing about with sending audio to the controllers at #76 . Depending how into SBC encoding you want to go, the SBC specs are available in the A2DP docs. Mostly useful for knowing exactly how long SBC frames and headers are going to be. The pulseaudio docs are absolute crap. If you want more info on that, I can go into my development approach there.

[parkerlreed]

@poconbhui Did something stop working with https://github.com/poconbhui/ds4drv/tree/add-audio ?

Tried both with and without sudo, pulse sink shows up either way, but no audio can be heard on speaker or headset.

EDIT: Oh I forgot to connect with Bluez first and use --hidraw, woops

Not sure if interference but latency is really really bad at the moment.

[jonas222]

@poconbhui I know this a very old thread, but I'm hoping you will be able to help me with the audio branch for ds4drv. I cloned your 'add-audio' branch and installed it using command:
sudo python3 setup.py install

I had to download a few dependence, but I was able to get the project to compile and run.

I first connect my DS4 (version 1, original controller) with the bluetooth that comes with Ubuntu 16.04
While connected, I run sudo ds4drv --hidraw, this creates a ds4drv sink, but I still get no audio from the controller. I have tested with mutliple DS4 controllers (version 1 only).

Here is my terminal output:

`[info][PulseaudioSBCStream] Connecting to Pulseaudio

[info][PulseaudioSBCStream] Stream format s16le

[info][PulseaudioSBCStream] Stream sample rate 32000

[info][PulseaudioSBCStream] Stream codesize: 512

[info][PulseaudioSBCStream] Stream frame_length: 112

[info][controller 1] Created devices /dev/input/js1 (joystick) /dev/input/event23 (evdev)

[info][controller 1] Connected to Bluetooth Controller (2A:26:1D:3C:91:5B hidraw5)

[info][hidraw] Scanning for devices

[info][controller 1] Battery: 50%

[info][controller 1] Audio: Speaker

[info][controller 1] Audio: Headphones

[info][controller 1] Audio: Headset
`

Do you see a problem that I could be missing or running in to?

[norcalli]

@mungewell Your script is no longer hosted on dropbox. Would you mind reuploading in a more permanent location, such as a gist or fork?

[pablospe]

@jonas222 I have the same problem. Have you find a solution to make? I am in ubuntu 18.04.

> sudo ds4drv --trackpad-mouse --led ff0000 
[info][PulseaudioSBCStream] Connecting to Pulseaudio
[info][PulseaudioSBCStream] Stream format s16le
[info][PulseaudioSBCStream] Stream sample rate 32000
[info][PulseaudioSBCStream] Stream codesize: 512
[info][PulseaudioSBCStream] Stream frame_length: 112
[info][controller 1] Created devices /dev/input/js0 (joystick) /dev/input/event20 (evdev) 
[info][bluetooth] Scanning for devices
[info][bluetooth] Found device 1C:A0:B8:8A:16:B3
[info][controller 1] Connected to Bluetooth Controller (1C:A0:B8:8A:16:B3)
[info][bluetooth] Scanning for devices
[info][controller 1] Battery: 50%

[kquote03]

apologies if necroposting but i get the same exact result as jonas222 on Fedora 38 using the same steps