Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forward audio #14

Open
benwa opened this issue Mar 9, 2018 · 225 comments
Open

Forward audio #14

benwa opened this issue Mar 9, 2018 · 225 comments

Comments

@benwa
Copy link

@benwa benwa commented Mar 9, 2018

Directing audio to the client would be pretty neat.

Currently you can accomplish the same thing by plugging a 3.5mm cable into the mic or line-in on the computer. Some manufacturers are dropping headphone ports though.

@MrLint
Copy link

@MrLint MrLint commented Mar 14, 2018

Adding this would really making streaming games would so much better. So far all the other existing solutions kinda suck.

@AdoPi
Copy link
Contributor

@AdoPi AdoPi commented Mar 14, 2018

Is it even possible without a custom rom?

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Mar 17, 2018

As suggested by Juan, one possibility to investigate is using AOA.

I already implemented something using HID over AOA in the past (I even have a publicly available sample), it should be quite similar.

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Mar 17, 2018

I just implemented a PoC (separated of scrcpy) to enable audio forwarding from the device to the computer: aoa-audio.

On Debian stable, it "works": the device sound is played on the computer.

This is quite manual though, and several issues must be solved before something similar could be implemented in scrcpy.

Any help is welcome 😉

@rom1v rom1v added the help wanted label Mar 17, 2018
@AdoPi
Copy link
Contributor

@AdoPi AdoPi commented Mar 17, 2018

May it be possible to use AudioRecord api server side?
Problem is that AudioRecord requires an Activity context, I don't know if we can spawn one by using app_process.

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Mar 17, 2018

AudioRecord may only record input sources (like microphone), not the device audio output.

@AdoPi
Copy link
Contributor

@AdoPi AdoPi commented Mar 17, 2018

I may be wrong but here is an example of what I thought:

https://github.com/tzutalin/ReadPCMData/blob/master/src/com/da/readpcmtest/MainActivity.java

Sorry I should have added this on my previous comment.

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Mar 17, 2018

Oh, you're right, I missed the REMOTE_SUBMIX audio source.

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Mar 17, 2018

Arf, just calling new AudioRecord(…) segfaults, probably because there is no Context (fault addr 0x0).

Note that AudioRecord itself does not require a Context explicitly.

@AdoPi
Copy link
Contributor

@AdoPi AdoPi commented Mar 17, 2018

Yes... :(
That's why I was wondering if we could create one even if we are using app process.

Audio support is really tricky :/

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Mar 17, 2018

Even from an app, the source REMOTE_SUBMIX does not work:

AudioRecord: Could not get audio input for record source 8, sample rate 44100, format 0x1, channel mask 0xc, session 73, flags 0

(while it works with MIC for example).

@rom1v rom1v self-assigned this Mar 20, 2018
@rom1v
Copy link
Collaborator

@rom1v rom1v commented Mar 21, 2018

Hi,

I would like to share with you my work-in-progress branch: audio. This branch can be rebased/rewritten at any time.

I implemented it, and it works fine for me! 🎉

I would like early feedbacks so that any major issue can be fixed.

Requirements

  • Linux (for now, I have problems to make it work on Windows and MacOS)
  • a device supporting AOA2
  • the device must be connected via USB (not adb connect)

How to

Install libusb:

sudo apt install libusb-1.0-0 libusb-1.0-0-dev

Checkout the audio branch:

git fetch
git checkout origin/audio

Use a debug build to report problems:

meson d
ninja -C d

Run with audio forwarding enabled:

./run d --forward-audio

The audio should be forwarded.

When closed, the audio forwarding may not be totally disabled. It may require to restart the current audio track (in VLC for example). If it is not sufficient, unplug the device. At worst (but it should not be necessary), disable/enable USB debugging.

How it works

Scrcpy connects to the device over USB to send AOA requests so that the computer becomes an "USB audio accessory" for the device.

This creates a new audio input source on the computer (which can be seen in PulseAudio for example).

Scrcpy records this input source and immediately plays it to the default output source.

Thank you for your feedbacks.

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Mar 23, 2018

In theory, it should now also work on Windows (in MSYS2/mingw64), with the libusb package:

pacman -S mingw-w64-x86_64-libusb

However, when it try to communicate to open the device over USB on Windows, libusb_open() returns -3, which is (according to libusb_strerror()):

Access denied (insufficient permissions)

I have no idea how to get the permissions granted. Any ideas?

@eyal-lezmy
Copy link

@eyal-lezmy eyal-lezmy commented Mar 24, 2018

On macOS you can install libusb with the following command: brew install libusb

On a MBP 15, macOS 10.13.3 it is not working.

I start the app while a podcast is playing on the device.
After the app starts I get some noise coming out from the computer's speakers, but nothing that I can recognise. At the same time the device's speakers stop emitting a sound.
After about 10 seconds, the scrcpy app stops itself and the podcast sound comes back on the device's speaker.

This behaviour is 100% reproducible on my side.

Here are the command line output:

2018-03-24 23:01:14.865 scrcpy[5291:36915] DEBUG: Device serial is ENU7N15B03004574
2018-03-24 23:01:14.866 scrcpy[5291:36915] DEBUG: USB device with serial ENU7N15B03004574 found: 18d1:4ee7
2018-03-24 23:01:14.867 scrcpy[5291:36915] DEBUG: Device AOA version: 2
2018-03-24 23:01:14.867 scrcpy[5291:36915] INFO: Audio forwarding enabled
2018-03-24 23:01:14.883 scrcpy[5291:36915] INFO: Waiting 2s for USB reconfiguration...
2018-03-24 23:01:17.144 scrcpy[5291:36915] INFO: Selecting input audio source: Built-in Microphone
d/server/scrcpy-server.jar: 1 file pushed. 2.9 MB/s (23985 bytes in 0.008s)
2018-03-24 23:01:17.918 scrcpy[5291:36915] DEBUG: Starting decoder thread
2018-03-24 23:01:17.918 scrcpy[5291:36915] DEBUG: Starting controller thread
2018-03-24 23:01:18.845 scrcpy[5291:36915] INFO: OpenGL shaders: ENABLED
2018-03-24 23:01:18.845 scrcpy[5291:36915] INFO: Created renderer: opengl
2018-03-24 23:01:18.849 scrcpy[5291:36915] INFO: Initial texture: 1440x2560
2018-03-24 23:01:20.662 scrcpy[5291:36915] DEBUG: Unknown touch device id -841524992, cannot reset
2018-03-24 23:01:25.184 scrcpy[5291:37036] DEBUG: End of frames
2018-03-24 23:01:25.188 scrcpy[5291:36915] DEBUG: Video decoder stopped
2018-03-24 23:01:25.188 scrcpy[5291:36915] DEBUG: quit...
2018-03-24 23:01:25.200 scrcpy[5291:36915] DEBUG: Server terminated
2018-03-24 23:01:25.408 scrcpy[5291:36915] DEBUG: USB device with serial ENU7N15B03004574 found: 18d1:4ee7
2018-03-24 23:01:25.409 scrcpy[5291:36915] DEBUG: Device AOA version: 2
2018-03-24 23:01:25.409 scrcpy[5291:36915] INFO: Audio forwarding disabled```
@rom1v
Copy link
Collaborator

@rom1v rom1v commented Mar 24, 2018

Thank you for the test 👍

Selecting input audio source: Built-in Microphone

It did not select the right input source, either because none were created, either because for now it just selects the last one, assuming it's the most recent (so it should probably be the input source from the device).

I'll check that. Thank you.

@eyal-lezmy
Copy link

@eyal-lezmy eyal-lezmy commented Mar 25, 2018

For your information I've done 2 tests one with a headset connected to the computer and another one without any headset plugged in.

Good luck

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Mar 26, 2018

@eyal-lezmy I investigated on Mac, and it appears that enabling "audio accessory" on the device does not create a new audio input source on the computer.

So unfortunately, I think that the feature will be Linux-only…

I updated the branch to reject the input source if its name does not contain the device model (so that it does not open the built-in microphone for example).

@MrLint
Copy link

@MrLint MrLint commented Mar 26, 2018

Ok so heres a wholly stupid question. Would it work via the Linux environment in windows 10?

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Mar 26, 2018

Would it work via the Linux environment in windows 10?

You tell me 😉

@yxtj
Copy link

@yxtj yxtj commented Jun 20, 2018

What is the plan of merging this excellent feature into the main branch?

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Jun 20, 2018

Did you test it? Does it work correctly for you?

There are several problems:

  • it's a bit hacky/fragile;
  • it only works on Linux (but not necessarily a problem);
  • if the computer consumes audio at a slightly different rate from the one it is produced by the device (which is not unlikely), the drift will increase and sound becomes very bad, and there is no mechanism to compensate.

So for now, I don't merge it.

Maybe I could, mentionning that it is very experimental...

@yimuchen
Copy link

@yimuchen yimuchen commented Jul 28, 2018

Hi, thanks for the package! I've been enjoying it a lot
I wanted to test the audio forwarding feature out on my machines, but I cannot work out why the program is failing to start with the forward audio enabled.

On my Archlinux machine, I have both libusb and libusb-compat installed besides the dependencies already listed in the AUR package for scrcpy. The testing scrcpy pacakge was compiled with the pre-built server.jar. When attempting to start scrcpy audio forwarding with my Nokia6.1 (Android 8.1, July 1st 2018 patch), the program exits with the following error message:

DEBUG: Device AOA version: 2
INFO: Audio accessory enabled
INFO: Waiting 2s for USB reconfiguration...
DEBUG: Audio input #0: Built-in Audio Analog Stereo
DEBUG: setpriority() failed
DEBUG: Device model is: Nokia 6.1
ERROR: Cannot find the Android accessory audio input source
DEBUG: Disabling audio accessory...
DEBUG: USB device with serial PL2GAMR832306185 found: 18d1:2d05
DEBUG: Device AOA version: 2

Without the --forward-audio flag, the program runs fine.

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Jul 31, 2018

@yimuchen Could you test manually with pactl and ffplay:

$ pactl list short sources
0   alsa_output.pci-0000_00_05.0.analog-stereo.monitor module-alsa-card.c  s16le 2ch 44100Hz   SUSPENDED
1   alsa_input.pci-0000_00_05.0.analog-stereo   module-alsa-card.c s16le 2ch 44100Hz   SUSPENDED

$ ffplay -vn -f pulse -i alsa_input.usb-LGE_Nexus_5_05f5e60a0ae518e5-01.analog-stereo
@yimuchen
Copy link

@yimuchen yimuchen commented Aug 1, 2018

@rom1v I'm a bit new to writing this sort of code, Can you help me with how I should get the string for my device, the alsa_input.usb-LGE_Nexus_5_05f5e60a0ae518e5-01.analog-stereo for my phone?

The output of pactl is:

0       alsa_output.pci-0000_00_1b.0.analog-stereo.monitor      module-alsa-card.c      s16le 2ch 48000Hz       SUSPENDED
1       alsa_input.pci-0000_00_1b.0.analog-stereo       module-alsa-card.c      s16le 2ch 44100Hz       SUSPENDED

On a side note, as a test, I edited the get_matching_audio_device in app/src/audio.c, so that the program uses the found Audio input #0: Built-in Audio Analog Stereo as the audio device, it seems to be able to pass the microphone input of my phone to my PC (I can hear myself talking to the phone from hy PC's headphone), but not the internal audio.

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Aug 1, 2018

Can you help me with how I should get the string for my device, the alsa_input.usb-LGE_Nexus_5_05f5e60a0ae518e5-01.analog-stereo for my phone?

Sorry, I copied incomplete instructions I kept in some textfile.

Instead, follow the steps I wrote in the README of aoa-audio.

@yimuchen
Copy link

@yimuchen yimuchen commented Aug 2, 2018

@rom1v Thanks for instructions! I think the issue that I have is that the udev rules for my phone isn't available yet. The vendor/product ID under lsusb is constantly in flux, and changes everytime I run the ./audio package. I guess I'll try again when I find and updated udev rule : /

@HilbertPurba
Copy link

@HilbertPurba HilbertPurba commented Jun 26, 2020

I use the Bluetooth Audio Receiver app from the microsoft store to forward my phone audio to PC thanks to the new Windows 10 May 2020 Update (2004). You need a bluetooth adapter and a playback device that supports A2DP SINK for this to work.

To make this possible you just need to pair your device with your PC and connect it using the app. I tested it and it works really well. Here's the link to the app if you're interested.
https://www.microsoft.com/en-us/p/bluetooth-audio-receiver/9n9wclwdqs5j

image

@srjbs11
Copy link

@srjbs11 srjbs11 commented Jun 26, 2020

@HillBird this is great. Can I know which Bluetooth adapter you are using? it should be 5.0?

@HilbertPurba
Copy link

@HilbertPurba HilbertPurba commented Jun 27, 2020

@HillBird this is great. Can I know which Bluetooth adapter you are using? it should be 5.0?

As long as you have a bluetooth feature available on your pc, you could give it a shot :)
Remember you need the latest update of Windows 10 (2004) to work.

@dheerendramehar
Copy link

@dheerendramehar dheerendramehar commented Jul 14, 2020

@HillBird Thank you! It works. Earlier I was getting audio with noise and crackles. I just turned off the WiFi and problem gone. I think there is some interference on WiFi and Bluetooth channels as both use 2.4GHz frequency. Any thoughts on this?

@HilbertPurba
Copy link

@HilbertPurba HilbertPurba commented Jul 21, 2020

@HillBird Thank you! It works. Earlier I was getting audio with noise and crackles. I just turned off the WiFi and problem gone. I think there is some interference on WiFi and Bluetooth channels as both use 2.4GHz frequency. Any thoughts on this?

Yeah when I play games on my phone, the noise sounds a little bit delayed and crackled too.

@Bluscream
Copy link

@Bluscream Bluscream commented Jul 21, 2020

These are all workarounds which don't solve the issue at hand

@hugonh
Copy link

@hugonh hugonh commented Sep 14, 2020

Actually it worked on my redmi note 8, windows 10 2004, using bluetooth audio receiver app!

@DiSavi
Copy link

@DiSavi DiSavi commented Nov 21, 2020

hey guys.. i need help with sndcpy.. it error me every time

@mmis1000
Copy link

@mmis1000 mmis1000 commented Nov 21, 2020

Although just use sndcpy works with things like obs(You can select video and audio from different program)

This don't really work with things like discord, because it expect video and audio from same program.

Which isn't the case currently.

Any workaround for this?

@lividhen
Copy link

@lividhen lividhen commented Nov 21, 2020

hey guys.. i need help with sndcpy.. it error me every time

What phone and OS?

@DiSavi
Copy link

@DiSavi DiSavi commented Nov 21, 2020

hey guys.. i need help with sndcpy.. it error me every time

What phone and OS?

Xiaomi note 8 pro and 10

@lividhen
Copy link

@lividhen lividhen commented Nov 22, 2020

hey guys.. i need help with sndcpy.. it error me every time

What phone and OS?

Xiaomi note 8 pro and 10

what is the specific error?

@DiSavi
Copy link

@DiSavi DiSavi commented Nov 22, 2020

hey guys.. i need help with sndcpy.. it error me every time

What phone and OS?

Xiaomi note 8 pro and 10

what is the specific error?

doesnt work sndcpy

@DiSavi
Copy link

@DiSavi DiSavi commented Nov 22, 2020

hey guys.. i need help with sndcpy.. it error me every time

What phone and OS?

Xiaomi note 8 pro and 10

what is the specific error?

doesnt work sndcpy

Waiting for device...
Performing Streamed Install
adb: failed to install sndcpy.apk: Security exception: You need the android.permission.INSTALL_GRANT_RUNTIME_PERMISSIONS permission to use the PackageManager.INSTALL_GRANT_RUNTIME_PERMISSIONS flag

java.lang.SecurityException: You need the android.permission.INSTALL_GRANT_RUNTIME_PERMISSIONS permission to use the PackageManager.INSTALL_GRANT_RUNTIME_PERMISSIONS flag
        at com.android.server.pm.PackageInstallerService.createSessionInternal(PackageInstallerService.java:548)
        at com.android.server.pm.PackageInstallerService.createSession(PackageUninstalling existing version first...
Success
Performing Streamed Install
adb: failed to install sndcpy.apk: Security exception: You need the android.permission.INSTALL_GRANT_RUNTIME_PERMISSIONS permission to use the PackageManager.INSTALL_GRANT_RUNTIME_PERMISSIONS flag

java.lang.SecurityException: You need the android.permission.INSTALL_GRANT_RUNTIME_PERMISSIONS permission to use the PackageManager.INSTALL_GRANT_RUNTIME_PERMISSIONS flag
        at com.android.server.pm.PackageInstallerService.createSessionInternal(PackageInstallerService.java:548)
        at com.android.server.pm.PackageInstallerService.createSession(PackageFailed with error #1.
Press any key to continue . . .
@rom1v
Copy link
Collaborator

@rom1v rom1v commented Nov 22, 2020

@DiSavi
Copy link

@DiSavi DiSavi commented Nov 22, 2020

so everyone got it work except me.. okay..

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Nov 22, 2020

@DiSavi I just pointed out the post explaining how to solve your problem (remove -g from the script).

@DiSavi
Copy link

@DiSavi DiSavi commented Nov 23, 2020

@DiSavi I just pointed out the post explaining how to solve your problem (remove -g from the script).

still dont get it.. never mind..

@bayazidbh
Copy link

@bayazidbh bayazidbh commented Dec 19, 2020

I've been testing sndcpy for a while now and it works well enough. That said, it's a separate command from scrcpy while the guiscrcpy still only use usbaudio from what I see.

I would like to see an implementation on guiscrcpy, especially if we could streamline the apk install for sndcpy so as to eliminate the delay in checking / reinstalling sndcpy.apk (which sometimes need me to do the sndcpy command a few times before the stream prompt appear). Something that can automate the process on guiscrcpy would be very nice.

@tranzzzistor
Copy link

@tranzzzistor tranzzzistor commented Dec 19, 2020

Maybe I'm missing something here, but I don't understand this issue. All I have to do to forward audio while scrcpy is running is to pair my phone and computer via bluetooth. And in the bluetooth settings on my phone, where it lists the computer it is paired with, make sure "Use for Phone Audio/ Media Audio" are both checked. That's it, now audio from my phone is played through my laptop speakers, or any speakers I connect to my laptop. I don't have to open anything or do anything extra on my laptop either. Is this not an option for y'all?

@rom1v
Copy link
Collaborator

@rom1v rom1v commented Dec 19, 2020

I would like to see an implementation on guiscrcpy

guiscrcpy is a separate project.

For the integration in scrcpy, read https://blog.rom1v.com/2020/06/audio-forwarding-on-android-10/#integration-in-scrcpy

@bayazidbh
Copy link

@bayazidbh bayazidbh commented Dec 19, 2020

I would like to see an implementation on guiscrcpy

guiscrcpy is a separate project.

For the integration in scrcpy, read https://blog.rom1v.com/2020/06/audio-forwarding-on-android-10/#integration-in-scrcpy

Ah, right, sorry about that. Didn't see which tab I was on. Still, thank you for the reply, I hope to see it eventually when you have the time for it.

@preIdiot
Copy link

@preIdiot preIdiot commented Jan 21, 2021

Maybe I'm missing something here, but I don't understand this issue. All I have to do to forward audio while scrcpy is running is to pair my phone and computer via bluetooth. And in the bluetooth settings on my phone, where it lists the computer it is paired with, make sure "Use for Phone Audio/ Media Audio" are both checked. That's it, now audio from my phone is played through my laptop speakers, or any speakers I connect to my laptop. I don't have to open anything or do anything extra on my laptop either. Is this not an option for y'all?

It works but the audio is either behind or ahead of the display

@Bluscream
Copy link

@Bluscream Bluscream commented Jan 21, 2021

Also scrcpy works via wifi which has more range than bluetooth

@LBensman
Copy link

@LBensman LBensman commented Jan 21, 2021

Also, BT doesn't work well when you have few devices in range, and especially several device on the same BT radio/controller.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet