New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PortAudio support #14

Merged
merged 10 commits into from Mar 30, 2016

Conversation

Projects
None yet
5 participants
@zeehio
Copy link
Contributor

zeehio commented Feb 25, 2016

This pull request aims to close issues #10 and #13.

What does it do?

  • Adds an audio module "portaudio" in src/audio/au_portaudio.c. See #10 to see the discussion.
  • Adds automatic detection of portaudio in configure.in and configure.
  • Changes some indentation from tabs to spaces. I don't care to use one or the other, but not both in the same file 馃槵.
  • Due to some differences in the audio module, play_wave_from_socket does not work yet, although we may not need it.
  • It fixes the ALSA detection in configure.in and configure scripts that failed with ALSA-1.1.0.

Main changes:

  • There is an audio_init and audio_exit function that needs to be run when mimic starts and exits.
  • audio_init has been added to mimic_init. mimic_exit has been created and calls audio_exit.
  • All programs calling mimic_init now call mimic_exit at the end.
  • The PortAudio module opens and closes an audio stream on each write operation, so the play_wave function needs to be edited to pass the whole buffer of samples at once. play_wave_sync and play_wave_from_socket have not been adapted to work with PortAudio, mainly because I don't think we are using them.

Testing

  • The PortAudio module has been tested under Linux Mint 14.04 64 bits with the portaudio19-dev 19+svn20140130-1 package from the Ubuntu repositories.
  • Someone (@siavasht ?) needs to test it on other platforms (Mac OS X, for instance). brew install portaudio is required on Mac OS to install portaudio, as far as I know.
  • The ALSA detection patch in this pull request needs to be tested, for instance on Debian Sid (@julianrichen ?)
@ryanleesipes

This comment has been minimized.

Copy link

ryanleesipes commented Feb 26, 2016

This is pretty awesome. Any noticeable effect on performance?

@forslund

This comment has been minimized.

Copy link
Member

forslund commented Feb 26, 2016

Great work, I have some comments but it's working fine on my Ubuntu machine.

When running mimic with portaudio I do get some line noice, is this to be expected or have I got some incorrect settings?
ake@Woodstock:~/projects/c/mimic/bin$ ./mimic -t "Hello" ALSA lib pcm_dsnoop.c:614:(snd_pcm_dsnoop_open) unable to open slave ALSA lib pcm_dmix.c:1024:(snd_pcm_dmix_open) unable to open slave ALSA lib pcm.c:2267:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear ALSA lib pcm.c:2267:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe ALSA lib pcm.c:2267:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side ALSA lib pcm_dmix.c:1024:(snd_pcm_dmix_open) unable to open slave

@forslund

This comment has been minimized.

Copy link
Member

forslund commented Feb 26, 2016

Great work! (And fast!) Let's see if this can be tested on Mac before we merge it. Is the output I get from my portaudio installation (had to build from source to get dev-files in ubuntu) or is it something that is expected?

@zeehio

This comment has been minimized.

Copy link
Contributor Author

zeehio commented Feb 26, 2016

With respect to performance (@ryanleesipes was asking about), I haven't noticed any significant change.

With respect to the annoying output, this 2012 page explains it great. Basically they claim this noise comes from ALSA configuration errors. Muting them may hide real errors and I would need to bypass PortAudio using libasound calls to do that. Not a clean fix available, as I see it.

@julianrichen

This comment has been minimized.

Copy link

julianrichen commented Feb 26, 2016

@zeehio Just tested the patch on Debian Sid 64bit & portaudio19-dev 19+svn20140130-1, works. Thank you :)

I'll be out most of the day with my Fedora 23 laptop so I'll try to compile and test that when I have free time.

@julianrichen

This comment has been minimized.

Copy link

julianrichen commented Feb 26, 2016

On Fedora 23 with portaudio-19-22.fc23 it's reporting audio device is missing. Same message:

$ ./bin/mimic -t "hello"
oss_audio: failed to open audio device /dev/dsp
@zeehio

This comment has been minimized.

Copy link
Contributor Author

zeehio commented Feb 26, 2016

@julianrichen, do you have portaudio-devel installed? It is necessary to compile the portaudio module in mimic

@julianrichen

This comment has been minimized.

Copy link

julianrichen commented Feb 26, 2016

@zeehio Oops, that was it. It works on Fedora 23 just fine.

Edit, like someone else mentioned above I also see some notices:

ALSA lib pcm_dsnoop.c:614:(snd_pcm_dsnoop_open) unable to open slave
ALSA lib pcm_dmix.c:1024:(snd_pcm_dmix_open) unable to open slave
ALSA lib pcm.c:2267:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2267:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2267:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib pcm_dmix.c:1024:(snd_pcm_dmix_open) unable to open slave
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
@ryanleesipes

This comment has been minimized.

Copy link

ryanleesipes commented Mar 4, 2016

@julianrichen these notices are typical with PortAudio in my experience (we use it in Mycroft Core as well)

@LongBoolean

This comment has been minimized.

Copy link
Collaborator

LongBoolean commented Mar 4, 2016

The sound output is just fine, but the errors I get are a little different from the ones @julianrichen posted:

ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
bt_audio_service_open: connect() failed: Connection refused (111)
bt_audio_service_open: connect() failed: Connection refused (111)
bt_audio_service_open: connect() failed: Connection refused (111)
bt_audio_service_open: connect() failed: Connection refused (111)
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started

@zeehio (for clarification) is the Cannot connect to server socket err = No such file or directory what you mean when you said "play_wave_from_socket does not work yet" or is that error from something else?

}

int audio_init_portaudio() {
int err = Pa_Initialize();

This comment has been minimized.

@zeehio

zeehio Mar 5, 2016

Author Contributor

Before and after Pa_Initialize, add an error message to ignore harmless error messages from Pa_Initialize if audio works fine

/* Read audio from stream and play it to audio device, converting */
/* it to pcm if required */
if (CST_AUDIOBUFFSIZE <= 0) {
cst_errmsg("play_wave_from_socket not supported with PortAudio\n");

This comment has been minimized.

@zeehio

zeehio Mar 5, 2016

Author Contributor

@LongBoolean This is the error message that would appear if play_wave_from socket was used.

The bin/mimic binary does not use it.

@zeehio

This comment has been minimized.

Copy link
Contributor Author

zeehio commented Mar 5, 2016

The printed messages depend on your audio devices, they are printed as portaudio lists the devices. On Sunday I will add an error message before and after messages are printed, so the user knows that if they hear the audio they dont have to worry about those errors.

@LongBoolean, With respect to play_wave_from_socket, your errors are not related to that. I have added a couple of notes in the pull request so you can read what is the error that would be printed

@ryanleesipes

This comment has been minimized.

Copy link

ryanleesipes commented Mar 16, 2016

Is this working properly? If so let's go ahead and merge it.

@zeehio

This comment has been minimized.

Copy link
Contributor Author

zeehio commented Mar 16, 2016

@ryanleesipes I believe we still lack Mac OS testing... it boils down to

# Install brew:
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
# Install portaudio using brew:
brew install portaudio
# compile mimic (making sure we use portaudio)
./configure --with-audio=portaudio
make
# Test
bin/mimic -t "Hello world"
@LongBoolean

This comment has been minimized.

Copy link
Collaborator

LongBoolean commented Mar 16, 2016

What platforms has it been tested on? @ryanleesipes Should we wait until we know it works on them or solve that problem when we come to it?
Edit: zeehio beat me to it with his lightning reflexes

@forslund

This comment has been minimized.

Copy link
Member

forslund commented Mar 16, 2016

I discussed this with @LongBoolean a while ago. We wanted to test it out on OS X and Windows before merge.

@ryanleesipes

This comment has been minimized.

Copy link

ryanleesipes commented Mar 16, 2016

This is a really good idea. Let me get @communitywireless to test it on Windows. Then we need to find someone to test it on Mac. @LongBoolean should have a Mac show up at his house shortly.

@julianrichen

This comment has been minimized.

Copy link

julianrichen commented Mar 17, 2016

If you need another Windows tester I can switch to my Windows partition. However, I have never compiled under Windows (only Linux) so I can't guarantee anything. I'll try and give it a go tomorrow later in the day.

I assume all you need is MinGW & Automake installed?

@LongBoolean

This comment has been minimized.

Copy link
Collaborator

LongBoolean commented Mar 17, 2016

@julianrichen I'm not quite sure of all that a windows install will require, thats part of the fun!!!

@julianrichen

This comment has been minimized.

Copy link

julianrichen commented Mar 18, 2016

I've used Windows for a long a time but never for development, this is very much different then an unix environment when it comes to setting-up a dev environment. Only had a bit of time trying things out but still not sure what I'm doing... but as you said, it's part of the fun! I'll try again on the weekend.

Hopefully someone with more experience can also test.

@LongBoolean

This comment has been minimized.

Copy link
Collaborator

LongBoolean commented Mar 29, 2016

@zeehio
Tried it out on the mac mini, following your instructions above. I get this error in the make process:

making in ...
making in include ...
making in src ...
making in src/audio ...
gcc   -I. -DCST_AUDIO_PORTAUDIO -I../../include -g -O2 -Wall -no-cpp-precomp    -c au_portaudio.c -o ../../build/x86_64-darwin15.4.0/obj/src/audio/au_portaudio.o
au_portaudio.c:46:10: fatal error: 'portaudio.h' file not found
#include <portaudio.h>
         ^
1 error generated.
make[2]: *** [../../build/x86_64-darwin15.4.0/obj/src/audio/au_portaudio.o] Error 1
make[1]: *** [../build/x86_64-darwin15.4.0/obj/src/.make_build_dirs] Error 2
make: *** [build/x86_64-darwin15.4.0/obj//.make_build_dirs] Error 2

I have already installed portaudio with brew. Any ideas?

@zeehio

This comment has been minimized.

Copy link
Contributor Author

zeehio commented Mar 29, 2016

can you run pkg-config --cflags portaudio and pkg-config --ldflags portaudio?

I did not consider that brew installs software out of the default paths, I have to use pkg-config inside the configure script to append the right build flags accordingly

@LongBoolean

This comment has been minimized.

Copy link
Collaborator

LongBoolean commented Mar 29, 2016

after installing pkg-config with brew install pkg-config I get this error when running pkg-config --cflags portaudio:

Package portaudio was not found in the pkg-config search path.
Perhaps you should add the directory containing `portaudio.pc'
to the PKG_CONFIG_PATH environment variable
No package 'portaudio' found

And the other command gives this error: Unknown option --ldflags

@zeehio

This comment has been minimized.

Copy link
Contributor Author

zeehio commented Mar 29, 2016

Can you find the portaudio.h in your file system?

@LongBoolean

This comment has been minimized.

Copy link
Collaborator

LongBoolean commented Mar 29, 2016

portaudio.h is in /usr/local/include
there is no portaudio.pc in the file system.

@zeehio

This comment has been minimized.

Copy link
Contributor Author

zeehio commented Mar 29, 2016

I hope I have fixed it with the last commit..

Otherwise, you may want to CFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" ./configure and see if it works, or simply edit config/config after ./configure and try to add those directories.

It's hard to fix stuff without being able to properly test things. Thanks @LongBoolean (and everyone) for your patience. 馃槄

@LongBoolean

This comment has been minimized.

Copy link
Collaborator

LongBoolean commented Mar 29, 2016

We have sound!!!!
We also get a warning when running mimic:

If audio works ignore messages below
If audio works ignore messages above
2016-03-29 18:10:35.639 mimic[19125:184616] 18:10:35.638 WARNING:  140: This application, or a library it uses, is using the deprecated Carbon Component Manager for hosting Audio Units. Support for this will be removed in a future release. Also, this makes the host incompatible with version 3 audio units. Please transition to the API's in AudioComponent.h.

I don't know how big of a deal this is but it is good to know.

@zeehio

This comment has been minimized.

Copy link
Contributor Author

zeehio commented Mar 29, 2016

The warning is related to this issue

https://www.assembla.com/spaces/portaudio/tickets/218-pa-coreaudio-uses-some--quot-deprecated-quot--apis----this-is-by-design-but-needs-to-be-monitored/details

Future PortAudio versions may deal with this.

I will do a final review tomorrow, make sure instructions and documentation are clear and merge

@LongBoolean

This comment has been minimized.

Copy link
Collaborator

LongBoolean commented Mar 29, 2016

I have not tested it out with mycroft on the mac to see if they play nicely together. (I still need to get mycroft set up on the mac) It may not be a big deal, so if you really want to merge to development before I get that tested that is fine by me.

@forslund

This comment has been minimized.

Copy link
Member

forslund commented Mar 30, 2016

Wooo! works on windows as well!

I've added pkg-config --cflags --libs portaudio-2.0 to CFLAGS and I've added ``pkg-config --libs portaudio-2.0` to AUDIO_LIBS and to OTHER_LIBS. not sure if all are needed though. pkg-config might be essential on windows since portaudio can use several different APIs.

However, if you have a working osx and linux I say merge!

Windows isn't compilable without some modifications (took me the weekend to figure out) and I can make the modifications needed for portaudio in my pull request.

zeehio added some commits Feb 25, 2016

[AUDIO] PortAudio module.
Some play_* functions still need fixing
[AUDIO] PortAudio support (play_wave)
There are several versions of the play_wave function in Flite.

 - play_wave: Plays a cst_wave
 - play_wave_from_socket: Read audio from stream and play it to audio device
 - play_wave_client: Send wave to socket

I believe the only used in mimic is play_wave.
PortAudio does not support play_wave_from_socket right now.
PortAudio code review fixes
- Removed unused commented code
- "if (cond) statement" in two lines for readability.
- fprintf(stderr..) -> cst_errmsg
Fix portaudio detection on Mac.
Someone with autoconf experience could do the right thing for
portaudio autodetection.
I do my best.

@zeehio zeehio force-pushed the zeehio:portaudio branch from 41dca01 to dd51ef9 Mar 30, 2016

zeehio added some commits Mar 30, 2016

ALSA and PortAudio detection using pkg-config
Let pkg-config define the right CFLAGS and LDFLAGS for PortAudio and ALSA
detection. Further cleaning of configure script will come in future commits.

pkg-config makes sure all the variables needed for finding headers and linking
are provided, no matter the OS.

If pkg-config is not available or if the user wants to, it is possible to use
ALSA_CFLAGS ALSA_LIBS PORTAUDIO_CFLAGS and PORTAUDIO_LIBS to specify the
respective required variables.

@zeehio zeehio merged commit c7ad20c into MycroftAI:development Mar 30, 2016

@zeehio

This comment has been minimized.

Copy link
Contributor Author

zeehio commented Mar 30, 2016

This pull requests closes issue #10 and issue #13.

Last changes are:

  • Rebase against latest development
  • Use pkg-config in configure script. This may be enough to fix @forslund issues on Windows, otherwise @forslund can make another pull request.
  • Added NEWS.md with PortAudio as a feature. Updated README.md build instructions.

This was referenced Mar 30, 2016

@zeehio zeehio deleted the zeehio:portaudio branch Jul 15, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment