bad audio quality in Ubuntu 20.04

[Thanh-Binh]

Currently, audio quality is too bad with echo and noises.
Maybe sample rate for audio (16000) is set not correctly, or other problems

[jrsnen]

I'm sorry you have encountered this issue. I was hoping that the audio issue would have been fixed or at least functional on Linux.

Setting the following line to 48000 or otherwise trying to find the correct sample rate might help:
https://github.com/ultravideo/kvazzup/blob/2eacc81468a546fa644e3523db8ed8d8a58940d5/src/media/processing/filtergraph.cpp#L49

If that does not fix the issue, you may try enabling opus. Currently the opus is disabled on Linux since it did not work earlier. You can try re-enabling it by removing this ifdef
https://github.com/ultravideo/kvazzup/blob/2eacc81468a546fa644e3523db8ed8d8a58940d5/src/initiation/negotiation/mediacapabilities.h#L15

and using the windows version on linux (opus requires the sample rate to be 48000 in filter graph). This might solve the audio issues or make them worse.

The permanent solution for this problem would involves getting the opus working, if it does not work currently.

There are also DSP options in audio settings. Make sure those are all enabled. The speex AEC is not very good (but better than nothing), but that is currently not something I can do anything about.

[Thanh-Binh]

I did exactly as you wrote + using AEC and denoise, but have always echo problem

[jrsnen]

What is your setup? Are you calling yourself or another computer? Is the echo present if you use headphones?

[Thanh-Binh]

here is my audio setup:

[audio]
Device=default
DeviceID=0
aec=1
agc=1
bitrate=24000
complexity=10
denoise=1
dereverb=1
signalType=Auto

I called myself and auch call other computer

[jrsnen]

Is the echo present if you use headphones?

What I'm trying to understand is if you are dealing with a bug or if the echo cancellation is just not good enough. If I understood correctly, you are talking about a bug in audio quality on Linux?

[Thanh-Binh]

my headphone does not function well, so that I test only with 2 computer which are separately in different rooms.
But I have: echo + very strong noise with high frequency.
I think about a bug in audio

[jrsnen]

In that case, it seems Kvazzup still has an audio bug on Linux in Audio. Unfortunately I don't have a good quick solution for this one. It may take some time before this is fixed.

[Thanh-Binh]

Yes. I am trying to save audio data from microfon to file to know where audio problem is happened

[jrsnen]

Thank you, that would be very helpful. The issue is probably either in audio input or audio output.

[Thanh-Binh]

But it is not easy in your framework. I tested it outside your framework, and showed the audio capturing works well.
I believe the audio processing chain makes noisy...

[jrsnen]

Here the input data is copied to sample: https://github.com/ultravideo/kvazzup/blob/2eacc81468a546fa644e3523db8ed8d8a58940d5/src/media/processing/audiocapturefilter.cpp#L158

Copying this data to file and listening would help with this issue by identifying if the issue is in Kvazzup input or output.

[Thanh-Binh]

how can I save it into file? Could you pls give me a short code, because I am very beginner with Qt? Thanks

[Thanh-Binh]

@polusto I saved it and replay well data. That means input is read correctly, but output is somehows not correctly

[Thanh-Binh]

@polusto i tested today morning with a headset and had also heavy echo and noise problems with audio.
Do you have any solution idea?

[jrsnen]

Newest master saves the state of the mic button, so you can close it as a temporary solution.

If you want to help, can you save the audio data from this https://github.com/ultravideo/kvazzup/blob/ec7f69d3d7726927307a344c84578f4aa8889e8c/src/media/processing/audiooutputdevice.cpp#L167

That is the last point at which the data is handled by Kvazzup.

[Thanh-Binh]

@polusto i found the current SW architecture with separate modules for mic, screen share and camera is very good.
Congratulation!
One thing I think we can do better by handling different cameras as follows:
Instead of using Boolean for camera states we use an index of cameras: 0 for inactive, 1 for camera 1, and 2 for camera 2.
Using maxNumberCamera in ini-file
When you click on camera button, it changes camera index from 0 to maxNumberCameras and back, so that you can select video from different cameras.

what do you think?

[jrsnen]

There is already a camera selection in the first page of settings, but it may not work on Linux due to a bug in Qt. I cannot comment on other solutions.

[Thanh-Binh]

Thanks.

I really do not find any way to save raw audio data at the last point of Kvazzup into a file!
Do you know any solution?
Thanks

[jrsnen]

I thought you saved the data at microphone? Why not use the same solution?

You can use any method available in Qt or in c++. I don't have any code for it at them moment. Open a file stream and write the audio data there I guess.

[Thanh-Binh]

When I save data at microphone, the audio quality is very good and without any echo.

I want to save raw data at this point for making sure if the audio quality will be changed after processing.
I recherched internet, but do not find any Qt example to save raw audio into file so that I can not debug audio problem.

Do you have any audio problem under Windows?

[Thanh-Binh]

@polusto to be honest, I am not able to solve this audio problem.

Could you please help me? Thanks

[jrsnen]

Now that I think about it, recording audio requires more effort than I first thought, because it needs a container with necessary information.

I will fix this issue, but it probably take some time since I don't know what is causing it. I also have other responsibilities I must take care of.

The audio on windows works fine.

[Thanh-Binh]

@polusto thanks for your help.
Fine that your window version works well.
Could you please share me your windows binary or libraries?

[jrsnen]

I made a new release version 0.09 of Kvazzup that includes both the Windows binaries and the dependency libraries compiled with Visual Studio. Available here: https://github.com/ultravideo/kvazzup/releases/tag/v0.09

[Thanh-Binh]

Which VS version do you use for creating libraries?

[jrsnen]

Some of them are a bit older (maybe 2017), but VS 2019 should work.

[Thanh-Binh]

thanks

[Thanh-Binh]

I run your app in my laptop win 10, but have the following problem:

1. Built-in camera turns on but no image
2. Crash by calling myself
   Any idea?

[Thanh-Binh]

@polusto I have just developed a Audio Framework for testing qtAudio and AECProcessor and found that AECProcessor does not work properly under Linux, it makes the audio quality worse.

I found other bugs in your framework:

1. the activating AEC in ni-file has only influence on AECProcessor, even though it should change setting in other modules, where no AECProcessor will be called if aec=0 in ini-file
2. kvazza-modul is not able to change image resolution automatically if we change from built-in camera to screenshot and vice verse, so that it crashes by changing.

[jrsnen]

Are you saying that without AECProcessor, the audio flow works? I can look at it at some point.

As for the bugs:

1. AEC is not supposed to disable AEC processor. It just disables the Echo cancellation. The module should probably be renamed to SpeexDSP or something so it is not a bug.
2. What do you mean by screenshot? Screen sharing? Can you make an issue about this if it is a bug in Kvazzup master? I haven't encountered this.

[Thanh-Binh]

@polusto in my test qt framework ( only read audio from mic and play it), without AEC (I.e. no use processFrameInput() and processEcho() methods) my output audio has echo but no noise!
With AEC I hear noises exactly same as Kvazzup has!

[Thanh-Binh]

By screen sharing: can you change screen sharing mode to camera mode in realtime callings?

[Thanh-Binh]

@polusto One question about data flow from camera or screen sharing module to internet sender module?
Camerafilter - rgb32toyuv - kazzaar - upgSender?
Thx

[jrsnen]

To keep these issue threads readable I will only answer questions related to audio quality in this issue. If you have issues with video or screen sharing, please make a new issue. It is very hard for me to keep track which issues are fixed and which are not if they are all discussed in one issue.

[Thanh-Binh]

Ok i move questions related video into another new issue

[jrsnen]

I have managed to set up a virtual machine with Ubuntu and can replicate this audio issue. I will start working on the fix, but cannot say when it will be ready.

[Thanh-Binh]

Fine that you can identify this issue in Ubuntu.
Thanks

[Thanh-Binh]

@polusto any success by bug-fix?

[jrsnen]

I fixed few other bugs. I will update and close this issue when this bug is fixed.

[jrsnen]

This issue is caused by audio output on Linux wanting smaller audio samples than Kvazzup currently supports. It will take some time to modify Kvazzup to support smaller audio samples.

[jrsnen]

The audio quality should now be better on Linux.

There are still some audio issue left however. The output filter cannot keep up with audio flow for some reason. I will continue to improve the audio flow next week.

[Thanh-Binh]

fine. I have just tested. It is much better than the old version.
Yes, it has also the echo problem. Thank you very much for sharing me and have a nice weekend...

[Thanh-Binh]

@polusto any improvement by audio quality?
Do you know why speexdsp can not solve the echo problem successfully?

[jrsnen]

I have some improvements coming, they just need some testing and fine tuning.

It may be that I input the echo frames to Speex DSP too soon, rendering the AEC useless. I found that they have a diagnostics tool for detecting this. If this does not solve it, I'm out of options.

I would guess that it will at least another week before I can get everything finished, because new issues keep popping up.

[Thanh-Binh]

@polusto thanks for your update information

[Thanh-Binh]

@polusto did you fix this bug? If yes, I will be very happy for bring my app to work soon!
Thanks

[jrsnen]

I pushed improvements to audio to master. The main improvement was that the sample rate is now much higher. The sample rate for Speex AEC was not set causing it to down sample the rate to 8000 (instead of 48000).

For the echo I managed to figure out that there needs to be less delay between the playback/speaker audio frames and the input/mic audio frames. The AEC is now works a little bit, but there is still much work to do to solve all the issues with it. If you are in a hurry, you can try changing the new playback delay or aec filter tail length variables to get better results. Currently, the recommended way to get rid of echo is to have one computer either muted or for one computer to have headphones connected. Either of these eliminates echo. In any case, the Speex AEC will probably never get rid of all the echo. I'm basically trying to get Speex AEC to do something to help a bit if that is not an option.

Additional note: AEC does not and will never do anything when calling oneself. Only calls between two computers are supported for AEC.

I have an important deadline coming in two weeks. I will try continuing to work on this as well, but I will have less time than before.

[Thanh-Binh]

@polusto best thanks for your update! It is also much improvement!

[Thanh-Binh]

@polusto do you have any improvement on audio quality? Thanks

[jrsnen]

Unfortunately, I haven't had time to work on the audio. I've taken on the responsibility for uvgRTP repo which has taken much of my time.

@Thanh-Binh What sort of audio issues are you experiencing at the moment?

I am considering integrating RTAudio into Kvazzup to reduce audio latency, since Qt is not very good for real-time audio. After the latency has been reduced, I will try to get the timing of echo cancellation closer to optimal, but I wouldn't hold my hopes up for much improvement in echo removal since I don't have much hope for Speex and I am not aware of better alternatives.

[Thanh-Binh]

@polusto a big problem is echo, so that we can not use it comfortably

[Thanh-Binh]

@polusto why do you choose RTAudio? In other applications of mine, I am very happy with SFML audio!

[jrsnen]

RTAudio is meant for real-time audio and is only focused on audio issues. SFML is a large framework and I try to avoid including large frameworks if I can avoid it, even if it would solve this issue.

[jrsnen]

This issue is getting pretty long and I'm closing it since most issues have been fixed apart from the echo.

For the echo issue, try the following if possible and you have not tried already:

• Adjust Audio settings -> Speec AEC -> AEC Playback delay until you find a point where the echo is cancelled. Make sure AEC Filter length is not too long. I'm not aware of any solution that would bring better results than this.
• Closing one of the microphones removes echo
• Getting a better microphone might help with the echo
• There is no echo if both use headphones

Please note that AEC will never work with self calling.

I feel my situation is such that putting more work into this is not guaranteed to improve the echo issue because I'm not at all confident that Speex DSP is good enough for this task, especially now that it is finally working and does not seem to help much.

I will eventually get back to this (hopefully in the fall), but from my point of view there are more critical issues with Kvazzup (such as conferencing and SIP functionality). You may create a new issue, and I will eventually start working on it if I have ideas. You may also try to fix the issue, but I'm not sure what can be done outside adjusting playback delay.

[Thanh-Binh]

@polusto please let me know if your RTAudio integration is finished.
I will try to follow your hints for avoiding echo problems

[jrsnen]

@Thanh-Binh I created issues for both the AEC (#15) and RTAudio (#14). There you can follow the progress of the issue.

You may also follow releases where I will probably mention at least RTAudio. Unfortunately, I have to start advancing Kvazzups main features at this point and it will probably be towards the end of this year when I can continue with audio issues unless there is something critical and relatively easy to fix.

[Thanh-Binh]

Ok