[SFOS] Prevent voice navigation stopping other audio

[paulvt]

If I am listening music while navigation, the music stops after each voice navigation announcement and I have to resume it again (while not paying attention to traffic). This probably also ties into #104.

There are a few options here, I guess:

• Let voice navigation fail/be dropped if there is audio
• Somehow "free" the audio channel after the voice announcement has been made (if possible)
• Dim the current audio, play the voice announcement and undim it again (best way, I'd say)

As a last remake: Pure Maps is an awesome application, almost got all, amazing!

[paulvt]

I have noticed that also a lot of other applications struggle with this, while the core Sailfish OS applications don't really. It might be that the latter have custom handling for this (e.g. music is dimmed for a short while to play an "incoming message" sound for example) through PulseAudio?

[rinigus]

Its not a priority for me right now, but I will look into it when time permits. If someone wants to investigate the issue and propose a solution, that would help a lot.

[bonanza123]

@paulvt maybe that's a thing to ask the jolla devs on TJC or the in the community meetings?

[paulvt]

@bonanza123 Yes, I was considering this too. Had no time for it yet, though!

[theyosh]

Hmm, I looks like quite easy:

Change:

pure-maps/qml/pure-maps.qml

Line 71 in e84bf7f

Audio {

from Audio to SoundEffect https://doc.qt.io/qt-5/qml-multimedia.html#soundeffect

And remove the lines with
autoLoad: true loops: 1

Those do not work. Then, when I have my media player running, the voice will play over the music. Only there is no volume option. So the voice is not always loud enough.

So when we need to crank up the volume, the voice generators should produce WAV files with higher volume in it.

[rinigus]

Is the SoundEffect quieter than Audio? I cannot increase WAV volume, at least using some switch for flite/mimic. Sounds line SoundEffect requires more resources.

Otherwise, we should surely test it. I don't have anything playing on the background, so cannot test that aspect.

[paulvt]

Does that not depend on the relative volume settings for media and notifications/ringtone in the OS?
I mean, I sometimes have that notifications are softer wrt the music that I am listening and the other way around. I might be wrong here. At least, on the desktop PulseAudio keeps track of these volumes for the user.

[rinigus]

Good question - I don't know what is it linked to.

[theyosh]

Hmm, I could only find a ringtone volume setting, no notification volume setting. And when the media player is running, the volume buttons it will increase the playback volume. And I guess that the SoundEffect will just use the same volume. I do get the feeling that when a notification sound is played, the media player volume is reduced... but I could also imagine that...

Also I have not investigated the difference between Audio and SoundEffect object and their limitations.
The new multimedia library in QT should also support 'types of audio', which has a notification type and voice command type. But that did not seem to work on my Sailfish OS. https://doc.qt.io/qt-5/qml-qtmultimedia-audio.html#audioRole-prop

So then I turned to the SoundEffect, and that just worked out of the box. With some minor changes in the QML file.

So, now I have found a way to increase the Voice volume. I was able to install ffmpeg and ffmpeg-tools on my Sailfish OS, and then used ffmpeg to increase the volume: https://trac.ffmpeg.org/wiki/AudioVolume

So, in order to make the audio crank up logic work for all Voice generators, I added the following code to the 'voice_worker' function at https://github.com/rinigus/pure-maps/blob/master/poor/voice.py#L220

if fname is not None:
          tmpfile = fname + ".tmp.wav"
          shutil.move(fname,tmpfile)
          subprocess.call(["ffmpeg",
                     "-i", tmpfile,
                     "-filter:a", "volume=11dB", "-y",fname])
       	  os.remove(tmpfile)

ffmpeg will take only miliseconds to process the WAV files. And you could make a setting for the amount of dB, so that you can make an adjustable Voice volume setting.

I test and made this code just by changing you app code on my device directly. That works the easiest for me. So no pull request at the moment

[rinigus]

I looked into audioRole as well, but it was a failure for me too. As for using ffmpeg - interesting. It adds one more dependency, but its probably OK. I'll have to check it, give me couple of days.

So, we are also looking for solution of #16 , which I closed since I couldn't see additional option on generator. Good job!

[theyosh]

Cool! Did not see that. :) As I did some testing, the value 11dB is the max value. If you go higher, it gets awful. Also, there is a 'loudnorm' option, which does its job, but had the feeling that was not loud enough for me. But it could maybe be added as an option somewhere at sound settings or voice settings.

[theyosh]

Also, ffmpeg itself is installed automatically. According to my zypper:

zypper info ffmpeg
Loading repository data...
Reading installed packages...


Information for package ffmpeg:
-------------------------------
Repository     : jolla                                     
Name           : ffmpeg                                    
Version        : 4.1.1+git1-1.2.1.jolla                    
Arch           : armv7hl                                   
Vendor         : meego                                     
Installed Size : 7.9 MiB                                   
Installed      : Yes (automatically)                       
Status         : up-to-date                                
Source package : ffmpeg-4.1.1+git1-1.2.1.jolla.src         
Summary        : FFmpeg video encoding and decoding library
Description    :                                           
    FFmpeg: a complete, cross-platform solution to record, convert and stream audio and video.

But you need the ffmpeg-tools to get the ffmpeg binary.

zypper info ffmpeg-tools

Loading repository data...
Reading installed packages...


Information for package ffmpeg-tools:
-------------------------------------
Repository     : jolla                            
Name           : ffmpeg-tools                     
Version        : 4.1.1+git1-1.2.1.jolla           
Arch           : armv7hl                          
Vendor         : meego                            
Installed Size : 300.6 KiB                        
Installed      : Yes                              
Status         : up-to-date                       
Source package : ffmpeg-4.1.1+git1-1.2.1.jolla.src
Summary        : FFmpeg tools package             
Description    :                                  
    Development tools for FFmpeg - a complete, cross-platform solution to record, convert and stream audio and video.

And they are available by Jolla itself, so I think you need only to depend on ffmpeg-tools. But that should be tested, as I have already installed it.

[theyosh]

Hi,

is it at least possible to update the code in https://github.com/rinigus/pure-maps/blob/master/qml/pure-maps.qml#L76 to:

SoundEffect {
        id: sound
}

basically, that is the answer to this issue. It is not the nicest option, it does do the job. Now I have to patch that. For now, the volume is secondary. I do like my music more.... ;)

[rinigus]

I will have to test it again, but I think that SoundEffect was not played via Bluetooth on my device. It could be down to the port or general SFOS feature. But let me test again to be sure

[rinigus]

Tested it finally. On my device, there are issues with Bluetooth (BT) switching when using SoundEffect that are not there with Audio. Namely, while running Pure Maps:

• on phone speaker, test is OK
• switch over to BT, test is OK
• swich off BT, test is silent. Get test working after restart of Pure Maps

For Audio, I haven't seen this issue. Tested using Testing feature in Pure Maps preferences. So, as it is, SoundEffect is not really a reliable solution

[paulvt]

When applying the change mentioned above and testing it with my BT headset I have different test results:

• on phone speaker, music plays, notification plays through music, test OK
• connect headset, music plays on headset, notification plays through music on headset, test OK
• disconnecting headset, music stops, notification plays (over nothing), test OK?

So maybe something changed in Sailfish OS in the meantime (they noticeably did fix some BT/audio routing stuff for me), our tests are different or something else is going on?

On a different note, I was confused by the discussion about the notification volume, because I had assumed that using the "sound effect" style, it would work the same as an incoming sms/XMPP message notification, namely that the media volume is slightly dampened for the duration, but that does not seem to be the case, that is very unfortunate! So, I still wonder how they do that in.. Lipstick?

[rinigus]

I will retest with the latest SFOS with SoundEffect, I don't understand your third test result. Did you hear the notification over speaker? I presume you tested it with SoundEffect, right?

[paulvt]

Yes, with SoundEffect patched in the QML and yes, sorry, the notification was still sent over the speaker, but not over the music, as that is stopped when one disconnects the BT headset.

[padu22]

After struggling with this on UT, I've opened a thread at the UBports forum sound issue playing music and voice navigation. There was a comment about PA providing "alert" audio streams that damp all other audio. Maybe that helps?

[rinigus]

Pure Maps is using QML Audio for sound. We tested SoundEffect as well on SFOS, but it had some issues with the bluetooth for me - will have to retest it. I don't know how to patch it to PA alert. You could see if SoundEffect will work for you (change it at https://github.com/rinigus/pure-maps/blob/master/qml/pure-maps.qml#L88) and whether it works better.

[padu22]

Reading QTMultimedia docs, I stumbled upon Audio property audioRole, which provides a value AlarmRole. So, I added the line

        audioRole: AlarmRole

to the Audio section, but when trying to run this on Ubuntu Touch (UT), the app refuses to start with the following log message

QQmlApplication failed to load component

file://opt/click.ubuntu.com/pure-maps-slim.jonnius/1.28.1/share/pure-maps-slim.jonnius/qml/pure-maps-slim.jonnius.qml:92
"Audio.audioRole" is not available in QtMultimedia 5.2.

A web search indicates that property audioRole might have been introduced in QT 5.6. Would it be possible to update that component? (Please forgive my ignorance, I have no idea whether that component is actually bundled with Pure Maps or provided by my OS (UT)).

[rinigus]

@padu22: try to bump https://github.com/rinigus/pure-maps/blob/master/qml/pure-maps.qml#L20 to 5.6. It is part of UT for you.

[padu22]

Thanks! Pure Maps starts indeed again when using audioRole: AlarmRole and requesting QTMultimedia 5.6. Yet, again I find a log message

file://opt/click.ubuntu.com/pure-maps-slim.jonnius/1.28.1/share/pure-maps-slim.jonnius/qml/pure-maps-slim.jonnius.qml:92:
ReferenceError: AlarmRole is not defined

Obviously, I'm stabbing in the dark.

[jonnius]

I think it should be Audio.AlarmRole instead.

[padu22]

I think it should be Audio.AlarmRole instead.

That's it! With Audio.AlarmRole other audio streams are damped. Though, alarm audio stream seems to be played at heart attacking maximum volume, regardless current audio volume. Audio.NotificationRole seems to be more appropriate.

[rinigus]

@padu22, what about adjusting volume by buttons when using notificationrole?

[jonnius]

You can use uVolMan to control the different volumes btw.

[padu22]

Have to get an API key first. Test voice is too short to do proper testing. Will report back.

[jonnius]

You can test with OSM Scout Server. No need for an API key then.

[padu22]

While the test voice works, I get no voice directions when doing true navigation, whether I use audioRole or not. I'm on the slim version with Pico TTS only, currently. Restarted device multiple times. The official full version installed in parallel has voice output. Any advice?

[jonnius]

I can't reproduce the issue. Are you sure that voice is enabled in the navigation menu?

[padu22]

Thanks! Works now. Need to find some more time for real life testing.

[padu22]

First quick tests show that notification volume can be changed via buttons. Volume then also applies to the next voice directions. Volume of background music (tested music app only) and notification are completely unrelated. Volume change always applies to current foreground audio source. Getting timing right, I have been able to get quieter voice directions than background music. Tested with phone speakers as well as Bluetooth headset with Nexus 5. Tested A2DP profile only, that is, volume change via phone buttons only. Can't test with AVRCP profile.

Will continue testing and report back if there are additional findings.

[padu22]

I've tested one more thing so far: Of the audio roles listed here, the entertaining ones MusicRole, VideoRole, and GameRole pause background music without resuming them. SonificationRole showed somewhat buggy behaviour. I didn't test UnknownRole and CustomRole.

The remaining audio roles VoiceCommunicationRole, AlarmRole, NotificationRole, RingtoneRole, AccessibilityRole all damp background music for the time the new audio stream is played. That is, any of these roles would do. Though, "notification" seems a good semantic fit.

[padu22]

Using NotificationRole brings another improvement in connection with a VW carkit. Without audioRole, navigation directions pause music app. When not resuming music manually, carkit – for whatever reason – automatically mutes Bluetooth audio (showing a cancelled speaker symbol). When the next navigation direction is coming in, carkit unmutes Bluetooth audio again, but with a noticeable delay. Resulting in cut off directions or no audible directions at all. Then, audio is auto-muted again. I can reproduce the old behaviour with MusicRole. (Note to testers, auto-muting might happen only when playing music in parallel.)

With NotificationRole, I have not yet observed carkit auto-muting Bluetooth audio.

Phone volume buttons affect current Bluetooth foreground audio stream. Since only A2DP profile is used, carkit volume controls affect carkit side volume only. Therefore, carkit volume controls have an effect on all (Bluetooth) audio streams, not the current foreground audio stream only. (Things might be different with AVRCP profile.)

Tested with Nexus 5 and OnePlus One (both Ubuntu Touch, OTA-12) against VW Composition Media carkit.

[rinigus]

OK, so from that testing, do you recommend NotificationRole then?

[padu22]

Speaking for UT only: By any means, yes!

For other platforms, I suggest some prior testing for obvious side effects. After all we're talking Bluetooth. What could go wrong with that? 😄

[rinigus]

Looks like we at least can use it on SFOS. I will test it in the car when I can

[rinigus]

To simplify testing by others, I made the changes in master: e34df3c

[rinigus]

Should be fixed in 5370762, closing. Reopen if bit fixed after the next release

[paulvt]

I have tested it with my Bluetooth headset on (on Sailfish OS 3.4). It works in so far that it pauses music during a notification and it resumes the music again afterwards, great! Except, it turns out that the the navigation notification come out of my phone speaker instead also out of my headphones, is this normal?
(When I was cycling I was even wondering why music was just pausing, not hearing the sound coming from my pocket.)

Do you want me to reopen, file a new issue or is this going to be it?

[rinigus]

Frankly, I tend to consider this as a bug of SFOS. In some respect, Pure Maps does use the correct stream for notifications but they are voiced over a wrong channel. Maybe we should open an issue at SFOS forum.

[Karry]

I just found out that audio class also affects pulseaudio routing. It is configured in /etc/pulse/xpolicy.conf on SFOS. When route_audio flag is added to navigator group, navigation sound is played via bluetooth headset:

[group]
name   = navigator
flags  = set_sink, set_source, route_audio
sink   = droid.output.media_latency@equals:"true"
source = droid.input.builtin@equals:"true"

So, for proper voice routing and good behaviour with music player, there are two options:

• add route_audio flag to navigator group in /etc/pulse/xpolicy.conf
• change audio class to game ;-)

[rinigus]

@Karry: What a find!!! Please file an issue at FSO and hopefully it will get fixed on device level. As the same code is used for different platforms, it would be silly to start swapping to game instead. I will have to test it on my ported devices...

Thank you very much for reporting it here as well. It is just thanks to you the original solution was established and now I hope it will be restored in future.

[Thaodan]

I just found out that audio class also affects pulseaudio routing. It is configured in /etc/pulse/xpolicy.conf on SFOS. When route_audio flag is added to navigator group, navigation sound is played via bluetooth headset:

[group]
name   = navigator
flags  = set_sink, set_source, route_audio
sink   = droid.output.media_latency@equals:"true"
source = droid.input.builtin@equals:"true"

So, for proper voice routing and good behaviour with music player, there are two options:

* add `route_audio` flag to `navigator` group in `/etc/pulse/xpolicy.conf`

* change audio class to `game` ;-)

I wonder did you report that in the forums?

[rinigus]

Yes, @Karry did: https://forum.sailfishos.org/t/navigation-instructions-doesnt-play-via-bluetooth-headphones-but-phone-speaker/7826/6

[JacekJagosz]

Coming back to the original issue with playback being completely stopped, the fix you implemented in 2020 is unfortunately not complete. With native music applications it does in fact work, so after the voice finishes the native music gets unpaused. But Android applications don't do so.
So if I am using Spotify Android app I have to unpause it each time after a navigation voice command. And Spotify does show the mediaplayer controls on the lock screen, so I don't get why SFOS doesn't play the sound again.
Is there a way to completely disable pausing of music? Also I guess this is something that should be reported on SFOS forum, but I don't know how to properly describe the media types and such.
PS I am on Android 4.5, maybe that is a regression?

[rinigus]

In principle, this is not Pure Maps bug, but a bug of underlying OS. Pure Maps does not issue a command "STOP" to other apps running on device, it just plays its own sounds. The fact that OS stops other audio streams and forgets to resume them, is up to OS. On Pure Maps side, after @Karry's findings, correct audiostream was selected that is sufficient to indicate to OS the intention. Unless Pure Maps has to use some other stream, I don't see much we should do here.

Indeed, it would be good to report the issue to SFOS forum. Just state what you observe and feel free to link here - no need to use specific terms.

Closing it here as it is not Pure Maps bug.