[Safari] The audio is broken on certain MacOS/Safari versions #4411

Softvision-RemusDranca · 2021-07-14T12:37:53Z

[Notes]:

The issue is reproducible when communicating with other users and also when playing spawned videos.
The performance also seems to be affected.

[Affected Versions]:

Safari - Version 14.1.1 (15611.2.7.1.6, 15611)

[Affected Platforms]:

MacOS Catalina 10.15.7 (19H1217)

[Prerequisites]:

Have a room opened on "https://hubs.mozilla.com/".

[Steps to reproduce]:

Spawn a video inside the room.
Play the video for a small period of time.
Observe the behavior.

[Expected results]:

The video is correctly played.

[Actual results]:

The audio is broken.

[Additional Notes]:

I reproduced this issue using different scenes.
I reproduced the issue when playing videos inside the room.
I reproduced the issue on 3 different devices using the same version.
According to statcounter macOS Catalina is the most used version for MacOS.
Could be related to MacOS Safari: Strange Echo after a few minutes of using #4288
Attached is a screen recording of the issue where you can see the performance being affected: link.
Here are the logs received while reproducing the issue: link.

┆Issue is synchronized with this Jira Task

Softvision-RemusDranca · 2021-07-14T14:36:03Z

I've encountered an additional behavior while testing. If for example a YouTube video is playing in a different tab, when focusing the Hubs tab the video's audio from the Youtube tab is broken.

Softvision-RemusDranca · 2021-07-16T15:55:29Z

I've managed to get some help from another team and they were also able to reproduce the issue using:
- MacBook Pro macOS Big Sur Version 11.3.1 (20E241), Safari Version 14.1 (16611.1.21.161.6).
- MacBook Pro macOS Big Sur Version 11.4 (20F71), Safari Version 14.1.1 (16611.2.7.1.4).
- I was also able to reproduce the issue on Iphone SE 2 having OS Version 14.6, but the audio was not as cranky as on MacOS. Here is a recording where you can see the video playing being interrupted.
The sound was cranky after approximately a half of minute of playing media or communicating with others.
I was only able to test on MacBook Pro devices, I could probably get some help with an iMac next week.
On a different matter I also ran into some performance issues, on MacOS Catalina 10.15.7 (19H1217) using Safari Version 14.1.1 (15611.2.7.1.6, 15611). Here is a recording and also the console logs. Not really sure if everything is connected but I could open a separate issue for this if necessary.

takahirox · 2021-07-16T18:28:40Z

On my MacBook Pro, macOS Big Sur 11.4, Safari Version 14.1.1, I can reproduce the audio issue.

But I can't reproduce the performance issue. According to the console logs you shared, the shader compile fails and I speculate it's the cause. But I don't see shader compile error on my end. Could it be scene or object specific issue? Cab you create a scene and pin objects where you encounter the compile error and the performance issue?

brianpeiris · 2021-07-18T15:34:42Z

I'd suggest we open a new issue for the performance problem.

brianpeiris · 2021-07-18T16:26:27Z

I've reproduced the audio distortion issue consistently in Safari 14.1.1 on macOS (11.4 MacBook Air M1 2020) and iOS (14.6 iPhone 11). The issue reproduces at different rates in different environments, but it does happen eventually in all cases. The issue does not occur on Chrome or Firefox on macOS.
I've reproduced it with a simplified test case, using only WebAudio. https://brianpeiris.github.io/spatial-audio-test/ The key factor is a PannerNode that updates its transform continuously. I've also posted my details in an existing webkit bug report: https://bugs.webkit.org/show_bug.cgi?id=227199
I think our only option for now is to disable spatial audio in Safari until this bug is fixed in the browser. We might emulate attenuated audio instead.

takahirox · 2021-07-19T19:07:02Z

Nice investigation!

I think our only option for now is to disable spatial audio in Safari until this bug is fixed in the browser. We might emulate attenuated audio instead.

I agree with it. Disabling spatial audio should be much better than broken audio.

Regarding the performance issue, I confirmed that CPU usage on Mac is very high even on a simple Three.js positional audio example so I speculate the performance problem may be very relevant to the audio issue as @Softvision-RemusDranca mentioned.

@brianpeiris Did you already start to work on the workaround? If not and you don't have time, I can take it. I hope we add the workaround soon and test to check if the performance issue can also be resolved.

brianpeiris · 2021-07-23T11:13:58Z

#4442 Fixes avatar audio issues for the next Hubs Cloud release, but I'll have to deal with the current version of the code separately for HMC, since the new audio code is quite different.

takahirox · 2021-08-09T21:20:58Z

I would like to try to apply the same change to master, just in case after an audio related PR #4487 will be merged, unless @brianpeiris wants to work on it because hubs.mozilla.com is very useless on MacOS/iOS Safari and very hard to even test on them now.

brianpeiris · 2021-08-09T21:25:06Z

Go ahead @takahirox I'm going to continue focusing on Hubs Cloud.

takahirox · 2021-08-10T21:42:00Z

OK, made a PR #4503

takahirox · 2021-09-02T03:31:20Z

Thanks to #4594, the distorted audio on MacOS Safari seems to be resolved by sacrificing PositionalAudio.

The iOS Safari one seems much better than before, but I realized it stills has some audio distortion issues. After being a room for like 5-10mins, the audios are distorted and broken.

How to reproduce.

Enter a room on MacOS Safari and iOS Safari
Speak and move around like for 5-10 mins (It seems the moving around is necessary to cause the problem, or it makes the problem happens earlier)
Confirm on iOS Safari that audios are distorted, broken, and laggy.
Confirm on MacOS Safari that audio "from the iOS Safari" is distorted and broken. After this problem happened, I entered the same room on MacOS Firefox and the audio from/to MacOS Firefox is fine on MacOS Safari. So the problem seems to be on iOS safari end.

@Softvision-RemusDranca reported the same problem.

Also, I can reproduce the problem with just a video. I spawned a video on iOS Safari, move around for like 5-10 mins, and then the audio started to be distorted and broken.

I couldn't reproduce the problem on MacOS Safari so far.

When the problem happens, my iPhone is very hot. I have a impression that the problem happens when CPU is busy (or it can be Audio buffer problem). So potentially the problem could be related to #4423.

takahirox · 2021-09-14T16:08:46Z

Regarding the Positional (Panner) audio bug fixed in #4594, we thought it was a Safari specific bug but it seems that I can reproduce the problem even on iOS Chrome/Firefox. (I speculate iOS Safari/Chrome/Firefox use the same core webkit library?)

Currently we disable Positional audio for Safari but we may need to disable it for Safari or iOS.

Can anyone here check if you can reproduce the problem on iOS Chrome/Firefox on your end, too?

johnshaughnessy · 2021-09-14T17:12:12Z

@takahirox If we can't diagnose and fix the root cause, I wonder if there's a workaround we can do where we disconnect from dialog, tear down all the voice connections, and then reconnect/recreate them every 5 minutes or so.

takahirox · 2021-09-14T17:25:04Z

@johnshaughnessy It might be worth to try, although I'm not really sure if it can resolve the problem. (And I'm really not sure yet if we have a workaround)

takahirox · 2021-09-16T00:25:54Z

I made a simple test case.

https://takahirox.github.io/ios-webaudio-test/

It plays music with or without WebAudio.

On my iOS 14.8, the audio with WebAudio is broken after about two minutes regardless of browsers (Safari/Chrome/Firefox) while the one without WebAudio isn't broken.

I'm going to report this problem to webkit bugzilla. (I guess all the major iOS browsers use the same webkit engine for WebAudio.)

WebAudio on iOS doesn't seem to be stable. I googled and found that WebAudio on iOS has repeatedly had bugs, fixed them, and had regression bugs again.

I'm thinking of avoiding to use WebAudio until iOS WebAudio will be matured enough.

A workaround in my mind is to add a new (Three.js) audio class which plays audio just with audioElement.play(), doesn't use WebAudio API, and is compatible with Three.js Audio API. It doesn't have panner, but has audio attenuation by controlling audioElement.volume.

takahirox · 2021-09-22T00:42:18Z

TLDR: I would like to suggest to force to disable echoChancellation for iOS to reduce the crackle audio noise.

MacOS BigSur 11.6 and iOS 15 have been released and I tested some audio on them.

I first thought we had a single root issue causing the broken audio problem even without panner node, but It seems we had two (or more) audio issues in iOS 14. One of them seems to have been resolved in iOS 15. But another one seems to still exist.

Distorted and broken audio problem with WebAudio createMediaElementSource()

This is the problem which seems to have been resolved in iOS 15.

WebAudio createMediaElementSource() seemed to cause audio noise in iOS 14. But it seems to have been resolved iOS 15. I confirmed it with simple test case https://takahirox.github.io/ios-webaudio-test/

Crackled audio with echoCancellation

With fixed createMediaElementSource(), the audio in iOS 15 sounds better. But the audio sounds still crackled. It seems another issue still exists in iOS 15.

The root issue seems echoCancellation of getUserMedia(). Somehow user media (mic) seems to cause a audio noise.

(Finally!) I made a simple test case reproducing the crackle noise problem on iOS

https://takahirox.github.io/ios-webaudio-test/video_user_media.html

If you click "run" button on your iOS, you would hear crackled audio from the video.

A workaround seems to be disabling echoCancellation when creating user media for mic. If you uncheck echoCancellation in the test, you would notice that the crackle noise is reduced. (Somehow the noise is not perfectly removed on my iOS, but much better.)

I also tried to disable echoCancellation in the preference on Hubs and confirmed that the audio becomes much better.

So, I would like to suggest to force to disable echoCancellation on iOS.

getUserMedia() can make the audio quieter

While doing the echoCancellation test, I noticed that if you call getUserMedia() the audio from the video can be quieter.

You can test it in the test by unchecking "Use getUserMedia() for mic".

I haven't come up with a solution yet. We might be able to try other getUserMedia() options.

Panner node still seems problematic in iOS 15

I also checked if we still need disabling panner node workaround for Safari. It seems yes, we still need it. I tested panner node on Safari. But the audio is broken after being in a room after some minutes. It sounds even worse then before. The audio problem can be reproducible even in the Three.js audio orientation example.

https://threejs.org/examples/#webaudio_orientation

takahirox · 2021-09-27T23:50:54Z

I discussed the crackle audio noise problem and we found that it might be old iPhone specific problem.

We don't have many enough iOS devices, so iOS users, can I ask for volunteers?

I'm happy if you run the test to check whether you can reproduce the problem I encountered, and report to me here whether you can reproduce or not.

Go to https://takahirox.github.io/ios-webaudio-test/video_user_media.html on your iOS device with Safari (and other browsers)
Every item is checked and press run button
You hear crackle noise from video audio
Reload the page, uncheck "echoCancellation", and press run button
You hear less noise

Softvision-RemusDranca · 2021-09-29T14:13:43Z

@takahirox I have the same result as @jure , no crackling noise with or without echoCancellation. I used Iphone SE2 with Safari 15.

takahirox · 2021-09-29T19:14:26Z

Thanks all.

I also asked for the test at different places and we have tested five iPhone devices in total.

The models on which the audio noise problem is reproducible

iPhone 8+ (released in 2017)
iPhoneX (released in 2017)

The models on which the audio noise problem isn't reproducible

iPhone XS Max (released in 2018)
iPhone11 (released in 2019)
iPhone SE2 (released in 2020)

Sounds like the problem is reproducible only on old (2017 or before?) models (maybe meaning less powerful models). We couldn't find global iPhone model market share numbers, but I guess iPhone users generally use newer models?

So I don't think we should force to disable echo cancellation. (#4679)

We can share the tips with users instead

we suggest to upgrade iOS to iOS 15
if you still hear audio noise, we suggest to disable echo cancellation in the preference

takahirox · 2021-09-29T19:16:57Z

@Softvision-RemusDranca

Sounds like the broken audio issue shouldn't happen in iPhone SE 2 + iOS 15 and with the patches we recently applied. Do you mind if testing again to check if the problem is still reproducible?

vincentfretin · 2021-09-30T09:26:05Z

No crackling with echoCancellation on or off with iPad 2018 6th gen iOS 14.8.

Softvision-RemusDranca · 2021-10-04T14:25:07Z

@takahirox Unfortunately, I was able to reproduce the issue using Iphone SE2 after approximately ten minutes but it was only for a very short moment, after that the audio was working again. The issue reproduced while I was playing a video in the background and also having a conversation between two users.

takahirox · 2021-10-04T19:02:47Z

@vincentfretin

Thanks for testing. The problem doesn't seem to be occurred on iPad.

@Softvision-RemusDranca

It's good to know that at least it sounds better than before. But you couldn't reproduce the audio noise problem on https://takahirox.github.io/ios-webaudio-test/video_user_media.html but can still reproduce the noise problem on Hubs. Hm, there might be another issue or high resource usage may affect the audio quality.

I think it's hard to further investigate. I will report the problem to Apple devs so far.

takahirox · 2021-10-04T21:12:11Z

Reported to https://bugs.webkit.org/show_bug.cgi?id=227199 Please post there if you have additional context.

brianpeiris · 2022-04-08T23:48:20Z

Just to add more data points, I tested this today with my iPhone 11, running iOS 15.4 and Safari 15.4. I used two bots with movement and audio, and ran the test for 30 minutes. Audio did not distort or crackle. I guess this is primarily due to #4594. The Safari team may have also fixed some underlying issues, but we'd have to test with positional audio re-enabled to verify that.

I'd suggest we can close this bug, since it is not an issue for current models and the latest iOS/Safari. If it is still an issue on older iPhone models, and we care about fixing that issue, we can open a new ticket with up-to-date, summarized information about the issue.

pigubaoza · 2022-05-25T04:51:38Z

hi i would like to check if positional audio is re-enabled? it looks like it's still disabled here https://github.com/mozilla/hubs/blob/fe4eb6ffbc94fb411f8d98754376aed7e0c2b01a/src/systems/sound-effects-system.js#L141
I have also filed an issue that I think might be related: #5458

takahirox · 2022-05-25T15:29:29Z

Thanks for the report.

OK, just in case let's test again and if the problem is not reproducible we can re-enable it.

Safari WebAudio Panner node seemed to have a problem that the audio can be broken. We decided to force to disable panning audio on Safari. See #4411 for details. The problem seems to have been resolved on Safari end. We can re-enable the panning audio even on Safari.

takahirox · 2022-05-25T20:19:05Z

Made a PR #5460

takahirox · 2022-06-03T21:47:07Z

#5460 has been merged. Closing this issue. Please file a new issue if anyone encounters broken audio problem on Safari again.

Audio/Video noise problem seems to still exist. Let's track that issue in #4322

Safari WebAudio Panner node seemed to have a problem that the audio can be broken. We decided to force to disable panning audio on Safari. See #4411 for details. The problem seems to have been resolved on Safari end. We can re-enable the panning audio even on Safari.

Softvision-RemusDranca added bug needs triage For bugs that have not yet been assigned a fix priority [QA]:Major-issue Label for QA to mark major issues logged labels Jul 14, 2021

Softvision-RemusDranca mentioned this issue Jul 14, 2021

[Safari] The window freezes when uploading video, GIF and audio files #4405

Closed

brianpeiris self-assigned this Jul 15, 2021

takahirox mentioned this issue Jul 19, 2021

[MacOS][Safari] Performance issues occur and a high CPU usage while navigating the room #4423

Open

brianpeiris mentioned this issue Jul 23, 2021

Hubs Cloud - Temporary fix for Safari audio issues #4442

Merged

takahirox self-assigned this Aug 9, 2021

johnshaughnessy added the jira-hubs label Aug 9, 2021

takahirox mentioned this issue Aug 10, 2021

Disable positional audio for Safari as a temporal workaround for its positional audio bug #4503

Closed

brianpeiris removed their assignment Aug 24, 2021

takahirox mentioned this issue Sep 1, 2021

Fix audio mode command. Disable panner nodes in safari. #4594

Merged

takahirox mentioned this issue Sep 27, 2021

Disable echoCancellation for iOS as workaround to reduce crackle noise #4679

Closed

takahirox added safari P1 Address as quickly as possible and removed P3 Backlog labels Sep 29, 2021

This was referenced Sep 29, 2021

Noise occurred when playing audio in iPhone #4322

Open

MacOS Safari: Strange Echo after a few minutes of using #4288

Closed

takahirox mentioned this issue Oct 1, 2021

Audio from videos are crackling on iPad Safari #4701

Closed

takahirox added P3 Backlog and removed P1 Address as quickly as possible labels Oct 4, 2021

takahirox mentioned this issue Oct 4, 2021

About the microphone bug in mobile #4343

Open

Softvision-RemusDranca mentioned this issue Oct 5, 2021

No audio from videos on Safari - IOS 13.x - 14 #3960

Open

takahirox added the ios label Oct 19, 2021

brianpeiris mentioned this issue Nov 16, 2021

Force disable panner audio in safari #4441

Merged

takahirox mentioned this issue Mar 11, 2022

[iOS] Audio got noisy after several minutes #4948

Closed

pigubaoza mentioned this issue May 25, 2022

Audio Pannernode behavior bug on iOS Safari and iOS Chrome #5458

Open

takahirox mentioned this issue May 25, 2022

Re-enable Positional audio on Safari #5460

Merged

takahirox closed this as completed Jun 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Safari] The audio is broken on certain MacOS/Safari versions #4411

[Safari] The audio is broken on certain MacOS/Safari versions #4411

Softvision-RemusDranca commented Jul 14, 2021 •

edited by data-sync-user

Loading

Softvision-RemusDranca commented Jul 14, 2021

Softvision-RemusDranca commented Jul 16, 2021

takahirox commented Jul 16, 2021

brianpeiris commented Jul 18, 2021

brianpeiris commented Jul 18, 2021 •

edited

Loading

takahirox commented Jul 19, 2021 •

edited

Loading

brianpeiris commented Jul 23, 2021

takahirox commented Aug 9, 2021

brianpeiris commented Aug 9, 2021

takahirox commented Aug 10, 2021

takahirox commented Sep 2, 2021

takahirox commented Sep 14, 2021 •

edited

Loading

johnshaughnessy commented Sep 14, 2021

takahirox commented Sep 14, 2021 •

edited

Loading

takahirox commented Sep 16, 2021 •

edited

Loading

takahirox commented Sep 22, 2021 •

edited

Loading

takahirox commented Sep 27, 2021

Softvision-RemusDranca commented Sep 29, 2021

takahirox commented Sep 29, 2021 •

edited

Loading

takahirox commented Sep 29, 2021

vincentfretin commented Sep 30, 2021

Softvision-RemusDranca commented Oct 4, 2021

takahirox commented Oct 4, 2021

takahirox commented Oct 4, 2021

brianpeiris commented Apr 8, 2022

pigubaoza commented May 25, 2022

takahirox commented May 25, 2022

takahirox commented May 25, 2022

takahirox commented Jun 3, 2022

[Safari] The audio is broken on certain MacOS/Safari versions #4411

[Safari] The audio is broken on certain MacOS/Safari versions #4411

Comments

Softvision-RemusDranca commented Jul 14, 2021 • edited by data-sync-user Loading

Softvision-RemusDranca commented Jul 14, 2021

Softvision-RemusDranca commented Jul 16, 2021

takahirox commented Jul 16, 2021

brianpeiris commented Jul 18, 2021

brianpeiris commented Jul 18, 2021 • edited Loading

takahirox commented Jul 19, 2021 • edited Loading

brianpeiris commented Jul 23, 2021

takahirox commented Aug 9, 2021

brianpeiris commented Aug 9, 2021

takahirox commented Aug 10, 2021

takahirox commented Sep 2, 2021

takahirox commented Sep 14, 2021 • edited Loading

johnshaughnessy commented Sep 14, 2021

takahirox commented Sep 14, 2021 • edited Loading

takahirox commented Sep 16, 2021 • edited Loading

takahirox commented Sep 22, 2021 • edited Loading

takahirox commented Sep 27, 2021

Softvision-RemusDranca commented Sep 29, 2021

takahirox commented Sep 29, 2021 • edited Loading

takahirox commented Sep 29, 2021

vincentfretin commented Sep 30, 2021

Softvision-RemusDranca commented Oct 4, 2021

takahirox commented Oct 4, 2021

takahirox commented Oct 4, 2021

brianpeiris commented Apr 8, 2022

pigubaoza commented May 25, 2022

takahirox commented May 25, 2022

takahirox commented May 25, 2022

takahirox commented Jun 3, 2022

Softvision-RemusDranca commented Jul 14, 2021 •

edited by data-sync-user

Loading

brianpeiris commented Jul 18, 2021 •

edited

Loading

takahirox commented Jul 19, 2021 •

edited

Loading

takahirox commented Sep 14, 2021 •

edited

Loading

takahirox commented Sep 14, 2021 •

edited

Loading

takahirox commented Sep 16, 2021 •

edited

Loading

takahirox commented Sep 22, 2021 •

edited

Loading

takahirox commented Sep 29, 2021 •

edited

Loading