Audio transmission issue during stream #1208

mkrn · 2023-05-26T10:02:46Z

Describe the bug

We've had at least 3 instances of sound changing to be really bad during the stream:

Local device recording is not affected, streaming only!


rtmpStream.audioSettings = AudioCodecSettings(
            bitRate: 128 * 1000
          )
// (not adjusting audio bitrate during the stream) 

func startAVSession() {
        let session = AVAudioSession.sharedInstance()
        do { 
            if #available(iOS 10.0, *) {
                try session.setCategory(.playAndRecord, mode: .videoRecording, options: [.defaultToSpeaker, .allowBluetooth])
                
            } else {
                session.perform(NSSelectorFromString("setCategory:withOptions:error:"), with: AVAudioSession.Category.record, with: [
                    AVAudioSession.CategoryOptions.allowBluetooth,
                    AVAudioSession.CategoryOptions.defaultToSpeaker]
                ) 
                try session.setMode(.videoRecording) 
            }
            if #available(iOS 14.5, *) {
                try session.setPrefersNoInterruptionsFromSystemAlerts(true);           
            }

            do {
                try session.setAllowHapticsAndSystemSoundsDuringRecording(false)
                try session.setActive(true)
            } catch {
                print("Error: \(error.localizedDescription)")
            }
              
        } catch {
            NSLog("Error start AV session")
        }
    }

To Reproduce

Unfortunately we are not sure.. it seems like someone touches the device in both examples.

Expected behavior

The audio should be fine all along

Version

1.4.5
The problem appeared in recent updates

Smartphone info.

Devices:
Apple iPad Pro 11" 3rd Gen (Wi-Fi/Cellular)
iOS 16.5

Apple iPad Pro 11 inch 2nd Gen
iPadOS 16.4.1

Apple iPhone 12 Pro Max
ios 16.4.1

Additional context

No response

Screenshots

No response

Relevant log output

No response

The text was updated successfully, but these errors were encountered:

shogo4405 · 2023-05-26T11:19:24Z

It sounds like the sampling rate changed in the middle of the audio. Could you please let me know the previous version of HaishinKit that you were using?

I understand that there may be cases where the sampling rate changes during the audio due to notifications or opening/closing of the control panel during the live stream. I downloaded your video. Please delete if needed.

mkrn · 2023-05-26T13:14:10Z

Thank you. Yes, seems to be happening after notification panel opening.

I debugged and got the issue seemingly fixed by changing how I create AudioSession:

try session.setCategory(.playAndRecord, mode: .videoRecording, options: [.defaultToSpeaker, .allowBluetooth])
=>
try session.setCategory(.playAndRecord, mode: .voiceChat, options: [.defaultToSpeaker, .allowBluetooth])

mkrn · 2023-06-23T15:02:08Z

Unfortunately setting it to .voiceChat breaks external microphones functionality. Is there any way to pause recording while app is in notification center...

shogo4405 · 2023-06-24T02:10:53Z

I see. It seems like a good idea to maintain a certain sampling rate on the framework side. Implementing a resampler will take some time, but I will give it a try.

I would like to test it, so please let me know the product name of the external microphone you are using.

mkrn · 2023-06-26T10:23:19Z

Interestingly, inline audio microphones (for example, any mic with TRRS cable and lightning adapter) works even in .voiceChat mode.
But, MFi certified microphones don't (they work for videoRecording but internal microphone is used in .voiceChat mode)

And the problem with notification center occurs with or without microphones.

I think the reason is not sampling rate, but because in Notification center video is not captured, there's a sync issue between video and audio. In recording I hear audio like it's fast-forwarded after notification center swipe up.

I'm using didBecomeActiveNotification / willResignActiveNotification. And setting self.rtmpStream.paused = true; / self.rtmpStream.paused = false;

shogo4405 · 2023-07-16T08:46:46Z

I'm currently testing it, but in my environment, the capture stops even with .voiceChat(iPhone 13 Pro max iOS16.5.1). Does it work on your end?
I have noticed a bug in the process implemented in version 1.5.x for synchronizing video and audio. It is possible that it has been improved in the main branch.

1.5.4 (main branch)

2023-16-07 17:13:05.169 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 23.219954688102007
2023-16-07 17:13:05.169 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 23.219954455271363
2023-16-07 17:13:05.169 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 23.219954688102007
2023-16-07 17:13:05.169 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 23.219954688102007
2023-16-07 17:13:05.170 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 23.219954688102007
2023-16-07 17:13:05.179 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 44.5578231010586
2023-16-07 17:13:05.200 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 23.219954688102007

——

1.5.3

2023-16-07 17:14:01.922 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 23.219954455271363
2023-16-07 17:14:01.922 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 23.219954688102007
2023-16-07 17:14:01.922 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 23.219954688102007
2023-16-07 17:14:01.922 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 23.219954688102007
2023-16-07 17:14:01.933 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 1798.7755101639777 // huge timestamp diff
2023-16-07 17:14:01.954 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 21.6780046466738
2023-16-07 17:14:01.975 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 13.922902289777994
2023-16-07 17:14:01.997 [Info] [com.haishinkit.HaishinKit] [RTMPMuxer.swift:43] audioCodec(_:didOutput:presentationTimeStamp:) > 23.219954688102007

mkrn · 2023-07-27T21:53:38Z

Currently getting this issue on a lot of streams:
https://eventlive-misc.nyc3.cdn.digitaloceanspaces.com/issuesound.mp4
It seems to appear during the stream, maybe due to connection issues,
Not seem to be connected with audio session configuration anymore..

Tested your #1236 but it makes streaming almost impossible after reconnect on 4g, or non-ideal internet..

mkrn · 2023-07-27T22:05:51Z

By the way, local recording is fine, it's a transmission issue clearly
And it seems to happen regardless with external mic or built-in after some time.

shogo4405 · 2023-07-27T22:11:25Z

Does this mean that the issue is not reproducible when opening and closing the control panel?

mkrn · 2023-07-27T22:14:46Z

No it seems like not control panel related, we fixed control panel by adding delegate methods to sessionWasInterrupted / sessionInterruptionEnded and we do stopPublish / startPublish in those...
Any ideas?

shogo4405 · 2023-07-27T22:23:50Z

I checked the mp4, it appears that the audio codec is set to stereo. I believe the standard microphone records in mono, so is this using an external microphone?

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'issuesound.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: isommp41mp42
    creation_time   : 2023-07-27T21:50:00.000000Z
  Duration: 00:00:56.02, start: 0.000000, bitrate: 3344 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1280x720, 3145 kb/s, 30 fps, 30 tbr, 16k tbn (default)
    Metadata:
      creation_time   : 2023-07-27T21:50:00.000000Z
      handler_name    : Core Media Video
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 136 kb/s (default)
    Metadata:
      creation_time   : 2023-07-27T21:50:00.000000Z
      handler_name    : Core Media Audio
      vendor_id       : [0][0][0][0]
 181.97 A-V: -0.005 fd=   5 aq=    0KB vq=    0KB sq=    0B f=0/0

shogo4405 · 2023-07-27T22:30:26Z

we do stopPublish / startPublish in those...

I'm considering the possibility that after reconnection, the Audio header might not be sent properly. I will test and verify this.

mkrn · 2023-07-27T22:38:45Z

This is a video file as-transmitted, not recoded FYI:
https://sfo2.digitaloceanspaces.com/eventliveflv/YVNVRSNH-1690480861.flv
Strangely I can't reproduce on my phone even though this is the same model..

mkrn · 2023-07-27T22:44:25Z

Happened on a few different phones/tablets 16.6, 16.5 and even 15.6.1

shogo4405 · 2023-07-27T22:52:13Z

Raw FLV can be helpful for analysis. Thank you for providing it.

Input #0, flv, from 'YVNVRSNH-1690480861.flv':
  Duration: 00:01:00.86, start: 0.064000, bitrate: 2771 kb/s
  Stream #0:0: Audio: aac (LC), 48000 Hz, stereo, fltp
  Stream #0:1: Video: h264 (High), yuv420p(progressive), 1280x720, 30 fps, 30 tbr, 1k tbn

mkrn · 2023-07-27T23:00:54Z

I have sent another one directly on discord where issue appears mid-stream. Thank you

mkrn · 2023-07-28T13:14:44Z

packets_timestamps.csv
Noticed some transmissions have a huge difference between audio and video pts timestamps. Is this related? Fixed in 57cd7db ?
If not I can try to reproduce and open another issue. This is 1.5.2

shogo4405 · 2023-07-28T23:20:45Z

How was this timestamp dumped? I can't determine its origin because the output varies depending on the method used for dumping.

I tried opening and closing the control panel in the sample (UIKit version) to support stereo output(f7e6c41). As a result, I was able to reproduce similar noise to the FLV mentioned in #1208 (comment).

shogo4405 · 2023-07-29T02:26:29Z

I found an error in the implementation of the RingBuffer for stereo format, which was causing the audio waveform to be corrupted. I have already fixed it in this pull request: #1255.

shogo4405 · 2023-07-29T02:26:58Z

The issue does not occur when the number of samples captured from the microphone is exactly 1024. However, the problem arises when the number of samples is less than 1024 or greater than 1024. In most cases, the microphone captures 1024 samples, but there are instances where it can be less than 1024 due to certain circumstances. Thank you.

mkrn changed the title ~~Audio~~ Audio transmission issue during stream May 26, 2023

shogo4405 mentioned this issue Jul 15, 2023

Reviewed the process for filling the gap. #1236

Merged

3 tasks

shogo4405 mentioned this issue Jul 29, 2023

fixed break audio with stereo format. #1255

Merged

1 task

shogo4405 closed this as completed in #1255 Jul 29, 2023

CedricEugeni mentioned this issue Sep 8, 2023

Audio issue while transmitting. WebRTC. 4 channels #1289

Closed

github-actions bot locked as resolved and limited conversation to collaborators Apr 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Audio transmission issue during stream #1208

Audio transmission issue during stream #1208

mkrn commented May 26, 2023 •

edited

Loading

shogo4405 commented May 26, 2023

mkrn commented May 26, 2023

mkrn commented Jun 23, 2023

shogo4405 commented Jun 24, 2023

mkrn commented Jun 26, 2023

shogo4405 commented Jul 16, 2023 •

edited

Loading

mkrn commented Jul 27, 2023

mkrn commented Jul 27, 2023

shogo4405 commented Jul 27, 2023

mkrn commented Jul 27, 2023

shogo4405 commented Jul 27, 2023

shogo4405 commented Jul 27, 2023

mkrn commented Jul 27, 2023 •

edited

Loading

mkrn commented Jul 27, 2023

shogo4405 commented Jul 27, 2023

mkrn commented Jul 27, 2023

mkrn commented Jul 28, 2023

shogo4405 commented Jul 28, 2023 •

edited

Loading

shogo4405 commented Jul 29, 2023

shogo4405 commented Jul 29, 2023

Audio transmission issue during stream #1208

Audio transmission issue during stream #1208

Comments

mkrn commented May 26, 2023 • edited Loading

Describe the bug

To Reproduce

Expected behavior

Version

Smartphone info.

Additional context

Screenshots

Relevant log output

shogo4405 commented May 26, 2023

mkrn commented May 26, 2023

mkrn commented Jun 23, 2023

shogo4405 commented Jun 24, 2023

mkrn commented Jun 26, 2023

shogo4405 commented Jul 16, 2023 • edited Loading

mkrn commented Jul 27, 2023

mkrn commented Jul 27, 2023

shogo4405 commented Jul 27, 2023

mkrn commented Jul 27, 2023

shogo4405 commented Jul 27, 2023

shogo4405 commented Jul 27, 2023

mkrn commented Jul 27, 2023 • edited Loading

mkrn commented Jul 27, 2023

shogo4405 commented Jul 27, 2023

mkrn commented Jul 27, 2023

mkrn commented Jul 28, 2023

shogo4405 commented Jul 28, 2023 • edited Loading

shogo4405 commented Jul 29, 2023

shogo4405 commented Jul 29, 2023

mkrn commented May 26, 2023 •

edited

Loading

shogo4405 commented Jul 16, 2023 •

edited

Loading

mkrn commented Jul 27, 2023 •

edited

Loading

shogo4405 commented Jul 28, 2023 •

edited

Loading