萌新提问关于直播中视频avc codec发生变化的问题 #355

alphonsetai · 2018-07-31T08:38:49Z

直播过程中(音视频都有)，一开始时用h264 main的编码进行直播。其中在不断流的情况下切换为h264 high，并发送新的avc confg和视频数据，时间戳写入正常。

通过调试发现推送数据给source buffer时，总是updating，导致音视频数据同时不播放。遇到这样的情况下，该如何查。
web开发萌新。

xqq · 2018-07-31T08:41:09Z

logcat (flv.js & chrome://media-internals) please

alphonsetai · 2018-07-31T08:55:45Z

media-internals

00:00:00 00 origin_url http://192.168.1.111/

00:00:00 00 frame_url http://192.168.1.111/a

00:00:00 00 frame_title AAC_TEST

00:00:00 00 url blob:http://192.168.1.111/a

00:00:00 00 info ChunkDemuxer: buffering by DTS

00:00:00 07 pipeline_state kStarting

00:00:01 127 found_video_stream true

00:00:01 127 video_codec_name h264

00:00:01 127 found_audio_stream true

00:00:01 127 audio_codec_name aac

00:00:01 130 audio_dds false

00:00:01 130 audio_decoder FFmpegAudioDecoder

00:00:01 130 info Selected FFmpegAudioDecoder for audio decoding, config: codec: aac bytes_per_channel: 2 channel_layout: 3 channels: 2 samples_per_second: 44100 sample_format: 2 bytes_per_frame: 4 seek_preroll: 0ms codec_delay: 0 has extra data? false encryption scheme: Unencrypted discard decoder delay? false

00:00:01 130 debug Video rendering in low delay mode.

00:00:01 217 video_dds false

00:00:01 217 video_decoder GpuVideoDecoder

00:00:01 217 info Selected GpuVideoDecoder for video decoding, config: codec: h264 format: 1 profile: h264 high coded size: [320,240] visible rect: [0,0,320,240] natural size: [320,240] has extra data? false encryption scheme: Unencrypted rotation: 0°

00:00:01 217 pipeline_state kPlaying

00:00:01 217 warning Failed to reconcile encoded audio times with decoded output.

00:00:01 218 audio_buffering_state BUFFERING_HAVE_ENOUGH

00:00:01 243 video_buffering_state BUFFERING_HAVE_ENOUGH

00:00:01 243 height 240

00:00:01 243 width 320

00:00:01 243 for_suspended_start false

00:00:01 243 pipeline_buffering_state BUFFERING_HAVE_ENOUGH

00:00:01 243 info Effective playback rate changed from 0 to 1

00:00:01 243 event PLAY

00:00:01 127 duration unknown

00:00:11 496 video_buffering_state BUFFERING_HAVE_NOTHING

00:00:14 496 for_suspended_start false

00:00:14 496 pipeline_buffering_state BUFFERING_HAVE_NOTHING

00:00:00 00 info ChunkDemuxer: buffering by DTS

00:00:00 00 pipeline_state kStarting

00:00:00 69 found_video_stream true

00:00:00 69 video_codec_name h264

00:00:00 69 found_audio_stream true

00:00:00 69 audio_codec_name aac

00:00:00 72 audio_dds false

00:00:00 72 audio_decoder FFmpegAudioDecoder

00:00:00 72 info Selected FFmpegAudioDecoder for audio decoding, config: codec: aac bytes_per_channel: 2 channel_layout: 3 channels: 2 samples_per_second: 44100 sample_format: 2 bytes_per_frame: 4 seek_preroll: 0ms codec_delay: 0 has extra data? false encryption scheme: Unencrypted discard decoder delay? false

00:00:00 72 debug Video rendering in low delay mode.

00:00:00 171 video_dds false

00:00:00 171 video_decoder GpuVideoDecoder

00:00:00 171 info Selected GpuVideoDecoder for video decoding, config: codec: h264 format: 1 profile: h264 high coded size: [320,240] visible rect: [0,0,320,240] natural size: [320,240] has extra data? false encryption scheme: Unencrypted rotation: 0°

00:00:00 171 pipeline_state kPlaying

00:00:00 172 audio_buffering_state BUFFERING_HAVE_ENOUGH

00:00:00 196 video_buffering_state BUFFERING_HAVE_ENOUGH

00:00:00 196 height 240

00:00:00 196 width 320

00:00:00 196 for_suspended_start false

00:00:00 196 pipeline_buffering_state BUFFERING_HAVE_ENOUGH

00:00:00 196 info Effective playback rate changed from 0 to 1

00:00:00 196 event PLAY

00:00:00 69 duration unknown

00:00:17 801 video_buffering_state BUFFERING_HAVE_NOTHING

00:00:20 802 for_suspended_start false

00:00:20 802 pipeline_buffering_state BUFFERING_HAVE_NOTHING

flv.js - log

[download.url] > http://192.168.1.111/flv/1.flv
[FLVDemuxer] > Parsed AudioSpecificConfig
[MSEController] > Received Initialization Segment, mimeType: audio/mp4;codecs=mp4a.40.5

[download.url] > http://192.168.1.111/flv/2.flv
[FLVDemuxer] > Parsed AVCDecoderConfigurationRecord
[FLVDemuxer] > Parsed AudioSpecificConfig
[MSEController] > Received Initialization Segment, mimeType: video/mp4;codecs=avc1.64000b
[MSEController] > Received Initialization Segment, mimeType: audio/mp4;codecs=mp4a.40.5

[download.url] > http://192.168.1.111/flv/3.flv
[FLVDemuxer] > Parsed AVCDecoderConfigurationRecord
[FLVDemuxer] > Parsed AudioSpecificConfig
[MSEController] > Received Initialization Segment, mimeType: video/mp4;codecs=avc1.64000b
[MSEController] > Received Initialization Segment, mimeType: audio/mp4;codecs=mp4a.40.5

[download.url] > http://192.168.1.111/flv/4.flv
[FLVDemuxer] > Parsed AVCDecoderConfigurationRecord
[FLVDemuxer] > Parsed AudioSpecificConfig
[FLVDemuxer] > Parsed AVCDecoderConfigurationRecord
[MSEController] > Received Initialization Segment, mimeType: video/mp4;codecs=avc1.4d4016
[MSEController] > Notice: video mimeType changed, origin: video/mp4;codecs=avc1.64000b, target: video/mp4;codecs=avc1.4d4016
[MSEController] > Received Initialization Segment, mimeType: audio/mp4;codecs=mp4a.40.5
[MSEController] > Received Initialization Segment, mimeType: video/mp4;codecs=avc1.4d4016

[download.url] > http://192.168.1.111/flv/5.flv
[FLVDemuxer] > Parsed AVCDecoderConfigurationRecord
[FLVDemuxer] > Parsed AudioSpecificConfig
[MSEController] > Received Initialization Segment, mimeType: video/mp4;codecs=avc1.4d4016
[MSEController] > Received Initialization Segment, mimeType: audio/mp4;codecs=mp4a.40.5

alphonsetai · 2018-07-31T08:56:44Z

第4个虚拟flv的时候，视频流发生了变化。

alphonsetai · 2018-07-31T09:03:48Z

@xqq 情况和我说的可能有点不同，因为我把测试的结果混淆了，我测试过从main -> high或者从high -> main都会有问题。

chyingp · 2019-04-15T07:35:26Z

codec变化后，音视频播放同时卡住似乎是在1.4.0引入的，具体是这次提交：MP4Remuxer: Not process if only one sample existed to avoid guessing …

对比：把 _remuxAudio、_reduxVideo 方法的这段判断去掉，就可以正常播放。（得研究下为什么加这段判断，去掉会有什么问题。）

        if (!samples || samples.length === 0) {
            return;
        }

@xqq @alphonsetai

alphonsetai · 2019-04-17T03:33:57Z

codec变化后，音视频播放同时卡住似乎是在1.4.0引入的，具体是这次提交：MP4Remuxer: Not process if only one sample existed to avoid guessing …

对比：把 _remuxAudio、_reduxVideo 方法的这段判断去掉，就可以正常播放。（得研究下为什么加这段判断，去掉会有什么问题。）
        if (!samples || samples.length === 0) {
            return;
        }
@xqq @alphonsetai

老铁给力，后来我也发现似乎和这附近的代码有些关系。不过非web开发，看不懂怎么改。

chyingp · 2020-01-09T08:36:25Z

@alphonsetai 找到具体原因了，当codec发生变化时，flv.js 会生成新的 init segment，然后 append 到 mse里。

因为 mp4-remuxer 中 _videoStashedLastSample、_audioStashedLastSample 的存在，导致这个切换过程大概率会失败，因为新的 init segment 已经 append 到mse里，但是旧的 video sample 或audio sample 还在缓存里。

此时会出现『用新的codec 去解新的 sample』，因此会失败。

alphonsetai · 2020-01-09T09:11:14Z

@alphonsetai 找到具体原因了，当codec发生变化时，flv.js 会生成新的 init segment，然后 append 到 mse里。

因为 mp4-remuxer 中 _videoStashedLastSample、_audioStashedLastSample 的存在，导致这个切换过程大概率会失败，因为新的 init segment 已经 append 到mse里，但是旧的 video sample 或audio sample 还在缓存里。

此时会出现『用新的codec 去解新的 sample』，因此会失败。

一年多了，兄弟还记得这个事情定位，点个赞啊。因为我自己是非web开发人员，当时再看media-internals的日志时，对比了hls.js的情况，确实没看到ffmpeg re-init。但又无法确认，确实不晓得该不该说。兄弟厉害，加个联系方式，以后多多交流啊

chyingp · 2020-01-10T14:35:23Z

试了下，可以做如下改动： @alphonsetai
1、codec 切换时，将 videoTrack、audioTrack 里的 sample 都送去 remux （包括 stashedSample） -- 处理卡死问题
2、videoSample、audioSample 一起 remux时，检测 audioSample、videoSample 之间的 dts 差，超过一定的值时，丢弃一些sample -- 处理音画不同步问题

更优雅合理的处理，静候 @xqq 大佬发版 😄

thinkboy · 2020-01-15T13:41:44Z

@chyingp 刚遇到相同问题了。大佬有没有简单粗暴的修改方式，fork一份代码修改看一下？

另外我还找到了另一个看上去解决问题但一直没合并到master的修复（我觉得估计作者已经放弃该项目了）：#136

xqq · 2020-01-16T07:35:37Z

过阵子有空了再改

SuperJolly · 2020-03-12T08:15:01Z

codec变化后，音视频播放同时卡住似乎是在1.4.0引入的，具体是这次提交：MP4Remuxer: Not process if only one sample existed to avoid guessing …

对比：把 _remuxAudio、_reduxVideo 方法的这段判断去掉，就可以正常播放。（得研究下为什么加这段判断，去掉会有什么问题。）
        if (!samples || samples.length === 0) {
            return;
        }
@xqq @alphonsetai

if (samples.length === 1 && !force) {
    // If [sample count in current batch] === 1 && (force != true)
    // Ignore and keep in demuxer's queue
    return;
}

commit中貌似应该是这段新增导致的

bilibili#355 (comment)

SuperJolly · 2020-03-13T05:29:54Z

@alphonsetai 可以试试这个PR: #551 ，我这边验证了下不同的场景，没有复现之前的问题。

alphonsetai · 2020-03-15T10:01:15Z

@alphonsetai 可以试试这个PR: #551 ，我这边验证了下不同的场景，没有复现之前的问题。

感谢兄弟的定位，帮助很大，不过我这边由于换工作的关系，暂时没有再跟这个问题了。由于分工原因，我对前端了解少的离奇，看了一下兄弟修改代码，似乎符合当时定位和后续大家讨论的认知范围。待后续我造一个环境，我再看看。

bilibili#355

bilibili#355 (comment)

SuperJolly · 2020-05-21T03:32:12Z

@xqq 大佬最近有空改下吗？😄

wy513222130 · 2021-02-19T03:25:56Z

@chyingp 刚遇到相同问题了。大佬有没有简单粗暴的修改方式，fork一份代码修改看一下？

另外我还找到了另一个看上去解决问题但一直没合并到master的修复（我觉得估计作者已经放弃该项目了）：#136

哈哈，就是我，小白前端，本身对音视频完全不懂的，debug发现音频的duration会不断变长。我的这个方法确实不能完全解决问题。只能在网络环境比较好的情况下有用。现在很多直播用rtc推流，还是会出现音画不同步的问题。修复好音画不同步，还是要通过丢弃一部分帧或者插入一些音频帧来解决。最近发现safari（14版本的）不能播放混合流。又是头疼的事。

wechat361 · 2023-06-14T09:22:02Z

试了下，可以做如下改动： @alphonsetai 1、codec 切换时，将 videoTrack、audioTrack 里的 sample 都送去 remux （包括 stashedSample） -- 处理卡死问题 2、videoSample、audioSample 一起 remux时，检测 audioSample、videoSample 之间的 dts 差，超过一定的值时，丢弃一些sample -- 处理音画不同步问题

更优雅合理的处理，静候 @xqq 大佬发版 😄

您好，这一块修改的内容，可以提供下相关代码给参考下吗？对音视频一点都不懂，不知道该如何改起

chyingp mentioned this issue Jan 9, 2020

播放直播流过程中卡住且不能恢复 #398

Open

xqq self-assigned this Jan 9, 2020

SuperJolly added a commit to SuperJolly/flv.js that referenced this issue Mar 12, 2020

--bug=解决连麦混流场景下codec变化导致的视频卡死

50930fa

bilibili#355 (comment)

SuperJolly added a commit to SuperJolly/flv.js that referenced this issue Mar 13, 2020

--bug=解决连麦混流场景下codec变化导致的视频卡死

00b7404

bilibili#355 (comment)

SuperJolly mentioned this issue Mar 13, 2020

--bug=解决连麦混流场景下codec变化导致的视频/音频卡死 #551

Closed

SuperJolly added a commit to SuperJolly/flv.js that referenced this issue Apr 3, 2020

--bug=萌新提问关于直播中视频avc codec发生变化的问题

d283c76

bilibili#355

SuperJolly added a commit to SuperJolly/flv.js that referenced this issue Apr 3, 2020

--bug=解决连麦混流场景下codec变化导致的视频卡死

d8f7780

bilibili#355 (comment)

SuperJolly mentioned this issue Apr 3, 2020

--bug=解决连麦混流场景下codec变化导致的视频卡死 #558

Open

datagutt pushed a commit to GuacLive/flv.js that referenced this issue Apr 28, 2020

--bug=解决连麦混流场景下codec变化导致的视频卡死

e17a640

bilibili#355 (comment)

shady-xia mentioned this issue Mar 19, 2024

AVCDecoderConfigurationRecord 变化 shady-xia/flvExtend#28

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

萌新提问关于直播中视频avc codec发生变化的问题 #355

萌新提问关于直播中视频avc codec发生变化的问题 #355

alphonsetai commented Jul 31, 2018

xqq commented Jul 31, 2018

alphonsetai commented Jul 31, 2018 •

edited

Loading

alphonsetai commented Jul 31, 2018

alphonsetai commented Jul 31, 2018

chyingp commented Apr 15, 2019 •

edited

Loading

alphonsetai commented Apr 17, 2019

chyingp commented Jan 9, 2020

alphonsetai commented Jan 9, 2020

chyingp commented Jan 10, 2020

thinkboy commented Jan 15, 2020

xqq commented Jan 16, 2020

SuperJolly commented Mar 12, 2020

SuperJolly commented Mar 13, 2020 •

edited

Loading

alphonsetai commented Mar 15, 2020

SuperJolly commented May 21, 2020

wy513222130 commented Feb 19, 2021

wechat361 commented Jun 14, 2023

萌新提问关于直播中视频avc codec发生变化的问题 #355

萌新提问关于直播中视频avc codec发生变化的问题 #355

Comments

alphonsetai commented Jul 31, 2018

xqq commented Jul 31, 2018

alphonsetai commented Jul 31, 2018 • edited Loading

alphonsetai commented Jul 31, 2018

alphonsetai commented Jul 31, 2018

chyingp commented Apr 15, 2019 • edited Loading

alphonsetai commented Apr 17, 2019

chyingp commented Jan 9, 2020

alphonsetai commented Jan 9, 2020

chyingp commented Jan 10, 2020

thinkboy commented Jan 15, 2020

xqq commented Jan 16, 2020

SuperJolly commented Mar 12, 2020

SuperJolly commented Mar 13, 2020 • edited Loading

alphonsetai commented Mar 15, 2020

SuperJolly commented May 21, 2020

wy513222130 commented Feb 19, 2021

wechat361 commented Jun 14, 2023

alphonsetai commented Jul 31, 2018 •

edited

Loading

chyingp commented Apr 15, 2019 •

edited

Loading

SuperJolly commented Mar 13, 2020 •

edited

Loading