Does hls.js support extracting SEI frame from videos?

[dongrixinyu]

Is your feature request related to a problem? Please describe.

No

Describe the solution you'd like

I really appreciated that the fabulous hls.js favored my daily frontend work.

However I still have a problem concerning the SEI frame data from videos encoded in h264

the situation is as follows:

My video is encoded in h264 with SEI data after every I frame. and I wanna extract SEI info in the fontend browser and continue to parse this data by myself.
So I am confused that if hls.js provide such api or sdk functions for me to extract every SEI.

really look forward to your reply.

Thank you!

Additional context

No response

[dylanjha]

@dongrixinyu would this event help?

• Hls.Events.FRAG_PARSING_USERDATA - fired when parsing sei text is completed
  data: { id : demuxer id, frag: fragment object, samples : [ sei samples pes ] }

Found it in the list of runtime events

[dongrixinyu]

Thank you for your answer. and I ll reply after i tried this hook

[dongrixinyu]

@dongrixinyu would this event help?

• Hls.Events.FRAG_PARSING_USERDATA - fired when parsing sei text is completed
  data: { id : demuxer id, frag: fragment object, samples : [ sei samples pes ] }

Found it in the list of runtime events

I tried this hook api via code below:

    <script>
        var video = document.getElementById('video');
        var hls = new Hls();
        // bind them together
        hls.attachMedia(video);
        // MEDIA_ATTACHED event is fired by hls object once MediaSource is ready
        hls.on(Hls.Events.MEDIA_ATTACHED, function () {
            // hls.loadSource('https://live.ig-dev.uavcmlc.com:443/live/test_rtsp_stream.m3u8?token=E5xZiWsNEQ6D');
            hls.loadSource('http://192.168.35.51:26789/test.m3u8');
            hls.on(Hls.Events.MANIFEST_PARSED, function (event, data) {
                console.log(
                    'manifest loaded, found ' + data.levels.length + ' quality level'
                );
            });
            hls.on(Hls.Events.FRAG_PARSING_USERDATA, function (event, data) {
                console.log('event =====> ', event, 'data =====> ', data)
            })
        });
    </script>

and my source is a m3u8 file containing several mpegs-ts files with SEI info.
when I get the ts file, the hls.js indeed get the log as below:

then what I am confused about is:

• when hls.js get a mpegs-ts file, this hook can get all SEI info at once? Can we get the SEI info one frame after another when I play the video file?
• in the picture above, userData is the SEI info,right?

Thanks again for your guide and help!

[dylanjha]

@dongrixinyu I'm not sure the answer to your 2 questions.

Did you try this code above with the Hls.Events.FRAG_PARSING_USERDATA listener? What were the results?

[antonm76]

@robwalch I set SEI user data in my stream and then I able to read them with FRAG_PARSING_USERDATA events but it is only possible for HLS streams with MPEGTS segments.
When I try to use FMP4 (with ffmpeg's hls muxer) or M4S (which are generated by the ffmpeg's dash muxer) user data samples are always emty.

[antonm76]

@robwalch I use v.1.1.5