Transmuxing a subtitles stream

[okycelt]

Issue and steps to reproduce the problem

We're using UDT together with Widevine iOS CDM to play DASH on iOS devices. Video and audio streams are transmuxed correctly, but we have a problem with subtitle streams (tried web-vtt and srt). We've extended Stream, MpdParser and PlaylistBuilder classes in the Widevine iOS example project to support textual streams, but Udt_ParseDash funciton returns kDashToHlsStatus_BadDashContents when trying to parse a textual stream. Is the transmuxer able to transmux a textual stream at all?

What is the expected result?

HLS with subtitles.

What happens instead?

Udt_ParseDash() returns kDashToHlsStatus_BadDashContents.

Additional info

Generated master playlist

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA:URI="0.m3u8",TYPE=AUDIO,GROUP-ID="audio",NAME="audio0",DEFAULT=NO,AUTOSELECT=YES
#EXT-X-MEDIA:URI="1.m3u8",TYPE=SUBTITLES,GROUP-ID="subtitles",NAME="subtitles1",DEFAULT=NO,AUTOSELECT=YES
#EXT-X-STREAM-INF:BANDWIDTH=400000,CODECS="avc1.4d401e",RESOLUTION=720x480,AUDIO="audio",SUBTITLES="subtitles"
2.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=544775,CODECS="avc1.4d401e",RESOLUTION=720x480,AUDIO="audio",SUBTITLES="subtitles"
3.m3u8

Log

2018-08-02 14:53:13:687 [28275:1839693] Downloading data from https://../ct24-9_subtitle_init_2.mp4.
2018-08-02 14:53:13.709221+0800 [28275:1839689] Downloading data from https://../ct24-9_subtitle_init_2.mp4.
2018-08-02 14:53:14.047118+0800 [28275:1839693] parseInitializationData
2018-08-02 14:53:14.047410+0800 [28275:1839693] Udt_ParseDash(..., DASHMediaType: 5, ...)
2018-08-02 14:53:14.047861+0800 [28275:1839693] {
	'Message':'Unknown Media Type',
	'File':'/var/lib/jenkins/workspace/iOS_CDM_Tests/udt/library/dash_to_hls_api.cc',
	'Line':225,
	'Reason':'Missing Audio/Video Box',
	'Extra':''
}
2018-08-02 14:53:14.048015+0800 [28275:1839693] status: 3, status == kDashToHlsStatus_BadDashContents: 1
2018-08-02 14:53:14:048 [28275:1839693] Error:
     failed to parse dash
2018-08-02 14:53:14.048344+0800 [28275:1839689] Error:
     failed to parse dash
2018-08-02 14:53:14.048608+0800 [28275:1839693] 0-Box<ftyp:32 Unknown> 
32-Box<moov:669 Movie> 
   32-Box<mvhd:108 Movie Header> Version 0 flags 0 Creation Time: 3616030889 Modification Time: 3616030889 Timescale: 1000 Duration: 0
   140-Box<meta:137 Unknown> 
   277-Box<trak:376 Track> 
      277-Box<tkhd:92 Unknown> 
      369-Box<mdia:276 Media> 
         369-Box<mdhd:32 Movie Header> Version 0 flags 0 Creation Time: 3616030889 Modification Time: 3616030889 Timescale: 1000 Duration: 0 Language code: 12
         401-Box<hdlr:44 Unknown> 
         445-Box<minf:192 MediaInformation> 
            445-Box<dinf:36 DataInformation> 
               445-Box<dref:28 Unknown> 
            481-Box<stbl:136 SampleTable> 
               481-Box<stsd:60 SampleDescription> Version 0 flags 0 descriptions: 1
                  481-Box<wvtt:44 Unknown> 
               541-Box<stts:16 Unknown> 
               557-Box<stsc:16 Unknown> 
               573-Box<stsz:20 SampleTable> Version 0 flags 0 samples: 0
               593-Box<stco:16 Unknown> 
            617-Box<sthd:12 Unknown> 
   653-Box<mvex:40 MovieExtend> 
      653-Box<trex:32 TrackExtends> Version 0 flags 0 Track ID:1 Default Sample Description Index:1 Default Duration:0 Default Size:0 Default Flags:0
2018-08-02 14:53:14:049 [28275:1839693] Error:
     failed to initialize stream from https://../ct24-9_subtitle_init_2.mp4
2018-08-02 14:53:14.049283+0800 [28275:1839735] Error:
     failed to initialize stream from https://../ct24-9_subtitle_init_2.mp4

[SeawardT]

I assume in the short term, skipping subtitle streams works fine? And simply handling the text stream at the player level works?

[okycelt]

I assume in the short term, skipping subtitle streams works fine?

Yes, skipping subtitle streams works fine. But in some countries, subtitles are required by law so in the long term, skipping them is not an option.

And simply handling the text stream at the player level works?

Yes, we've managed to feed the player subtitles from an HLS stream and the player was displaying them correctly.

The problem is that DASH has subtitles in .m4s format whereas HLS wants subtitles as a plain text, eg. .webvtt. So if I'm not mistaken just bypassing the transmuxer and feeding the .m4s to the player from the local server is not an option (tried that already as well).

[SeawardT]

For clarity, I was not referring to ignoring the subtitles, I suggesting handling the subtitles out of band. Basically, doing a convert from whatever format you have coming in and translating the text format to be displayed as you need. I know not the easiest solution, but was curious if that is how you were already doing it.

…

On Tue, Aug 21, 2018 at 1:13 AM Ondřej Kycelt ***@***.***> wrote: I assume in the short term, skipping subtitle streams works fine? Yes, skipping subtitle streams works fine. But in some countries, subtitles are required by law so in the long term, skipping them is not an option. And simply handling the text stream at the player level works? Yes, we've managed to feed the player subtitles from an HLS stream and the player was displaying them correctly. The problem is that DASH has subtitles in .m4s format whereas HLS wants subtitles as a plain text, eg. .webvtt. So if I'm not mistaken just bypassing the transmuxer and feeding the .m4s to the player from the local server is not an option (tried that already as well). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AI03s5KC13GOVGXOrCFRlytcdsPKnLYwks5uS8E0gaJpZM4Vr0BK> .

[okycelt]

For clarity, I was not referring to ignoring the subtitles, I suggesting
handling the subtitles out of band. Basically, doing a convert from
whatever format you have coming in and translating the text format to be
displayed as you need. I know not the easiest solution, but was curious if
that is how you were already doing it.

Sorry, I misunderstood. We've been trying to avoid translating the .m4s to .webvtt so far, but it more and more seems to me like our best shot now. Do you have any other ideas?

[SeawardT]

For the short term, that would be the quickest solution. Supporting Subs will be a low priority issue. We are working on a longer term solution however, but dont have a firm date on when that would be ready and subtitles will be supported.

…

On Tue, Aug 21, 2018 at 11:27 PM Ondřej Kycelt ***@***.***> wrote: For clarity, I was not referring to ignoring the subtitles, I suggesting handling the subtitles out of band. Basically, doing a convert from whatever format you have coming in and translating the text format to be displayed as you need. I know not the easiest solution, but was curious if that is how you were already doing it. Sorry, I misunderstood. We've been trying to avoid translating the .m4s to .webvtt so far, but it more and more seems to me like our best shot now. Do you have any other ideas? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AI03s2DU2AvPQ6tXzjhzXCeBKDsReWHMks5uTPmygaJpZM4Vr0BK> .

[Streebor]

For the short term, that would be the quickest solution. Supporting Subs will be a low priority issue. We are working on a longer term solution however, but dont have a firm date on when that would be ready and subtitles will be supported.
…
On Tue, Aug 21, 2018 at 11:27 PM Ondřej Kycelt @.***> wrote: For clarity, I was not referring to ignoring the subtitles, I suggesting handling the subtitles out of band. Basically, doing a convert from whatever format you have coming in and translating the text format to be displayed as you need. I know not the easiest solution, but was curious if that is how you were already doing it. Sorry, I misunderstood. We've been trying to avoid translating the .m4s to .webvtt so far, but it more and more seems to me like our best shot now. Do you have any other ideas? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AI03s2DU2AvPQ6tXzjhzXCeBKDsReWHMks5uTPmygaJpZM4Vr0BK .

Any updates on this?

[joeyparrish]

The project is dead. We apologize for any inconvenience. Feel free to continue it in a fork if you wish. See also a bit more detail in #19.