Support VobSub subtitles

[AliZgheib]

Hello everyone, I hope you are doing well today.

Introduction:
since the last few days, I've been working on a mobile application where I'm required to build a video player that supports subtitles/ audio track control and standard (forward, backward, play, pause) buttons
and this is where I've found about exo player, I've installed the main demo and ran it on my phone it worked really well.

Issue:
The media that I'm looking to play in the exoplayer is encoded using H 265 method ( the audio/text track aren't found in the meta data, they are embedded in the media it self) that is what I've learnt so far.
When I ran my media using the player, the video ran.. which is good, I was able to change audio track ( that were extracted from the media and shown in the dropdown menu), however the text track didn't show any option. I'm expected to find (English and Portuguese) and none of these options were showing. I was able to see options like Auto or Default (on some of the test videos) but changing between them doesn't show any subtitles.

Possible problem:
(I'm not sure if these are the real issues, maybe I'm missing something... and I cant see the subtitles for a different reason).

1. I've read somewhere that this is maybe caused by the text subtitles embedded in my media file being encoded by a method not supported by exoplayer.

2. CLOSED_CAPTIONS=NONE which is read that it can prevent exoplayer from reading the subtitles even if they exist

Not sure if these problems are causing my issue, any help to verify the issue and help me solve it would be really greatly appreciated :)

Additional:

Link to test the media file (if needed):
H 265 video link

Thanks alots in advance

[icbaker]

Thanks for providing the media - it looks like the subtitles are in VobSub format. This isn't supported by ExoPlayer. We would accept a high quality pull request implementing support for this - there are several existing decoders in the text package that might provide inspiration.

I'll leave this issue open to track potentially adding support for this format.

$ mediainfo ~/Downloads/file_mp4.mp4
General
Complete name                            : /usr/local/google/home/ibaker/Downloads/file_mp4.mp4
Format                                   : MPEG-4
Format profile                           : Base Media / Version 2
Codec ID                                 : mp42 (mp42/isom/hvc1)
File size                                : 2.67 GiB
Duration                                 : 2 h 17 min
Overall bit rate mode                    : Variable
Overall bit rate                         : 2 771 kb/s
Movie name                               : Sobre Meninos e Lobos  BluRay 1080p
Encoded date                             : UTC 2020-09-01 18:05:19
Tagged date                              : UTC 2020-09-01 18:09:31
Writing application                      : DVDFab 11.0.8.9
Cover                                    : Yes

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main@L4@Main
Codec ID                                 : hvc1
Codec ID/Info                            : High Efficiency Video Coding
Duration                                 : 2 h 17 min
Source duration                          : 2 h 17 min
Bit rate                                 : 1 860 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 24.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Bits/(Pixel*Frame)                       : 0.037
Stream size                              : 1.79 GiB (67%)
Source stream size                       : 1.79 GiB (67%)
Encoded date                             : UTC 2020-09-01 18:05:19
Tagged date                              : UTC 2020-09-01 18:09:31
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
Menus                                    : 5
Codec configuration box                  : hvcC

Audio #1
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : mp4a-40-2
Duration                                 : 2 h 17 min
Bit rate mode                            : Constant
Bit rate                                 : 448 kb/s
Channel(s)                               : 6 channels
Channel layout                           : C L R Ls Rs LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 442 MiB (16%)
Title                                    : English (aac 5.1)
Language                                 : English
Default                                  : Yes
Alternate group                          : 1
Encoded date                             : UTC 2020-09-01 18:05:19
Tagged date                              : UTC 2020-09-01 18:09:31

Audio #2
ID                                       : 3
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : mp4a-40-2
Duration                                 : 2 h 17 min
Bit rate mode                            : Variable
Bit rate                                 : 448 kb/s
Maximum bit rate                         : 441 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 442 MiB (16%)
Title                                    : Portuguese (aac 2.0)
Language                                 : Portuguese
Default                                  : No
Alternate group                          : 1
Encoded date                             : UTC 2020-09-01 18:05:19
Tagged date                              : UTC 2020-09-01 18:09:30

Text
ID                                       : 4
Format                                   : VobSub
Codec ID                                 : mp4s-E0
Codec ID/Info                            : The same subtitle format used on DVDs
Duration                                 : 2 h 11 min
Bit rate mode                            : Variable
Bit rate                                 : 10.9 kb/s
Maximum bit rate                         : 3 767 Mb/s
Stream size                              : 10.3 MiB (0%)
Language                                 : Portuguese
Default                                  : Yes
Forced                                   : No
Alternate group                          : 2
Encoded date                             : UTC 2020-09-01 18:05:19
Tagged date                              : UTC 2020-09-01 18:09:19

Menu
ID                                       : 5
Codec ID                                 : text
Duration                                 : 2 h 17 min
Encoded date                             : UTC 2020-09-01 18:05:19
Tagged date                              : UTC 2020-09-01 18:09:31
Menu For                                 : 1
Source duration                          : 8276792
Source frame count                       : 36
Source stream size                       : 855
Frame count                              : 36
Stream size                              : 855
00:00:00.000                             : Chapter 1
00:03:18.500                             : Chapter 2
00:06:07.791                             : Chapter 3
00:08:05.916                             : Chapter 4
00:12:21.625                             : Chapter 5
00:14:36.583                             : Chapter 6
00:18:38.416                             : Chapter 7
00:23:32.083                             : Chapter 8
00:28:04.875                             : Chapter 9
00:32:45.666                             : Chapter 10
00:35:03.875                             : Chapter 11
00:40:37.250                             : Chapter 12
00:42:41.625                             : Chapter 13
00:47:00.958                             : Chapter 14
00:51:26.833                             : Chapter 15
00:53:58.291                             : Chapter 16
01:00:09.500                             : Chapter 17
01:05:17.375                             : Chapter 18
01:07:59.500                             : Chapter 19
01:10:52.875                             : Chapter 20
01:15:30.916                             : Chapter 21
01:17:22.583                             : Chapter 22
01:23:15.000                             : Chapter 23
01:26:24.291                             : Chapter 24
01:30:53.583                             : Chapter 25
01:36:32.541                             : Chapter 26
01:40:12.125                             : Chapter 27
01:42:46.125                             : Chapter 28
01:45:33.458                             : Chapter 29
01:49:29.708                             : Chapter 30
01:55:41.333                             : Chapter 31
01:57:36.708                             : Chapter 32
02:00:10.791                             : Chapter 33
02:05:49.541                             : Chapter 34
02:10:15.916                             : Chapter 35
02:12:52.875                             : Chapter 36

[AliZgheib]

Hello @icbaker, Thank you for the reply and for checking out my media link.

Could you provide me with some resource that I can read and that will help me build my own decoder and hopefully submit a pull request in the future?

resources that might be useful for me:

1-Package where different decoders lives ( place where I've to implement VobSub decoder)
2-which method is responsible of checking encoding type and calling specific decoder
3- any general/VobSub media encoding encoding/decoding resource that can help me out

Thank you again for your time and effort!

Additional question: I tried checking the encoding method in the codec tab in VLC player, but some results were different (especially the text encoding methods). the methods that were showing for me are (spu / tx3g) not VobSub as mentioned by you. why is this happening? are these methods the same? or something wrong is happening.

[icbaker]

I haven't investigated too deeply, but some rough hints/suggestions:

• When I play this file in the demo app, the text track isn't even logged in logcat [1] - I'd expect an entry in the TextRenderer list. You can see a happier example by running the demo app and playing e.g. "Subtitles -> MPEG-4 Timed Text (tx3g, mov_text)" [2].
  • This suggests there needs to be changes in the extractor to even detect the track in the first place. For MP4 files this means changes in the the extractor.mp4 package. You can see there is already code to handle TTML and tx3g in AtomParsers - you may need to add something similar (I haven't looked into the file structure in detail, so I don't know exactly what you need to change here).
  • You'll need to make sure the subtitle data is being recognised, and its contents output to a text-type ExtractorOutput.
  • You'll probably need to add a VobSub entry to MimeTypes.
  • You might find MP4Box.js useful for analyzing the box structure of your MP4 file. You can also install it and run it as a command line tool, and extract individual tracks which could be handy to isolate the binary form of the text track (something like MP4Box file.mp4 -raw 1 to extract track 1 to its own file)
• You'll need to make a new text.vobsub package, and include at least a VobsubDecoder class, which will likely extend from SimpleSubtitleDecoder. There's lots of examples of decoders in the other text packages. Some decode text-based formats, like WebVTT, SSA, TTML and SRT, while others decode binary formats like PGS and DVB.
  • The decoder should consume the binary data provided by the extractor and convert it into Cue objects.
• Finally you'll need to register your decoder to the VobSub MIME type in SubtitleDecoderFactory#DEFAULT.
• Tests would also be great :)
  • At least a VobsubDecoderTest that takes binary samples and decodes them (see Tx3gDecoderTest as an example).
  • And a sample file for (Fragmented)Mp4ExtractorTest to show the samples are correctly extracted to a text track with the right MIME type.
  • It would be cool to have a test in the e2etest package as well, but it's not currently possible to add subtitle decoding here without introducing flakey tests - this is related to how the subtitle decoding happens on a background thread - so I recommend you don't try this (we're hoping to make this possible/easier in future).

P.S. In case you haven't seen it yet, you might also want to look at our contributing guide.

[1]

tracks [eventTime=4.61, mediaPos=0.00, window=0, period=0
  MediaCodecVideoRenderer [
    Group:0, adaptive_supported=N/A [
      [X] Track:0, id=1, mimeType=video/hevc, res=1920x1080, fps=24.0, supported=YES
    ]
  ]
  MediaCodecAudioRenderer [
    Group:0, adaptive_supported=N/A [
      [X] Track:0, id=2, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=6, sample_rate=48000, language=en, supported=YES
    ]
    Group:1, adaptive_supported=N/A [
      [ ] Track:0, id=3, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=48000, language=pt, supported=YES
    ]
    Metadata [
      TIT2: description=null: value=Sobre Meninos e Lobos  BluRay 1080p
      TPE1: description=null: value=
      TALB: description=null: value=
      COMM: language=und, description=
      TSSE: description=null: value=DVDFab 11.0.8.9
      APIC: mimeType=image/png, description=null
    ]
  ]
  TextRenderer []
  MetadataRenderer []
  CameraMotionRenderer []
]

[2]

TextRenderer [
  Group:0, adaptive_supported=N/A [
    [ ] Track:0, id=3, mimeType=application/x-quicktime-tx3g, language=en, supported=YES
  ]

[icbaker]

Ah I've realised I missed a couple of points from your comment:

3- any general/VobSub media encoding encoding/decoding resource that can help me out

I don't have any resources on this I'm afraid - if you do find a spec/reference it would be helpful to link it back here and also probably in a comment in the code itself so that a future reader/editor can refer to it as well.

Additional question: I tried checking the encoding method in the codec tab in VLC player, but some results were different (especially the text encoding methods). the methods that were showing for me are (spu / tx3g) not VobSub as mentioned by you. why is this happening? are these methods the same? or something wrong is happening.

Interesting - I see two text tracks in VLC:

"DVD Subtitles" is the same as VobSub I believe. These all seem to be different names for "the subtitle format used on DVDs".

"tx3g" is nominally supported by ExoPlayer, though we don't detect this as a subtitle track. It's interesting that mediainfo seems to think that track ID (4 in VLC (0-indexed), 5 in mediainfo (1-indexed)) is a "Menu" track. When I play the video in VLC and select that track it just says "Chapter 1" at the bottom of the screen permanently and doesn't seem to transcribe any of the speech in the video (it updates to e.g. "Chapter 3" if I seek forward), while the Portuguese (VobSub) subtitle track seems to have speech transcribed as I'd expect.

If you'd like us to look into why the tx3g track isn't detected by ExoPlayer can you please file a separate issue? It may well be working-as-intended, since it's not really a "subtitle" track but just chapter markers (and mediainfo knows this) - but if it's important we can dig into the MP4 file and help you understand what's going on.

[AliZgheib]

Hello again, and sorry for the late reply,

Thank you for your detailed answer and for giving me some references that i can work with. I'll definitely look into it when I start implementing the Vobsub decoder.

About the 2nd issue (tx3g):
I'll open another issue specific only to exoplayer not being able to detect tx3g subtitles (the chapters).

I'll keep this post updated regarding any news about the Vobsub decoder

Thank you again and have a nice day!

[icbaker]

When looking into this file for #8271 I noticed that the track box with id=4 (the vobsub one) has handler=subp which isn't mentioned in ISO 14496-12 as a valid handler - this might be why this track isn't detected at all by ExoPlayer.

If you're going to make a PR that changes to AtomParsers to detect subp tracks, it'll be important to link to a spec where that is defined - as otherwise it just looks like your media isn't compliant.

[neckaros]

@AliZgheib did you get time to work on this ?

[VanechikSpace]

@icbaker so if ExoPlayer doesn't support VobSub, then why does androidx.media3.common.MimeTypes have APPLICATION_VOBSUB field ?

[icbaker]

Interesting question. It looks like that was added in 27ab1a2. From the code in that commit it seems it implements parsing the vobsub data from the container, but no further decoding/rendering of that data.

WebmExtractor was generified to handle all Matroska content and renamed to MatroskaExtractor in 3c12398, and MatroskaExtractor still has the parsing logic present:

ExoPlayer/library/extractor/src/main/java/com/google/android/exoplayer2/extractor/mkv/MatroskaExtractor.java

Lines 2325 to 2328 in da5b9af

	case CODEC_ID_VOBSUB:
	mimeType = MimeTypes.APPLICATION_VOBSUB;
	initializationData = ImmutableList.of(getCodecPrivate(codecId));
	break;

However I can't find any trace of any vobsub decoding/rendering logic. For example, the commit above also contains logic to parse PGS subtitles from WebM/MKV, but I don't think any PGS decoding was present in the library at the time. PGS decoding was added two years later in #3673.

From that bit of archaeology, I think vobsub decoding was simply never added to complement the container parsing logic already present.