embedded subtitles not detected

[NicoLeOca]

Before submitting your issue:

Enable debug logging in Medusa settings, reproduce the error (be sure to disable after the bug is fixed)

Branch/Commit: last develop
OS: windows and ubuntu
What you did: web-dl video containing embedded subs were subtitles was downloaded by Medusa
What happened: the video is stuck in the PP folder and Medusa doesn't detect the embedded subs
What you expected: file to be PP as usual
Logs:

2016-11-04 17:41:04 INFO Thread-23 :: [f430875] Successfully processed
2016-11-04 17:41:04 INFO Thread-23 :: [f430875] Not deleting folder C:\Users*\Downloads\AutoDL\The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB found the following files: [u'The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB.mkv']
2016-11-04 17:41:04 DEBUG Thread-23 :: [f430875] No subtitles associated. Postponing the post-process of this file: The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB.mkv
2016-11-04 17:41:04 DEBUG Thread-23 :: [f430875] No associated files for C:\Users*\Downloads\AutoDL\The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB\The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB.mkv were found during this pass
2016-11-04 17:41:04 DEBUG Thread-23 :: [f430875] Parsed 'C:\Users*\Downloads\AutoDL\The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB\The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB.mkv' into title: The Mindy Project, season: 5, episode: [5], episode_title: Leland Breakfast Is The Miracle Worker, screen_size: 720p, format: WEBRip, audio_codec: AAC, audio_channels: 2.0, video_codec: h264, release_group: NTB, container: mkv, mimetype: video/x-matroska, type: episode, absolute_episode: [], quality: 720p WEB-DL
2016-11-04 17:41:03 DEBUG Thread-23 :: [f430875] Parsed 'The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB.mkv' into title: The Mindy Project, season: 5, episode: [5], episode_title: Leland Breakfast Is The Miracle Worker, screen_size: 720p, format: WEBRip, audio_codec: AAC, audio_channels: 2.0, video_codec: h264, release_group: NTB, container: mkv, mimetype: video/x-matroska, type: episode, absolute_episode: [], quality: 720p WEB-DL
2016-11-04 17:41:03 DEBUG Thread-23 :: [f430875] Processing folder C:\Users*\Downloads\AutoDL\The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB
2016-11-04 17:41:03 INFO Thread-23 :: [f430875] Not deleting folder C:\Users*\Downloads\AutoDL\Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD found the following files: [u'Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD.mkv']
2016-11-04 17:41:03 DEBUG Thread-23 :: [f430875] No subtitles associated. Postponing the post-process of this file: Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD.mkv
2016-11-04 17:41:03 DEBUG Thread-23 :: [f430875] No associated files for C:\Users*\Downloads\AutoDL\Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD\Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD.mkv were found during this pass
2016-11-04 17:41:03 DEBUG Thread-23 :: [f430875] Parsed 'C:\Users*\Downloads\AutoDL\Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD\Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD.mkv' into title: Hell on Wheels, season: 5, episode: [8], screen_size: 1080p, format: BluRay, video_codec: h264, release_group: SHORTBREHD, container: mkv, mimetype: video/x-matroska, type: episode, absolute_episode: [], quality: 1080p BluRay
2016-11-04 17:41:03 DEBUG Thread-23 :: [f430875] Parsed 'Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD.mkv' into title: Hell on Wheels, season: 5, episode: [8], screen_size: 1080p, format: BluRay, video_codec: h264, release_group: SHORTBREHD, container: mkv, mimetype: video/x-matroska, type: episode, absolute_episode: [], quality: 1080p BluRay
2016-11-04 17:41:03 DEBUG Thread-23 :: [f430875] Processing folder C:\Users*\Downloads\AutoDL\Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD
2016-11-04 17:41:03 INFO Thread-23 :: [f430875] Not deleting folder C:\Users*\Downloads\AutoDL\Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2 found the following files: [u'Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2.mkv']
2016-11-04 17:41:03 DEBUG Thread-23 :: [f430875] No subtitles associated. Postponing the post-process of this file: Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2.mkv
2016-11-04 17:41:03 DEBUG Thread-23 :: [f430875] No associated files for C:\Users*\Downloads\AutoDL\Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2\Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2.mkv were found during this pass
2016-11-04 17:41:03 DEBUG Thread-23 :: [f430875] Parsed 'C:\Users*\Downloads\AutoDL\Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2\Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2.mkv' into title: Gotham, season: 3, episode: [7], episode_title: The Red Queen, screen_size: 1080p, format: WEB-DL, audio_codec: DolbyDigital, audio_channels: 5.1, video_codec: h264, release_group: R2D2, container: mkv, mimetype: video/x-matroska, type: episode, absolute_episode: [], quality: 1080p WEB-DL
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Parsed 'Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2.mkv' into title: Gotham, season: 3, episode: [7], episode_title: The Red Queen, screen_size: 1080p, format: WEB-DL, audio_codec: DolbyDigital, audio_channels: 5.1, video_codec: h264, release_group: R2D2, container: mkv, mimetype: video/x-matroska, type: episode, absolute_episode: [], quality: 1080p WEB-DL
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Processing folder C:\Users*\Downloads\AutoDL\Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Parsed 'The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB.mkv' into title: The Mindy Project, season: 5, episode: [5], episode_title: Leland Breakfast Is The Miracle Worker, screen_size: 720p, format: WEBRip, audio_codec: AAC, audio_channels: 2.0, video_codec: h264, release_group: NTB, container: mkv, mimetype: video/x-matroska, type: episode, absolute_episode: [], quality: 720p WEB-DL
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Processing folder The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Parsed 'Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD.mkv' into title: Hell on Wheels, season: 5, episode: [8], screen_size: 1080p, format: BluRay, video_codec: h264, release_group: SHORTBREHD, container: mkv, mimetype: video/x-matroska, type: episode, absolute_episode: [], quality: 1080p BluRay
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Processing folder Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Parsed 'Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2.mkv' into title: Gotham, season: 3, episode: [7], episode_title: The Red Queen, screen_size: 1080p, format: WEB-DL, audio_codec: DolbyDigital, audio_channels: 5.1, video_codec: h264, release_group: R2D2, container: mkv, mimetype: video/x-matroska, type: episode, absolute_episode: [], quality: 1080p WEB-DL
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Processing folder Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Deluge Seedbox : No processable items found in the folder
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Unable to match Deluge Seedbox to a show in your database. Parser result: title: Deluge Seedbox, type: episode, absolute_episode: [], quality: Unknown, episode: [], season: None
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Processing folder Deluge Seedbox
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] PostProcessing VideoInRar:
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] PostProcessing RarContent:
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] PostProcessing VideoFiles: []
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] PostProcessing Files: []
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] PostProcessing Dirs: [u'Deluge Seedbox', u'Gotham.S03E07.The.Red.Queen.1080p.WEB-DL.DD5.1.H.264-R2D2', u'Hell.on.Wheels.S05E08.1080p.BluRay.x264-SHORTBREHD', u'The.Mindy.Project.S05E05.Leland.Breakfast.Is.The.Miracle.Worker.720p.HULU.WEBRip.AAC2.0.H.264-NTB']
2016-11-04 17:41:02 INFO Thread-23 :: [f430875] PostProcessing Path: C:\Users*\Downloads\AutoDL
2016-11-04 17:41:02 INFO Thread-23 :: [f430875] Feature 'postpone post-processing if no subtitle available' is enabled
2016-11-04 17:41:02 DEBUG Thread-23 :: [f430875] Processing folder C:\Users*\Downloads\AutoDL

I also tested with the embedded subtitles language setup to english (which is the requested language for subs) and it didn't get better.

I have this issue for a couple of shows lately (for the last few days).
I see that Subliminal was updated 9 days ago.
could it be linked?

[EDIT] I set subliminal logs to on but I can find them
where is it?

[NicoLeOca]

@medariox @fernandog
you both might be the experts here

[fernandog]

maybe @ratoaq2 knowit implementation to detect embedded subtitles

[ratoaq2]

Hi @NicoLeOca

Can you go to the Medusa lib folder and execute knowit ?

cd <path to medusa>/lib
python -m knowit "<path to the video file>"

[NicoLeOca]

@ratoaq2
"duration": "0:43:46.464000",
"overall_bit_rate": 5381147,
"video": [
{
"number": 1,
"language": "English",
"duration": "0:43:46.460000",
"size": 1639921485,
"width": 1916,
"height": 1076,
"scan_type": "Progressive",
"aspect_ratio": 1.781,
"pixel_aspect_ratio": 1.0,
"resolution": "1080p",
"frame_rate": 23.976,
"bit_rate": 4995077,
"bit_depth": 8,
"codec": "h264",
"profile": "High@L4",
"media_type": "video/H264"
}
],
"audio": [
{
"number": 2,
"language": "English",
"duration": "0:43:46.464000",
"size": 126070272,
"codec": "AC3",
"channels_count": 6,
"channels": "5.1",
"bit_rate": 384000,
"bit_rate_mode": "Constant",
"sampling_rate": 48000,
"compression": "Lossy"
}
],
"subtitle": [
{
"number": 3,
"language": "English",
"format": "SubRip",
"encoding": "utf-8"
}
]
}

[NicoLeOca]

@fernandog
how come is it a feature request? :-) not long ago it was integrated

[fernandog]

@NicoLeOca the UNKNOWN as wanted. getting the subtitle name ("english") and compare with wanted, we never implemented

anyway the feature is ready. doing final tests so you can test it

thanks to @ratoaq2

[NicoLeOca]

@fernandog
I am not sure I get what you mean
but anyway, if I'll be more than happy to get back to the previous behaviour.

thank you @fernandog and @ratoaq2

[fernandog]

@NicoLeOca we have a setting called "Consider unknown embedded subtitles as wanted" - that's why I added a while ago.

But we never "read" the embedded language name in the mkv. We only checked if it was 'und' (unknown)

[NicoLeOca]

@fernandog
are there any chance that language detection of embedded subs could be added in the future?

[NicoLeOca]

@ratoaq2
when will knowit be used for files PPed and sorted already?

For: /media/local/TV Shows/Gotham/Season 03/Gotham.S03E09.1080p.WEB-DL.x264-CASSTUDIO.mkv
Knowit 0.1.4 found:
{
"duration": "0:43:15.296000",
"overall_bit_rate": 10255872,
"video": [
{
"number": 1,
"language": "English",
"duration": "0:43:15.304000",
"size": 3053303406,
"width": 1920,
"height": 1080,
"scan_type": "Progressive",
"aspect_ratio": 1.778,
"pixel_aspect_ratio": 1.0,
"resolution": "1080p",
"frame_rate": 23.976,
"bit_rate": 9411780,
"bit_depth": 8,
"codec": "h264",
"profile": "High@L4.1",
"encoder": "x264",
"media_type": "video/H264",
"default": true
}
],
"audio": [
{
"number": 2,
"language": "English",
"duration": "0:43:15.296000",
"size": 207623680,
"codec": "EAC3",
"channels_count": 6,
"channels": "5.1",
"bit_rate": 640000,
"bit_rate_mode": "Constant",
"sampling_rate": 48000,
"compression": "Lossy",
"default": true
}
],
"subtitle": [
{
"number": 3,
"language": "English",
"format": "SubRip",
"encoding": "utf-8"
}
]
}

[NicoLeOca]

@ratoaq2
so that subtitles are not downloaded again if they are embedded with the proper requested language

[fernandog]

@NicoLeOca full logs?

[NicoLeOca]

@fernandog
just rescan files
and here it is:
https://gist.github.com/NicoLeOca/335b51bfafc7c7b18120262323c5928e

but it seems to use cached info
if I force a full update, will it rescan files as well?

[medariox]

Knowit isn't used on rescans yet.

[NicoLeOca]

@medariox
so knowit is only used to pass PP?
cause the flag doesn't appear on the show page
and then Medusa will search for subs whatever knowit found during the PP
Am I right?

any plan to make full use of knowit in the future?

[fernandog]

yes, knowit is used only in PP to detect embedded subs.

the flag in display show don't use knowit - for now. no plans yet. depends on @ratoaq2

[medariox]

@NicoLeOca
For now, yes. Subtitle information in shows is still handled by enzyme (subliminal) atm. This means that embedded subs most likely won't be recognized.

I think the plan is to use knowit for everything in the future, but it's something that will take time, no ETA yet.