This repository has been archived by the owner. It is now read-only.

[Regression] only first video stream of dvb-t2 recording since v12.0.0 found #1990

Closed
nicodorn opened this Issue May 21, 2017 · 10 comments

Comments

@nicodorn

nicodorn commented May 21, 2017

I have a dvb-t2 ts recording of an entire transponder that has 5 channels, 10 audio streams, and 10 subtitle streams. ffmpeg lists the streams as follows:

Duration: 02:00:02.14, start: 39911.671467, bitrate: 20578 kb/s
Program 2001
	Metadata:
		service_name    : ?ZDF HD
		service_provider: ?ZDFmobil
	Stream #0:11[0x83e]: Video: hevc (Main) ([36][0][0][0] / 0x0024), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 50 tbc
	Stream #0:21[0x848](deu): Audio: eac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, fltp, 256 kb/s
	Stream #0:14[0x849](mis): Audio: eac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, fltp, 128 kb/s
	Stream #0:8[0x852](deu): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)
	Stream #0:22[0x853](deu): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006) (hearing impaired)
	Stream #0:23[0x87a]: Unknown: none ([5][0][0][0] / 0x0005)
Program 2002 
	Metadata:
		service_name    : ?ZDFinfo HD
		service_provider: ?ZDFmobil
	Stream #0:7[0x8a2]: Video: hevc (Main) ([36][0][0][0] / 0x0024), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 50 tbc
	Stream #0:25[0x8ac](deu): Audio: eac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, fltp, 256 kb/s
	Stream #0:12[0x8ad](mis): Audio: eac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, fltp, 128 kb/s
	Stream #0:10[0x8b6](deu): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)
	Stream #0:26[0x8b7](deu): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006) (hearing impaired)
	Stream #0:27[0x8de]: Unknown: none ([5][0][0][0] / 0x0005)
Program 2003 
	Metadata:
		service_name    : ?zdf_neo HD
		service_provider: ?ZDFmobil
	Stream #0:15[0x906]: Video: hevc (Main) ([36][0][0][0] / 0x0024), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 50 tbc
	Stream #0:16[0x910](deu): Audio: eac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, fltp, 256 kb/s
	Stream #0:9[0x911](mis): Audio: eac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, fltp, 128 kb/s
	Stream #0:17[0x91a](deu): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)
	Stream #0:18[0x91b](deu): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006) (hearing impaired)
	Stream #0:19[0x942]: Unknown: none ([5][0][0][0] / 0x0005)
Program 2004 
	Metadata:
		service_name    : ?3sat HD
		service_provider: ?ZDFmobil
	Stream #0:28[0x96a]: Video: hevc (Main) ([36][0][0][0] / 0x0024), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 50 tbc
	Stream #0:24[0x974](deu): Audio: eac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, fltp, 256 kb/s
	Stream #0:13[0x975](mis): Audio: eac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, fltp, 128 kb/s
	Stream #0:20[0x97e](deu): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)
	Stream #0:29[0x97f](deu): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006) (hearing impaired)
	Stream #0:30[0x9a6]: Unknown: none ([5][0][0][0] / 0x0005)
Program 2005 
	Metadata:
		service_name    : ?KiKA HD
		service_provider: ?ZDFmobil
	Stream #0:0[0x9ce]: Video: hevc (Main) ([36][0][0][0] / 0x0024), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 50 tbc
	Stream #0:1[0x9d8](deu): Audio: eac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, fltp, 256 kb/s
	Stream #0:2[0x9d9](mis): Audio: eac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, fltp, 128 kb/s
	Stream #0:3[0x9e2](deu): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)
	Stream #0:4[0x9e3](deu): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006) (hearing impaired)
	Stream #0:5[0xa0a]: Unknown: none ([5][0][0][0] / 0x0005)
No Program
	Stream #0:6[0x1114]: Unknown: none

Until v11.0.0 I could open files like that with MKVToolNix GUI and mux the streams I want to keep in a mkv file. Since v12.0.0 MKVToolNix GUI shows only one video stream (in this case program 2005) and all 10 audio streams - at least in this case. Opening a recording of a different transponder resulted in showing only one video stream and the two corresponding audio streams. Besides subtitle streams nothing else showed up.

Please tell me if you need more info.

@nicodorn

This comment has been minimized.

Show comment
Hide comment
@nicodorn

nicodorn May 21, 2017

A picture is worth a thousand words ;)

streams

nicodorn commented May 21, 2017

A picture is worth a thousand words ;)

streams

@nicodorn nicodorn closed this May 21, 2017

@nicodorn nicodorn reopened this May 21, 2017

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus May 21, 2017

Owner

Please upload a sample file to my FTP server so that I can take a look. Thanks.

Note that I'll be out of town for several weeks. I will most likely not have time to look into it before I get back.

Owner

mbunkus commented May 21, 2017

Please upload a sample file to my FTP server so that I can take a look. Thanks.

Note that I'll be out of town for several weeks. I will most likely not have time to look into it before I get back.

@mbunkus mbunkus changed the title from [Regression] GUI only lists first video stream of dvb-t2 recording since v12.0.0 to [Regression] only first video stream of dvb-t2 recording since v12.0.0 found May 21, 2017

@nicodorn

This comment has been minimized.

Show comment
Hide comment
@nicodorn

nicodorn May 21, 2017

Thank you!

I did two recordings from two different transponders, each 1 min. long, and uploaded them to your server. I did two because each one behaves slightly different:

  • "Bug #1990, Transponder 2.ts" gives me the exact result you can see in the picture above: 1 video stream, all audio streams listed.
  • "Bug #1990, Transponder 1.ts" does only list 1 video stream, and two audio streams despite the file has 5 video an 12 audio streams.

nicodorn commented May 21, 2017

Thank you!

I did two recordings from two different transponders, each 1 min. long, and uploaded them to your server. I did two because each one behaves slightly different:

  • "Bug #1990, Transponder 2.ts" gives me the exact result you can see in the picture above: 1 video stream, all audio streams listed.
  • "Bug #1990, Transponder 1.ts" does only list 1 video stream, and two audio streams despite the file has 5 video an 12 audio streams.
@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus May 21, 2017

Owner

Thanks. The issue is that your streams contain multiple programs. mkvmerge simply doesn't support selecting the program to use. It just uses the first program.

At the moment I have no plans for adding support for program presentation and selection. I'll leave this report open, though, as it would be a useful feature to have in general. Just don't expect a solution any time soon.

Owner

mbunkus commented May 21, 2017

Thanks. The issue is that your streams contain multiple programs. mkvmerge simply doesn't support selecting the program to use. It just uses the first program.

At the moment I have no plans for adding support for program presentation and selection. I'll leave this report open, though, as it would be a useful feature to have in general. Just don't expect a solution any time soon.

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus May 21, 2017

Owner

That mkvmerge finds certain streams from different programs (or doesn't find them) is more or less an accident. Like I said, proper support for programs is simply not present.

Owner

mbunkus commented May 21, 2017

That mkvmerge finds certain streams from different programs (or doesn't find them) is more or less an accident. Like I said, proper support for programs is simply not present.

@nicodorn

This comment has been minimized.

Show comment
Hide comment
@nicodorn

nicodorn May 21, 2017

An accident that worked just perfectly until v11 ;)

But okay, I'll dump MKVNixTools and stick to ffmpeg from now on.

nicodorn commented May 21, 2017

An accident that worked just perfectly until v11 ;)

But okay, I'll dump MKVNixTools and stick to ffmpeg from now on.

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus May 22, 2017

Owner

mkvmerge v11.0.0 did not really handle your source files correctly. For example, muxing the audio track with ID 12 from your uploaded file Bug #1990, Transponder 1.ts results in an unplayable audio track. The track is detected, yes, but its content is pure garbage. So if you used mkvmerge < 12 for such files, then please make sure that the resulting files are fully playable, that their track content isn't broken.

I'm working on fixing mkvmerge's part in all of this so that it'll detect all tracks of all programs (think of "programs" meaning "TV channel") and handles them correctly. That'll restore detection of all tracks and it wil actually fix the broken handling of tracks from a prorgram other than the first.

However, that's only half of what I plan to implement in order to properly solve this issue. The second part will be adjusting the GUI. At the moment the GUI only presents a long list of tracks without any indication which program each track belongs to. I plan to let the GUI present a list of programs along with their station names upon adding the file so that the user can make an informed decision which tracks to copy and which to skip. There's probably be a separate column in the tracks list view for that information, too.

For example, in your first example file there are five different programs: Das Erste HD, arte HD, PHOENIX HD, tagesschau24 HD and ONE HD; that's why there are five video tracks, too. At the moment there's no indication that e.g. track ID 12 (the third video track) is the PHOENIX HD channel.

Owner

mbunkus commented May 22, 2017

mkvmerge v11.0.0 did not really handle your source files correctly. For example, muxing the audio track with ID 12 from your uploaded file Bug #1990, Transponder 1.ts results in an unplayable audio track. The track is detected, yes, but its content is pure garbage. So if you used mkvmerge < 12 for such files, then please make sure that the resulting files are fully playable, that their track content isn't broken.

I'm working on fixing mkvmerge's part in all of this so that it'll detect all tracks of all programs (think of "programs" meaning "TV channel") and handles them correctly. That'll restore detection of all tracks and it wil actually fix the broken handling of tracks from a prorgram other than the first.

However, that's only half of what I plan to implement in order to properly solve this issue. The second part will be adjusting the GUI. At the moment the GUI only presents a long list of tracks without any indication which program each track belongs to. I plan to let the GUI present a list of programs along with their station names upon adding the file so that the user can make an informed decision which tracks to copy and which to skip. There's probably be a separate column in the tracks list view for that information, too.

For example, in your first example file there are five different programs: Das Erste HD, arte HD, PHOENIX HD, tagesschau24 HD and ONE HD; that's why there are five video tracks, too. At the moment there's no indication that e.g. track ID 12 (the third video track) is the PHOENIX HD channel.

@mbunkus mbunkus added the type:bug label May 22, 2017

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus May 22, 2017

Owner

In essence this issue contains a pre-existing bug, a kind-of-regression and an enhancement.

Owner

mbunkus commented May 22, 2017

In essence this issue contains a pre-existing bug, a kind-of-regression and an enhancement.

mbunkus added a commit that referenced this issue May 22, 2017

MPEG TS: parse all PMTs in streams with multiple programs properly
Earlier versions of mkvmerge used to detect all tracks in MPEG
transport streams with multiple programs, even though the code wasn't
really implemented & tested for that. However, some tracks (usually
those from the second or a later program) were broken: they might not
contain any data, or only invalid data.

On top of that mkvmerge v12.0.0 contains a fix for #1980 where a track
isn't part of a PMT at all. An unintentional consequence of that fix
was that mkvmerge no longer detected all of the tracks in
multi-program streams. The reason is that in order to detect tracks
not mentioned in a PMT mkvmerge has to do detection by content in the
PES packets. That's only implemented for AAC at the moment. All other
tracks will be blacklisted as soon as they're found.

This wouldn't be a problem if all PMTs of all programs were always
located right at the start of the file with nothing in
between. Unfortunately many files contain track content between
PMTs. So that workflow was:

• mkvmerge finds first PMT, determines types for tracks listed in it
• mkvmerge now considers the PMT to be found
• Continuing scanning the file mkvmerge encounters content for tracks
  not listed in the first PMT, attempting type detection by content,
  failing for most and blacklisting their PIDs
• Next a second PMT is found, however, the PIDs listed in that PMT may
  have already been found and blacklisted before — therefor they won't
  be considered anymore

With this fix mkvmerge actively looks for the PMTs for all
programs. Detection by content is only attempted once all PMTs have
been located. That way all tracks will be detected again.

A side effect of either this patch or one of the other ones before is
that the track content is now OK. I don't know exactly why or which
commit actually fixed it.

Fixes parts of #1990.
@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus May 22, 2017

Owner

Here are new pre-builds for Windows you can try. With those all tracks should be found properly.

Owner

mbunkus commented May 22, 2017

Here are new pre-builds for Windows you can try. With those all tracks should be found properly.

mbunkus added a commit that referenced this issue May 26, 2017

MPEG TS: parse SDT, output program info as container properties in id…
…entification result

SDT = service description table

The information output is a list of three-element maps:

• the program number
• the service provider's name (think TV station name, e.g. "ARD")
• the service's name (think TV channel name, e.g. "arte HD")

The program number corresponds to the track property `program_number`.

See #1990 for the future use case: presenting this information in the
GUI.

mbunkus added a commit that referenced this issue May 27, 2017

GUI: parse program information from container identification
This is in preparation of presenting the program data for #1990.

mbunkus added a commit that referenced this issue May 27, 2017

@mbunkus mbunkus closed this May 27, 2017

@nicodorn

This comment has been minimized.

Show comment
Hide comment
@nicodorn

nicodorn May 28, 2017

Thank you very much for your efforts! Unfortunately, I have no acces to a windows machine, so I cannot test your fix. But I will look into it, as soon as you release the next version. If I experience any bugs regarding this issue, I will report them.

ffmpeg also has the sound problem, especially with files from the transponder you mentioned. Recoding the sound them during the muxing process helps.

nicodorn commented May 28, 2017

Thank you very much for your efforts! Unfortunately, I have no acces to a windows machine, so I cannot test your fix. But I will look into it, as soon as you release the next version. If I experience any bugs regarding this issue, I will report them.

ffmpeg also has the sound problem, especially with files from the transponder you mentioned. Recoding the sound them during the muxing process helps.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.