This repository has been archived by the owner. It is now read-only.

Remove "zero byte stuffing" filler bytes from MPEG2-ES #734

Closed
mbunkus opened this Issue Jan 24, 2015 · 7 comments

Comments

2 participants
@mbunkus
Owner

mbunkus commented Jan 24, 2015

Original reporter: Basic.Master

(a request especially regarding digital television via DVB:)

mkvmerge already has support for removing filler bytes from H.264 streams. The MPEG2 specification implicit also allows a kind of filler bytes called "zero byte stuffing". This means that after a block finished, it is possible to insert any amount of null bytes until the next startcode comes along in the stream.
So via the startcode, we know the end of such region. By interpreting the specification, it is also possible to recognize the beginning of such region, if it starts after a slice block (the filler data is usually between the last slice end and the following picture).
This filler technique is used to ensure a minimum bitrate; it is i.e. used at the Astra 19,2° satellite on the programs "rbb Fernsehen", "NDR Fernsehen", "MDR Fernsehen" or "BR-alpha". The broadcaster rbb told me the reason for that: There are some satellite chipsets which have problems with too low bitrates.
While the first ones have a variable filler rate (I measured i.e. 100 MB/h), "BR-alpha" has a constant filler rate of 0,6 MBit/s, resulting in 270 MB/h of wasted space. If you are going to burn a movie on DVD, it would be helpful to remove such filler bytes, in order to i.e. use a single layer instead of a double layer DVD.

I will upload two samples to the FTP server, with the prefix "mpeg2_zero_stuffing". You will find that null byte blocks everytime before the beginning of a new picture.

Here is also a thread I start in a german digital television board regarding that topic, with bitrate diagram comparison regarding a whole movie:
http://forum.digitalfernsehen.de/forum/digitale-audio-und-videobearbeitung/295718-mpeg2-fuelldaten.html

If that feature seems to be helpful for mkvmerge, I could try to make a patch; alternatively I could provide the source code of my tiny program that fulfils this single task. Or both.

[BTW to ease communication, I also speak german as my first language]

@mbunkus mbunkus self-assigned this Jan 24, 2015

@mbunkus mbunkus added this to the later milestone Jan 24, 2015

@mbunkus mbunkus closed this Jan 24, 2015

@mbunkus mbunkus modified the milestones: 5.6.0, later Jan 24, 2015

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus Jan 24, 2015

Owner

Original reporter: mbunkus

Thanks for the upload. I'll try to implement that for the release after the next one.

Owner

mbunkus commented Jan 24, 2015

Original reporter: mbunkus

Thanks for the upload. I'll try to implement that for the release after the next one.

@mbunkus mbunkus modified the milestones: later, 5.6.0 Jan 24, 2015

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus Jan 24, 2015

Owner

Original reporter: Moritz Bunkus

MPEG-1/-2 video: remove stuffing bytes

Implements #734.
Changeset: 49ecf2a

Owner

mbunkus commented Jan 24, 2015

Original reporter: Moritz Bunkus

MPEG-1/-2 video: remove stuffing bytes

Implements #734.
Changeset: 49ecf2a

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus Jan 24, 2015

Owner

Original reporter: mbunkus

A build for Windows including this feature is available here (build numbers 459 and up).

Owner

mbunkus commented Jan 24, 2015

Original reporter: mbunkus

A build for Windows including this feature is available here (build numbers 459 and up).

mbunkus added a commit that referenced this issue Jul 15, 2017

MPEG-1/-2: remove feature "remove stuffing bytes"
The feature was implemented by removing all 0 bytes in before the next
start code (and all 0 bytes at the end of the buffer). The problem is
that a slice structure may very well end in 0 bytes. The only way to
determine the end of the slice structure with confidence is
implementing a parser for the whole slice structure.

The result of removing bytes belonging to the slice structure may or
may not end in visual artifacts upon decoding. Other results include
error message by the decoder (e.g. ffmpeg which reports errors such as
"slice mismatch" or "motion vectors not available").

I lack the time and motivation to implement a proper slice parser. As
the current behavior is dangerous and just plain wrong, I'm removing
the feature again. It was introduced in release 5.8.0 response to
issue #734, which will now remain not implemented.

Fixes #2045.
@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus Jul 15, 2017

Owner

The feature has been removed again. See the commit linked above this comment for the rationale.

Owner

mbunkus commented Jul 15, 2017

The feature has been removed again. See the commit linked above this comment for the rationale.

@basicmaster

This comment has been minimized.

Show comment
Hide comment
@basicmaster

basicmaster Jul 21, 2017

Hmm, what happens if you limit the stuffing removal to only stuffing that occurs after slice blocks? Because if I get the now disabled code right, the removal has been done for all block types (which will not necessarily work).

basicmaster commented Jul 21, 2017

Hmm, what happens if you limit the stuffing removal to only stuffing that occurs after slice blocks? Because if I get the now disabled code right, the removal has been done for all block types (which will not necessarily work).

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus Jul 21, 2017

Owner

Please read the commit comment linked to. It lays out why removing stuffing bytes from after slices is what's causing the problem.

Owner

mbunkus commented Jul 21, 2017

Please read the commit comment linked to. It lays out why removing stuffing bytes from after slices is what's causing the problem.

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus Jul 21, 2017

Owner

The sample file I have gets broken by my code because too many 0 bytes are removed from the end of a slice. The same file shows the same corruption with the removal code you yourself had provided way back when it was implemented. Removing 0 bytes from the end that way is simply broken. That it works most of the time is nice and all, but that doesn't make it any more correct.

Owner

mbunkus commented Jul 21, 2017

The sample file I have gets broken by my code because too many 0 bytes are removed from the end of a slice. The same file shows the same corruption with the removal code you yourself had provided way back when it was implemented. Removing 0 bytes from the end that way is simply broken. That it works most of the time is nice and all, but that doesn't make it any more correct.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.