This repository has been archived by the owner. It is now read-only.

Files with more than 127 tracks are corrupt #2103

Closed
JuliettWhiskey opened this Issue Sep 25, 2017 · 7 comments

Comments

3 participants
@JuliettWhiskey

JuliettWhiskey commented Sep 25, 2017

It is not possible to create a mka or mkv with more than 127 tracks. As soon as you select another track, the file is written, no errors or warnings are shown, but the file is corrupt. It doesn't matter if the tracks are audio, video or subtitle tracks - 127 is the limit (wondering if it's something with 2^8?). Maybe it's a limitation of the matroska media container, but I didn't found any information about this.

If you multiplex the corrupt file again, MKVToolNix throw errors like:

Error in the Matroska file structure at position 1526180. Resyncing to the next level 1 element.

Still resyncing at position 389969196.
Resync failed: no valid Matroska level 1 element found.
... File.mkv: Error in the Matroska file structure at position 1526180. Resyncing to the next level 1 element.
The last timecode processed before the error was encountered was 00:00:00.162539678.

At least it should not be possible to select more than 127 tracks, but it would be great, if it's possible to create files with more tracks.

@mkver

This comment has been minimized.

Show comment
Hide comment
@mkver

mkver Sep 25, 2017

Matroska has a way to write numbers of varying size (numbers where you do not know the size (in bytes) in advance so that you have to code both the size in bytes and the number): The first nonzero bit tells you how many bytes the whole number has and the rest tells you the number; see the Matroska specification for details). In this scheme, 127 (and numbers like (2^(7n)-1) are special cases: If one coded 127 by the above recipe, it could be coded as 1111 1111 (in binary), but this is actually forbidden because it indicates an unknown length (to make Matroska more suitable for streaming, it might be advantageous to allow elements with an unknown length). The TrackNumber of the blocks (and simpleblocks) in the file are also encoded using the EBML-system, but here it doesn't make sense any more to treat the case 1111 1111 (and similar) as an exception (an unknown trackid doesn't make any sense at all) and according to the specs one can indeed use one byte for the first 127 tracks. Maybe MKVToolNix matroska demuxer does not know that 1111 1111 is a legal way to encode the number 127 as TrackNumber because it uses the ordinary function (for EBML-length) to parse the file? I'll test this. [Edit]: This is not the problem. Somehow I read that 127 tracks make problems, but this appears to work fine. MKVToolNix uses 1 byte (0xFF) to encode the TrackNumber in this case.
The Matroska file format actually allows an unlimited number of tracks (or more specifically: the current limit is 2^56-1; but you will hit other limits (length of a segment) first).

mkver commented Sep 25, 2017

Matroska has a way to write numbers of varying size (numbers where you do not know the size (in bytes) in advance so that you have to code both the size in bytes and the number): The first nonzero bit tells you how many bytes the whole number has and the rest tells you the number; see the Matroska specification for details). In this scheme, 127 (and numbers like (2^(7n)-1) are special cases: If one coded 127 by the above recipe, it could be coded as 1111 1111 (in binary), but this is actually forbidden because it indicates an unknown length (to make Matroska more suitable for streaming, it might be advantageous to allow elements with an unknown length). The TrackNumber of the blocks (and simpleblocks) in the file are also encoded using the EBML-system, but here it doesn't make sense any more to treat the case 1111 1111 (and similar) as an exception (an unknown trackid doesn't make any sense at all) and according to the specs one can indeed use one byte for the first 127 tracks. Maybe MKVToolNix matroska demuxer does not know that 1111 1111 is a legal way to encode the number 127 as TrackNumber because it uses the ordinary function (for EBML-length) to parse the file? I'll test this. [Edit]: This is not the problem. Somehow I read that 127 tracks make problems, but this appears to work fine. MKVToolNix uses 1 byte (0xFF) to encode the TrackNumber in this case.
The Matroska file format actually allows an unlimited number of tracks (or more specifically: the current limit is 2^56-1; but you will hit other limits (length of a segment) first).

@mkver

This comment has been minimized.

Show comment
Hide comment
@mkver

mkver Sep 25, 2017

I did some tests and uploaded a file ("Lots.of.Tracks.mks") to @mbunkus' ftp-server. It contains 132 tracks; it has been created from a file with 44 tracks and contains every track three times. Track number 128 is wrong:

A0 AC A1 A6 D4 0F B1 00 49 63 68 20 77 61 72 20 65 73 2C 20 64 65 72 20 64 69 63 68 20 67 65 66 75 6E 64 65 6E 20 68 61 74 2E 9B 82 07 FA
A0 AD A1 A7 40 80 0F B1 49 63 68 20 77 61 72 20 65 73 2C 20 64 65 72 20 64 69 63 68 20 67 65 66 75 6E 64 65 6E 20 68 61 74 2E 9B 82 07 FA A0 5A 3E

The first of these is track nr. 84, the second is track nr. 128. Both tracks are actually identical. But the block inside the second block group isn't correct: It misses the 0x00 for the flags after the relative timecode (actually it has a bit set that is reserved and should be set to zero). But it's even worse than that: The block's EBML length (0x27) is bigger than the length for track 84, so it is actually set to accomodate the larger TrackNumber size (same for the blockgroup). But because 0x00 is omitted, the length is wrong and the file structure is damaged: The 9B is still considered part of the block, not its duration. 82 07 FA A0 is part of the blockgroup, but the A0 is actually the beginning of another BlockGroup. 5A3E is then considered an unknown DummyElement.
[Edit]: In case of simpleblocks it's essentially the same story: The bytes for the flags is gone; in case of lacing it is overwritten with the number of blocks in the lace, otherwise it is overwritten by the first byte of the payload. If a block uses lacing, its flag byte is overwritten with number of frames in the lace, too. I don't know what happens when one has 2^14 or more tracks, thus needing three bytes to write the TrackNumber.
[Edit2]: I have no idea if MKVToolNix can read tracks with number >127 correctly. ffmpeg crashes if it has to write a track with number >126, so its Matroska muxer seems to have a bug and therefore creating samples is hard.

mkver commented Sep 25, 2017

I did some tests and uploaded a file ("Lots.of.Tracks.mks") to @mbunkus' ftp-server. It contains 132 tracks; it has been created from a file with 44 tracks and contains every track three times. Track number 128 is wrong:

A0 AC A1 A6 D4 0F B1 00 49 63 68 20 77 61 72 20 65 73 2C 20 64 65 72 20 64 69 63 68 20 67 65 66 75 6E 64 65 6E 20 68 61 74 2E 9B 82 07 FA
A0 AD A1 A7 40 80 0F B1 49 63 68 20 77 61 72 20 65 73 2C 20 64 65 72 20 64 69 63 68 20 67 65 66 75 6E 64 65 6E 20 68 61 74 2E 9B 82 07 FA A0 5A 3E

The first of these is track nr. 84, the second is track nr. 128. Both tracks are actually identical. But the block inside the second block group isn't correct: It misses the 0x00 for the flags after the relative timecode (actually it has a bit set that is reserved and should be set to zero). But it's even worse than that: The block's EBML length (0x27) is bigger than the length for track 84, so it is actually set to accomodate the larger TrackNumber size (same for the blockgroup). But because 0x00 is omitted, the length is wrong and the file structure is damaged: The 9B is still considered part of the block, not its duration. 82 07 FA A0 is part of the blockgroup, but the A0 is actually the beginning of another BlockGroup. 5A3E is then considered an unknown DummyElement.
[Edit]: In case of simpleblocks it's essentially the same story: The bytes for the flags is gone; in case of lacing it is overwritten with the number of blocks in the lace, otherwise it is overwritten by the first byte of the payload. If a block uses lacing, its flag byte is overwritten with number of frames in the lace, too. I don't know what happens when one has 2^14 or more tracks, thus needing three bytes to write the TrackNumber.
[Edit2]: I have no idea if MKVToolNix can read tracks with number >127 correctly. ffmpeg crashes if it has to write a track with number >126, so its Matroska muxer seems to have a bug and therefore creating samples is hard.

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus Sep 25, 2017

Owner

Yeah, this is most likely a bug in libMatroska. I'll see it get fixed.

Owner

mbunkus commented Sep 25, 2017

Yeah, this is most likely a bug in libMatroska. I'll see it get fixed.

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus Sep 25, 2017

Owner

BTW, in debug builds files with 128 and more tracks cannot even be written as libMatroska raises an assertion. The Windows build is not a debug build, though, therefore the assertion isn't compiled in, and execution continues.

Owner

mbunkus commented Sep 25, 2017

BTW, in debug builds files with 128 and more tracks cannot even be written as libMatroska raises an assertion. The Windows build is not a debug build, though, therefore the assertion isn't compiled in, and execution continues.

mbunkus added a commit that referenced this issue Sep 25, 2017

mbunkus added a commit that referenced this issue Sep 25, 2017

configure: require libMatroska v1.4.8
That version fixes writing block groups and simple blocks for tracks
with track number 128 (see #2103).

Also adds a test case for writing 130 tracks.
@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus Sep 25, 2017

Owner

The fix to libMatroska is Matroska-Org/libmatroska@655344d.

Owner

mbunkus commented Sep 25, 2017

The fix to libMatroska is Matroska-Org/libmatroska@655344d.

@mbunkus

This comment has been minimized.

Show comment
Hide comment
@mbunkus

mbunkus Sep 25, 2017

Owner

New pre-builds for Windows are available.

Owner

mbunkus commented Sep 25, 2017

New pre-builds for Windows are available.

@JuliettWhiskey

This comment has been minimized.

Show comment
Hide comment
@JuliettWhiskey

JuliettWhiskey Sep 25, 2017

Thanks mkver for your help and of course mbunkus for fixing it. I tried the new pre-build and now I can create working mkvs with more than 127 tracks. Thanks again. Great work!

JuliettWhiskey commented Sep 25, 2017

Thanks mkver for your help and of course mbunkus for fixing it. I tried the new pre-build and now I can create working mkvs with more than 127 tracks. Thanks again. Great work!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.