Please add support for auto-detecting 3D input format

[sre]

Hi,

The Matroska Container (*.mkv, *mk3d) has support for embedded 3D format specification. This information is exposed by ffmpeg (via metadata) and recently also by libav (via newly introduced stereo3d api [0]).

It would be nice if mpv would make use of the provided 3D information, so that the user only has to specify the 3D output format. Apart from that it would be nice if the default output format for 3D content could be specified in the configuration file.

-- Sebastian

[0] https://bugzilla.libav.org/show_bug.cgi?id=728

[ghost]

1. Are there any files like this in the wild?
2. What exactly should be supported?
3. Can you post a sample file that demonstrates all what is required (but not more)?

[sre]

Are there any files like this in the wild?

I don't know. I have files, which use the proper tags and I add the tags to files missing them. IMHO metadata should be part of the file.

What exactly should be supported?

Auto-Detection of 3D flags, which currently must always be set with mpv's stereo3d video filter.

Can you post a sample file that demonstrates all what is required (but not more)?

I currently have no small sample file with 3d content available. Is there some pool of sample files available? One can simply add the 3d flag in mkv files using mkvmerge's "--stereo-mode" option.

[ghost]

Auto-Detection of 3D flags, which currently must always be set with mpv's stereo3d video filter.

So that sounds like you want automatic filter insertion to convert the 3D stuff back to "normal" video? Actually, I have no clue how this 3D stuff works (it's an expensive fad anyway), so I don't know what the result should look like. But it's probably easily possible to do this.

I currently have no small sample file with 3d content available.

You could create a small sample from a bigger file with mkvmerge and --split.

Is there some pool of sample files available?

Not really. We have https://github.com/mpv-player/random-stuff where we sometimes dump test files. There's also http://samples.ffmpeg.org/ .

[sre]

So that sounds like you want automatic filter insertion to convert the 3D stuff back to "normal" video? Actually, I have no clue how this 3D stuff works (it's an expensive fad anyway), so I don't know what the result should look like. But it's probably easily possible to do this.

Right. I have some files flying around, which have 3d data and I want mono output. Basically there are a couple of possible 3D formats and mpv filters can be used to convert from one format to another one. For example the following mpv video filter converts from "3D data, side by side, half width" to "mono left": stereo3d=sbs2l:ml

Obviously mpv cannot know the output format, that I need/want, but it can autodetect the 3D input format from the file's metadata. Since the 3D output format is more or less a question of the hardware capabilities it would make sense to have a configuration option for the default 3D output format (so that the video filter can be omitted completely in most cases).

---

So here is a 10MB example file (based on Surfcup.mp4 from samples.ffmpeg.org), which is encoded side-by-side, half width. I added the stereo-mode metadata in the mkv file, so the "sbs2l" information from the lower command could be automatically detected from the file's metadata:

mpv --vf stereo3d=sbs2l:ml http://www.elektranox.org/test.mk3d

[ghedo]

So, if I understand things correctly, something like a default "auto" input format for the stereo3d vf that reads the file tags and autodetects the video 3d format would work for you? So you'd use something like mpv --vf=stereo3d=auto:ml <file> or mpv --vf=stereo3d=out=ml <file>, etc...

FWIW, libav has support for reading 3d data from matroska and converting it to stereo3d side data (AV_PKT_DATA_STEREO3D), so that could be used like we do with the replaygain side data in af_volume.

[ghost]

@sre: thanks for the sample. It looks pretty easy to support it (though mapping the right modes will probably require additional testing). The question is whether there are more complex cases that wouldn't be easy to handle. I know that Matroska has at least one 3D feature which is not easy to handle at all, and which would require changing everything. OTOH, fortunately apparently unused yet.

FWIW, libav has support for reading 3d data from matroska and converting it to stereo3d side data (AV_PKT_DATA_STEREO3D), so that could be used like we do with the replaygain side data in af_volume.

Note that mpv has its own mkv demuxer.

[sre]

So, if I understand things correctly, something like a default "auto" input format for the stereo3d vf that reads the file tags and autodetects the video 3d format would work for you? So you'd use something like mpv --vf=stereo3d=auto:ml or mpv --vf=stereo3d=out=ml , etc...

Yes, that would be nice. I guess the filter should be ignored if auto does not find 3D metadata. In that case 3D files and 2D files can be played using the same command. So basically the following should work:

$ grep vf .mpv/config
vf=stereo3d=auto:ml
$ mpv some-normal-file-without-3d-content.mkv
$ mpv some-file-with-3d-content.mkv

The same syntax could be useful for guys with 3D capable hardware:

$ grep vf .mpv/config
vf=stereo3d=auto:irl
$ mpv some-normal-file-without-3d-content.mkv
// plays as 2D file
$ mpv some-file-with-3d-content.mkv
// plays as 3D file in irl mode

@sre: thanks for the sample. It looks pretty easy to support it (though mapping the right modes will probably require additional testing).

Right, it shouldn't be that complicated.

The question is whether there are more complex cases that wouldn't be easy to handle. I know that Matroska has at least one 3D feature which is not easy to handle at all, and which would require changing everything. OTOH, fortunately apparently unused yet.

One can also specify, that the video data is anaglyph based. That is stored in another property.
But that kind of source material cannot be transformed into other formats. So probably no very important for mpv.

Apart from that one can put left eye and right eye into different video streams. I don't think that is used in the wild, though.

FWIW, libav has support for reading 3d data from matroska and converting it to stereo3d side data (AV_PKT_DATA_STEREO3D), so that could be used like we do with the replaygain side data in af_volume.

Note that mpv has its own mkv demuxer.

oh, I didn't know that.

[ghost]

I did something.

[sre]

👍 That's exactly the behaviour is tried to describe. Thanks!

[sre]

I just tested with some more videos and ab2l was incorrectly detected as abl.

---

After looking at the patch I can say, that the solution is incomplete. Probably for bandwidth reasons someone thought it would be a good idea to have a format side-by-side, where each frame is saved with half width. There is also a less creepy format, which stores the frames side-by-side in full width format, though. The same rules apply to top-bottom style (so there is full-height and half-height).

As a result the following formats exist:

• side-by-side, left eye first, full-width
• side-by-side, left eye first, half-width
• side-by-side, right eye first, full-width
• side-by-side, right eye first, half-width
• top-bottom, left eye first, full-height
• top-bottom, left eye first, half-height
• top-bottom, right eye first, full-height
• top-bottom, right eye first, half-height

Your patch currently maps Matroska's side-by-side definitions to half-width and Matroska's top-bottom definitions to full-width.

The Matroska specification uses a slightly more complicated method than mpv to model half-width vs full-width. The StereoMode property does not contain any information about that. Instead the properties "DisplayWidth" and "DisplayHeight" are supposed to be used.

The idea is, that the properties "PixelWidth" + "PixelHeight" contain the information about the raw image data (e.g. 3840x1080) and "DisplayWidth" + "DisplayHeight" contain the information about the merged image data (e.g. 1920x1080).

If half-width / half-height style is used PixelWidth equals DisplayWidth and PixelHeight equals DisplayHeight.

[ghost]

Some more guesswork in git.

[sre]

OK, with that change all my files work (all of them are half-width or half-height). IMHO the solution for full-width / full-height is not very elegant (*), but it seems to work, too (testfile: http://www.elektranox.org/test2.mk3d).

(*) I think the video is first squeezed together because of the display width/height property and then stretched again due to the half-width filter. Properly implemented it would be enough cut the image in the middle without any scaling being involved at all.

There is no proper and exact spec (Matroska tradition), so we probably have to rely on guessing for this.

You may want to read this: http://www.matroska.org/technical/specs/notes.html#3D

[ghost]

(*) I think the video is first squeezed together because of the display width/height property and then stretched again due to the half-width filter. Properly implemented it would be enough cut the image in the middle without any scaling being involved at all.

Not sure what you mean. I don't think the video is scaled anywhere with this mode.

You may want to read this: http://www.matroska.org/technical/specs/notes.html#3D

Yes, that is the "spec", but it's not very precise. (Although granted, the half-width thing can probably be deduced from this.)

[sre]

(*) I think the video is first squeezed together because of the display width/height property and then stretched again due to the half-width filter. Properly implemented it would be enough cut the image in the middle without any scaling being involved at all.

Not sure what you mean. I don't think the video is scaled anywhere with this mode.

Currently a full-width side-by-side video is also detected as a half-width side-by-side video:

$ mpv "http://www.elektranox.org/test2.mk3d"
...
Opening video filter: [stereo3d in=sbs2l out=mono]
Using conversion filter.
Using conversion filter.
VO: [xv] 960x540 => 960x540 yuv420p
...

So I assumed, that the video width is doubled by the stereo3d filter and halved by the display width property.

[ghost]

That specific video you posted is literally right left and right frame, as you can see with --video-stereo-mode=none --video-aspect=0. I don't think there's anything scaled.

Not sure why this shows up in your log though:

Using conversion filter.

Other filters in your filter chain?

[sre]

Compare those two test files:

• http://www.elektranox.org/test.mk3d (side by side, half width)
• http://www.elektranox.org/test2.mk3d (side by side, full width)

Both of them are currently detected as sbs2l (side-by-side, half width). So the stereo3d filter splits the image and doubles the width. This is OK for the half-width video, but obviously wrong for the full-width video.

Now the aspect ratio is calculated from the video width/height information. Since the aspect of the half-width video already matches the calculated aspect ratio nothing happens here. For the full-width video the aspect ratio does not match (its twice as wide, as it should be). Thus the video is rescaled to normal width.

Not sure why this shows up in your log though:

Using conversion filter.

Other filters in your filter chain?

One of them is because I use XV for video output, the other one is the stereo3d filter.

[ghost]

This is OK for the half-width video, but obviously wrong for the full-width video.

Well, I don't know, both look correct to me.

One of them is because I use XV for video output, the other one is the stereo3d filter.

You're probably not using libavfilter. It seems Libav doesn't have vf_stereo3d, so mpv uses the old MPlayer implementation, which behaves slightly differently. For test2.mk3d, it shows 960x540 => 1920x1080, which is strange, but harmless. The video isn't actually scaled twice. In my tests without libavfilter, both test files look correct to me, though.

Also, why are you using xv? In combination with not using libavfilter, the video is actually converted to YUV and then to RGB and then back to YUV. Must be very slow and probably has quality issues.

[sre]

This is OK for the half-width video, but obviously wrong for the full-width video.

Well, I don't know, both look correct to me.

I never said, that the result would be a problem. It looks correct.

One of them is because I use XV for video output, the other one is the stereo3d filter.

You're probably not using libavfilter. It seems Libav doesn't have vf_stereo3d, so mpv uses the old MPlayer implementation, which behaves slightly differently.

Yes, my system provides libav instead of ffmpeg.

For test2.mk3d, it shows 960x540 => 1920x1080, which is strange, but harmless. The video isn't actually scaled twice. In my tests without libavfilter, both test files look correct to me, though.

So the double scaling is optimized out at some point? I can see, that mpv --video-aspect=0 "http://www.elektranox.org/test2.mk3d" results in wrong aspect ratio (doubled width) and that mpv --video-stereo-mode=none "http://www.elektranox.org/test2.mk3d" results in wrong aspect ratio.

Also, why are you using xv? In combination with not using libavfilter, the video is actually converted to YUV and then to RGB and then back to YUV. Must be very slow and probably has quality issues.

I took this over from my mplayer config. I wasn't aware (until now), that xv has limited input formats. I have switched to opengl output (the error message about missing OpenGL 3 support is a bit anoying, though).

[ghost]

So the double scaling is optimized out at some point?

No, this are just aspect ratio hints, so there's no scaling. The aspect ratio hint just causes the VO to do scaling differently on final display. I don't know why 3D stuff sets an aspect ratio, but since vf_stereo3d apparently handles it correctly, there's no reason to worry.

Though, since the builtin stereo3d filter works in RGB only, there's some quality and performance loss to expect. The libavfilter filter seems to be able to work with YUV, though.

I took this over from my mplayer config.

Always a bad idea.

[sre]

No, this are just aspect ratio hints, so there's no scaling. The aspect ratio hint just causes the VO to do scaling differently on final display. I don't know why 3D stuff sets an aspect ratio, but since vf_stereo3d apparently handles it correctly, there's no reason to worry.

ok.

Though, since the builtin stereo3d filter works in RGB only, there's some quality and performance loss to expect. The libavfilter filter seems to be able to work with YUV, though.

I will file a feature request in the libav bugtracker.