"copy" output codecs when encoding

[ghedo]

avconv supports using the special "copy" audio/video codecs, that simply copy (duh!) the streams from input to output without doing any decoding/encoding, so I've looked into adding something similar for mpv.

My idea was to add acopy/vcopy flags into encode_lavc_context, dec_audio and dec_video so that ad/vd_lavc would stuff the read AVPacket somewhere instead of decoding it, and ao/vo_lavc would read that packet and write it to the output as is.

I haven't done anything yet though, because looking into the code there are a few things that I don't understand. E.g. what happens if audio/video_decode() return an empty buffer?

Or maybe there's a better way to do this...

[ghost]

This sounds very hard and non-trivial. There are various problems with it, most of all that the playback core is designed to actually decode the video/audio data. If you feel very adventurous, you can try it, but I'm not sure what's the best approach.

E.g. what happens if audio/video_decode() return an empty buffer?

No more to decode is considered EOF, and the player will quit.

[ghost]

@divVerent: cc

[divVerent]

Hi,

stream copy is simply not implemented yet due to quite high complexity.

Maybe you want to take a look at how ffmpeg.c/avconv.c do this - there are
actually codec/format specific hacks in the core converter application to
enable this! Also, readjusting timestamps may not always be trivial when
doing this, as some codecs also internally store timestamps.

Nevertheless, patches for this are always welcome, and we won't reject them
based on complexity as long as the resulting code is at least maintainable
(hint: avconv.c is not).

However, you may rather want to consider a restricted copy mode that only
works for a few codecs we are sure we can handle by instead introducing new
raw internal formats.

We already support raw AC3 and MPEG2 streams (IIRC that includes MP3 audio,
as MP3 can be packed into a MPEG2 stream) as internal audio format (e.g. to
be used over SPDIF), so if you want to make ao_lavc accept that format if
the output codec is AC3 and a flag for copying is enabled, that may be a
lot simpler to do in the end. The hardest part here would be changing some
parts in ao_lavc.c that expect samples. The main parts would be the
shutdown logic in uninit - which you can just skip in case of a packet
based raw format - and the frame selection logic - which also becomes a lot
simpler if you're dealing with raw packets. These places then will need
some if/else hackery, but shouldn't be too bad.

A next step then could be adding more raw audio formats, especially AAC
(unless that already fits in our existing MPEG2 raw format, not sure). That
then should only require very minor changes to the ao code, and I'd suppose
the rest of mpv wouldn't care for the format anyway.

Best regards,

Rudolf
Am 19.04.2014 13:48 schrieb "wm4" notifications@github.com:

@divVerent https://github.com/divVerent: cc

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/728#issuecomment-40867661
.

[ghost]

We already support raw AC3 and MPEG2 streams (IIRC that includes MP3 audio, as MP3 can be packed into a MPEG2 stream) as internal audio format (e.g. to be used over SPDIF), so if you want to make ao_lavc accept that format if the output codec is AC3 and a flag for copying is enabled, that may be a lot simpler to do in the end. The hardest part here would be changing some parts in ao_lavc.c that expect samples.

Careful about this. Raw AC3 etc. does actually not work - you'll lose A/V sync quickly, because everything in the mpv audio chain actually expects to samples (instead of compressed packets) too. The way it works is that uncompressed audio is put into SPDIF frames (or whatever the proper term is), which basically adds a header and pads the raw packet with zero bytes. So, it's not that easy. But yes, we can follow that approach if a maximum compressed packet size is known. ad_spdif supports a number of codecs, so I expect all these can be safely transferred. Other codecs might have unbounded packet sizes.

[ghost]

The way it works is that uncompressed audio is put into SPDIF frames (or whatever the proper term is), which basically adds a header and pads the raw packet with zero bytes.

Clarification: it padds the packets with zero to a maximum size, so that the average bitrate remains constant. This way SPDIF pretends to be PCM audio.

I've thought more than once to change the internal workings of the audio chain to use frames instead of bytes. Maybe this would make it easier for raw passthrough. But that'd be a big change, so the suggested approach might be preferable.

For video, this will be much easier, because there's already a mechanism for complicated passthrough of opaque data (used for hw decoding).

[ghost]

So, do you want to work on this? I don't think anyone else has an interest in adding this feature, although there are probably many who'd want this feature.

[ghedo]

Well, if I knew how, probably yes. The approach @divVerent suggested only works on audio streams I think, and I'm not sure if it's really worth it (at least in my case it isn't, since I'd like to copy video too).

My latest approach was to simply hide the avpacket inside an mp_image , but that didn't go very far either (arguably I didn't try very hard though), and after that I kinda gave up.

AFAICT, avconv does the AVPacket -> AVFrame -> AVPacket dance, and simply skips the AVFrame part when it copies, which is what I tried to replicate. but it seems this model isn't really applicable here.

[divVerent]

If you want to copy both streams, why not just use avconv then?

Or do you want to filter audio only?
Am 30.04.2014 19:28 schrieb "Alessandro Ghedini" notifications@github.com:

Well, if I knew how, probably yes. The approach @divVerenthttps://github.com/divVerentsuggested only works on audio streams I think, and I'm not sure if it's
really worth it (at least in my case it isn't, since I'd like to copy video
too).

My latest approach was to simply hide the avpacket inside an mp_image ,
but that didn't go very far either (arguably I didn't try very hard
though), and after that I kinda gave up.

AFAICT, avconv does the AVPacket -> AVFrame -> AVPacket dance, and simply
skips the AVFrame part when it copies, which is what I tried to replicate.
but it seems this model isn't really applicable here.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/728#issuecomment-41824578
.

[ghost]

My latest approach was to simply hide the avpacket inside an mp_image ,

Sounds like a reasonable approach. Hardware decoding already does something similar (i.e. keep a pointer to a complicated data structure, instead of image data).

[ghedo]

@divVerent

If you want to copy both streams, why not just use avconv then?

That's what I'm doing. But why does mpv even support encoding if everyone can use avconv/ffmpeg instead? As far as I'm concerned avconv (or ffmpeg for that matter) has an horrible interface, so I try to avoid it, and having to use it only for the copy feature seems a bit silly if it can be implemented in mpv.

@wm4

Hardware decoding already does something similar

Interesting, I did not think of that. It seems that, while the data in the hwdec mp_image is "dummy", the other parameters (e.g. width and height) have valid values. Are those important? In any case, I'm going to have a more in-depth look at this in the the next few days.

[ghost]

It seems that, while the data in the hwdec mp_image is "dummy",

Well, typically, mpi->planes[3] points to a hardware surface. For example, for vdpau it's actually a surface number, while for VDA (OSX stuff), it's actually a pointer to some complicated OSX datastructure.

the other parameters (e.g. width and height) have valid values. Are those important?

Yes. As far as hardware decoding is concerned, these values are important for e.g. determining correct scaling of the output.

For this hypothetical "copy" encoding, I'm not sure how much it will matter (other than displaying the information on the terminal). Of course vo_lavc currently uses the image dimensions to initialize the encoder, but you'd hack using encoders out of the copy code path.

Anyway, the really hard part is remuxing itself. For example, h264 in mkv is actually not the same as h264 in mp4, and you need a "bitstream filter" (some obscure ffmpeg API) to convert them. (Or maybe it wasn't mkv and mp4, but another pair.)

[divVerent]

Hi,

Oh, that's easy.

Even today, ffmpeg doesn't provide some really useful features: subtitle
merging, and some other video filters mpv has.

ffmpeg now has a half excuse of the feature that requires the subtitles to
come from an external .ass file and doesn't support .mkv embedded fonts in
any useful way.

Also, ffmpeg lacks a way to select audio/subtitle streams by language.

In the end, you can shoehorn all these features (except for the missing
video filters) into a ffmpeg wrapper that "sometimes" works (some things
were not fixable, e.g. the font selection for ass+mkv does not match
fontconfig), but that's quite fragile (I did so actually - look for ffstuff
in my github page). It needs an extra pass over the whole video file to
extract subtitles, and annoying temporary space use. IIRC subtitles were
also not rendered at the right font size.

Also, mpv has more input sources than ffmpeg (e.g. mpv can read from
extracted DVDs, while ffmpeg fails at that - the -f concat way fails when
DVD subtitles are involved, for example).

Now, as for audio, I am not aware of any relevant filters mpv has that
ffmpeg does not.

mpv's encoding feature was primarily designed to stay stable over time. It
is specifically written as a vo/ao driver set so that changes to the core
player don't need to be duplicated in encoding related code, as exactly
that was what made mencoder so broken (as mplayer guys did changes to the
core but never ported them from mplayer.c to mencoder.c). To avoid this
kind of bit rot, the encoding feature in mpv is written in a layer that
hardly changes (other than minor stuff like renames), and even if it
changes, it shouldn't be too hard to fix it (in worst case the fix will be
an adaptor that slows stuff down, like what happened for planar audio
support for a while).

Now, the issue with stream copy is that mpv's code makes some assumptions -
namely, that it can see every frame, and edit its timestamp. The latter is
needed for e.g. working with multiple input files, or fixing audio/video
pts discrepancies. If you can support raw packets in a way supporting this,
it shouldn't be too problematic to do.

One thing I wonder though - why don't we just use wrappers to
avcodec_encode_frame and avcodec_decode_frame (sorry, these names are not
exact, didn't look up) that support a special "raw" codec that just passes
through the input data (for audio e.g. in form of padded packets, for video
as a single plane pointing at the original encoded data)? This might
actually be easier than changing the encoding code. Not sure, though.

And if we make such wrappers - why not have them in lavc directly? :)

Best regards,

Rudolf Polzer
Am 01.05.2014 20:41 schrieb "Alessandro Ghedini" notifications@github.com:

@divVerent https://github.com/divVerent

If you want to copy both streams, why not just use avconv then?

That's what I'm doing. But why does mpv even support encoding if everyone
can use avconv/ffmpeg instead? As far as I'm concerned avconv (or ffmpeg
for that matter) has an horrible interface, so I try to avoid it, and
having to use it only for the copy feature seems a bit silly if it can be
implemented in mpv.

@wm4 https://github.com/wm4

Hardware decoding already does something similar

Interesting, I did not think of that. It seems that, while the data in the
hwdec mp_image is "dummy", the other parameters (e.g. width and height)
have valid values. Are those important? In any case, I'm going to have a
more in-depth look at this in the the next few days.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/728#issuecomment-41942012
.

[ghost]

And if we make such wrappers - why not have them in lavc directly? :)

Sounds like it could work... bonus points if it can use the bitstream filter stuff and all these headaches.

[ghost]

Ping?

[ghost]

Nobody even maintains our encoding mode, so I guess this is hopeless anyway.