Extracting frames from video exmple

[philtomson]

We need to extract frames from video (mpeg) and do some processing on each image (and eventually put them back together as a video stream). Primarily we want to do some comparisions between consecutive frames.

Is this sort of thing possible with VideoIO? Would it be possible to add an example that does something like this?

[maxruby]

Hi, I am working on this. I have been reading through the libav and ffmpeg documentation the last couple of days and started testing an example. Acquiring the video stream using VideoIO U (decoding) is trivial once you set up the camera device, but seeking and and encoding back is not as trivial. However, I found it is necessary to go back to the original ffmpeg documentation, i.e., how to do streaming with native ffmpeg in the first place before setting it up with the VideoIO wrapper.

see the post in the Jula-dev discussion group
https://groups.google.com/forum/#!searchin/julia-users/video$20stream/julia-users/ERhGoM6hhTc/YfuxkDJy_d0J

[kmsquire]

Hi Phil, as a stop gap, one option is to write out a series of images, and then call out to ffmpeg/avconv directly to convert the images back into a movie. But it would definitely be great to have this functionality in VideoIO. (Otherwise, it should just be called VideoInput!)

Max, it's great that you're working on this!

If you'd like, as you move past testing and onto implementing something with the main code base, you can open up a pull request marked with "WIP:" (work in progress), and I (and maybe others) can give feedback as you develop.

I've found this approach quite useful in mainline Julia, and for this repo especially, I've sometimes found it challenging to interact with the ffmpeg libraries. I've also come up with a few tricks to make things easier, which I can point out to you as you code.

One thing that will almost certainly be necessary is to implement support for AVOptions (#27). Have you looked into that at all? If not, I'll try to take a stab at it soon (but if anyone else wants to try, let me know!).

[maxruby]

Yes, I am more more familiar now how to work with ffmpeg directly to do the streaming although I am determined to try to get it working with VideoIO directly. The whole point of migrating to Julia from MATLAB for image processing is to avoid using too many dependencies (avoid the mex files) and take advantage of the beautiful syntax in Julia :) . . . @kmsquire, I will definitely pull when ready. But I think I need more time to make sure I have something really working in my own hands. I will certainly ask you about the ffmpeg and libav libraries soon - there is a lot to cover if we want to take full advantage of the functionality of ffmpeg (e.g., codecs and muxers).

Having gone through avio.jl and the ffmpeg documentation, the main challenge is figure out how to encode the images after getting the stream. I am trying to annotate everything in a markdown file so that we easily add this to VideoIO.

I will look immediately into the AVOptions again. I am sure you will be much quicker to fix it though! But I am really eager to learn and get the streaming working.

[kmsquire]

Excellent, thanks Max, I really appreciate your help!

[maxruby]

@kmsquire. I have been running and testing through the AVOptions in ffmpeg and there is obviously quite a few. However, since audio is not a priority right now and as far as I understand it is not functional yet with VideoIO, I would prefer to concentrate primarily on the video options for now.

Of the "generic" AVOptions, I found AVFormatContext in libavformat_h.jl (below), and AVCodecContext in libavcodec_h.jl. However, I am a little confused about the organization of these options (and their naming) in the VideoIO files relative to the original c/h files in libavformat, libavdevice and libavcodec. Could you explain how you organized these in your files? I need to find all the private options (containers, devices and codecs) as well. Thanks for your help.

In libavformat_h.jl
immutable AVFormatContext
av_class::Ptr{AVClass}
iformat::Ptr{AVInputFormat}
oformat::Ptr{AVOutputFormat}
priv_data::Ptr{Void}
. . .
end

In libavcodec_h.jl
immutable AVCodecContext
av_class::Ptr{AVClass}
bit_rate::Cint
bit_rate_tolerance::Cint
flags::Cint
. . .
end

[kmsquire]

@maxruby, see #27 (comment)

[IanButterworth]

Decoding and encoding is now well supported, with some simpler API's on the encoding side as of v0.9