Implement AVOptions support

[kmsquire]

AVOptions is meant to be a version-independent method for getting and setting options in ffmpeg/libav structs. It works kind of like a dictionary.

This would make solutions like #26 much nicer, in that they could properly call library functions instead of command line tools.

[kmsquire]

@kmsquire. I have been running and testing through the AVOptions in ffmpeg and there is obviously quite a few. However, since audio is not a priority right now and as far as I understand it is not functional yet with VideoIO, I would prefer to concentrate primarily on the video options for now.

Of the "generic" AVOptions, I found AVFormatContext in libavformat_h.jl (below), and AVCodecContext in libavcodec_h.jl. However, I am a little confused about the organization of these options (and their naming) in the VideoIO files relative to the original c/h files in libavformat, libavdevice and libavcodec. Could you explain how you organized these in your files? I need to find all the private options (containers, devices and codecs) as well. Thanks for your help.

In libavformat_h.jl
immutable AVFormatContext

av_class::Ptr{AVClass}
iformat::Ptr{AVInputFormat}
oformat::Ptr{AVOutputFormat}
priv_data::Ptr{Void}
. . .
end

In libavcodec_h.jl
immutable AVCodecContext
av_class::Ptr{AVClass}
bit_rate::Cint
bit_rate_tolerance::Cint
flags::Cint
. . .
end

@maxruby, sorry not to get back to you sooner. I moved this discussion to #27 because I thought the discussion might get long for #38.

The files you pointed out were auto generated by Clang.jl, using the script in the util directory. The README file there gives some details. One thing that's missing from there is related to your question.

When Clang.jl wraps a bunch of header files, such as those for each library in ffmpeg, it

1. Converts all structs to types
2. Converts #defines to either type aliases or constants
3. Wraps function definitions

The wrapped functions end up in a file with the same base name as the header file. Everything else (types, type aliases, and constants) are collected together and put in one big file (e.g., libavcodec_h.jl).

For ffmpeg, there's a larger overall structure which is described in detail in various README files.

For AVOption, I would suggest focusing first on getting something simple working with one of the types/structs that supports the AVOption interface, such as one of those you mentioned. I would focus on supporting the string interface first, since (I think) all options can be set using strings (ffmpeg does the conversion).

It's also possible that the structs currently in use aren't initialized properly to support AVOptions, so that might need to change.

Hope this helps. Let me know what might not be clear. Cheers!

[maxruby]

This definitely helps! The overall structure of ffmpeg is nicely described in README - I got that far. The part I did not understand is to do with calling all the private (rather than generic) options from AVOptions since they are all dependent on the container, device or codec. Although I am still trying to work it out, I think that you might be right that the structs are not currently initalized in a way that they can be used directly to support AVOptions. I am hoping to clear this up first before moving to the next level. . .

[maxruby]

Having gone through your files, its now much more clear what I need to do to access the parameters in the immutable types AVCodecContext and AVForrmatConext. I see you created e.g., the AVFrame() function in LIBAVCODEC.jl. to call all the parameters (types) from AVFrame (defined in libavcodec_h.jl). I have a couple of related questions:

1.LIBAVCODEC.jl in AVCodecs v53 and v54 contains the AVFrame() function, but there isn´t one in v55. Is there a reason for this? Are those the versions you were working with? The latest ffmpeg release of libavcodec (as of today) has changed to v56 and the rest of them too.
libavutil 54. 7.100
libavcodec 56. 1.100
libavformat 56. 4.101
libavdevice 56. 0.100
libavfilter 5. 1.100
libavresample 2. 1. 0
libswscale 3. 0.100
libswresample 1. 1.100
libpostproc 53. 0.100

Do you think we should take this into account when completing the wrappers? How much change is there between recent versions?

2.Since I plan to add a function AVCodecContext() similar tot AVFrame() to access the parameters from the AVCodecContext type, I wonder whether it is really is useful to support multiple version of the libraries (v52, v53, v55). I assume that support for v55/v56 should be sufficient to do the job?

Cheers,
Max

[kmsquire]

Addressing your second question first: it would certainly be easier to support only the latest version of ffmpeg/libav. However, the main reason I went the more complicated route is that I run Linux, and

1. Different Linux distributions distribute different versions of the libraries (e.g., Debian/Ubuntu only have libav in their repositories, and only a particular version).
2. Some users do not have admin rights on the machines they use, and are stuck with whatever version of the libraries are installed on their machines.

In particular, some users who have expressed interest in using this package (e.g., Lucas Beyer) have some of these issues.

This affects Mac and Windows users less, since we have greater control over what's installed there.

In general, the libraries haven't changed much since the first version I mentioned, so this hasn't been too much of a problem so far. But, e.g., AVFrame moved from libavcodec to libavutil in later versions of libav, so the AVFrame() function you found in LIBAVCODEC.jl for versions v53 and v54 appears instead in AVUtil/v53/LIBAVUTIL.jl. (One downside of the current organization is that it isn't obvious which library versions go with each other.)

We should wrap the latest ffmpeg versions. Would you mind creating an issue for it?

[maxruby]

Now it makes sense! Sure, I will create a new issue entitled "Wrapping latest versions of ffmpeg".

[lucasb-eyer]

Especially Ubuntu 12.04 LTS will still be used for quite some time in and around my office, and they only ship v53. That's why I'd like to keep support for that version as long as the effort for it stays reasonable.

Edit: if you ever need testing for something with these old versions, just ping me.

[maxruby]

OK. I will work on support for as many libraries as I can. Personally I would prefer to support the latest ones and those essential for Linux systems like v53.

[kmsquire]

Hopefully, the files under ffmpeg and libav are mostly create once--at least the auto-generated parts. For the others, some rearrangement/consolidation is probably possible, and may be worthwhile, e.g., for files like src/ffmpeg/AVCodecs/src/AVCodecs.jl. That still means duplication for some of the other files, though.

[maxruby]

@kmsquire. I have been testing several ideas about how to create access to the AVOptions in ffmpeh/libav within avio.jl and possibly some in LIBCODEC.jl, LIBAVUTIL.jl depending on what is appropriate. One is simple with minimal interface, the other might involve better documentation info, e.g., show() to the user, so that we can have better control to choose which parameters can be changed in e.g., AVInput, AVFrame, AVCodecContext, AVFrameContext, etc. I think this is actually something even FFMPEG/LIBAV could improve as it is challenging for a beginner especially to remember all the options available, even if they are available at the shell prompt.

I am finding it rather challenging to keep track of the many types and parameters from which AVCodecContext and AVFormatContext inherit their types. For example, in your avio.jl, I noticed your comment to create av_opt_set as follows:

     #av_opt_set(avin.apFormatContext[1], "probesize", "100000000", 0)
     #av_opt_set(avin.apFormatContext[1], "analyzeduration", "1000000", 0)

I got that avin is an AVInput type, with apFormatContext, a vector of pointers to AVFormatContext that contains "probesize" a AVRational type. I assume your idea here was to set probesize (Uint32) to "100000000".

Here are my questions:

1. Do you think it is worth trying to have as many of these options loaded with VideoIO or should AVOptions be considered a sort of separate module within VideoIO?
2. Is there a way in Julia to easily list all the parameters/types and functions declared within each AVFrame, AVFormatContext, etc - an inheritance tree for types and functions?

Thanks for your help.

[kmsquire]

Hi Max, I really appreciate you looking into this, and I think I haven't been that clear what I had in mind.

1. I would focus first on what the interface might look like. I haven't written this anywhere, but I had in mind something like supporting

   avin.apFormatContext[1][:probesize] = 100000000
   avin.apFormatContext[1][:analyzeduration] = 1000000

   This should be straightforward, and would only involve creating appropriate getindex and setindex! functions. Let me know if you need/want more details.

2. Initially I think access to this would be mostly for internal use. So instead of trying to support all possible options of all possible types, I would suggest focusing on one problem that needs to be solved using AVOptions:

   • For Windows Issues #25 (comment), the iformat field of AVFormatContext needs to be set (see Windows Issues #25 (comment) for some sample code) (Windows and Mac).
   • For Get available pixel formats for camera device #31, we need to use a AVDeviceCapabilitiesQuery struct (see https://www.ffmpeg.org/doxygen/trunk/structAVDeviceCapabilitiesQuery.html), which needs to be set up using AVOptions

1. Do you think it is worth trying to have as many of these options loaded with VideoIO or should AVOptions be considered a sort of separate module within VideoIO?

At this point, I would just include any needed functions in src/util.jl.

1. Is there a way in Julia to easily list all the parameters/types and functions declared within each AVFrame, AVFormatContext, etc - an inheritance tree for types and functions?

Again, I wouldn't want to support everything right now, because we don't use most of the ffmpeg/libav API. That said:

julia> using VideoIO

julia> collect(zip(names(AVCodecs.AVFrame), AVCodecs.AVFrame.types))
42-element Array{(Symbol,DataType),1}:
 (:data,Array_8_Ptr)                 
 (:linesize,Array_8_Cint)                           
 ⋮                                   
 (:sample_rate,Int32)                
 (:channel_layout,Uint64)            

(AVFrame doesn't support AVOptions, because the first member isn't an AVClass type.)

[maxruby]

That is a relief. Being a perfectionist-type, I thought that I should pursue a general support of as many AVOptions as possible. . . But I also agree that it is much better to start with a basic support for the immediately essential functions at least internally. That said, my main motivation is to access the filtering/encoding options from ffmpeg to do live video streaming for now (and hoping eventually to link this with an OpenCV wrapper) . I noticed that some people are using OpenCV with ffmpeg and it would be fantastic to transfer this combination to Julia for live streaming/computer vision.
http://siddhantahuja.wordpress.com/2011/08/15/ros-opencv-read-a-video-file-using-ffmpeg/

[kmsquire]

:+1

[maxruby]

Hi Kevin,
Thanks again for your patience. I apologize for asking so many questions.

Q1. Is it possible to use the av_setfield function (defined in util.jl) to set iformat, probesize and max_analyze_duration (the same way it was used to set pb -> apAVIOContext[1] in line 193 of avio.jl)?

iformat::Ptr{AVInputFormat} -> avin.apAVInputFormat[1]

    av_setfield(avin.apFormatContext[1], :iformat, avin.apAVInputFormat[1]) ? (Q2 below)

probesize::Uint32 -> 100000000

    av_setfield(avin.apFormatContext[1], :probesize, 100000000)

max_analyze_duration::Int32 -> 1000000

   av_setfield(avin.apFormatContext[1], :max_analyze_duration, 1000000)

Q2. Should iformat = avin.apAVInputFormat[1]? If so should apAVInputFormat be added to the type declaration?

   type AVInput{I} 
      . . .   
       apAVInputFormat::Vector{Ptr{AVInputFormat}} 
    end

[kmsquire]

Hi Max, No worries! I've been finding it difficult to make time to actually work on this stuff, and it really will be easier in the long run if someone other than just me understands it. So I'm quite happy to help you in that direction.

Q1. Is it possible to use the av_setfield function (defined in util.jl) to set iformat, probesize and max_analyze_duration (the same way it was used to set pb -> apAVIOContext[1] in line 193 of avio.jl)?

Short answer: only as a short-term solution. av_setfield itself is slightly sketchy. Useful, but sketchy.

Long answer: from version to version of ffmpeg, the structs supporting the AVOption interface can change (even for minor versions), and the fields set with the AVOption interface may disappear, even if the option itself doesn't (see, e.g., here).

If we implement things right, eventually we wouldn't even need definitions for structs that support AVOptions--we could just create, initialize, and destroy the objects with library calls, set values in them with the AVOptions API, and pass them around as opaque pointers. Getting rid of larger immutable type definitions in Julia should also allow compilation of VideoIO to be faster.

That said, it might be nice just to get something working for now, and if we can fix #25 and #31 with av_setfield, we could just do it and worry about AVOptions later. ;-)

Q2. Should iformat = avin.apAVInputFormat[1]? If so should apAVInputFormat be added to the type declaration?

I don't really know. For #25, we're wanting to get the list of video devices, which means we don't know the format. I only know that it needs to be set by looking at the source.

[maxruby]

I am happy to implement properly the library AVOptions API in VideoIO. After reading carefully the ffmpeg h files (e.g., avformat.h), it is more clear what these different options mean. I suggest the following plan:

1. I add code with av_setfield or something similar in src/util.jl to fix Windows Issues #25 and Get available pixel formats for camera device #31 (and for any other requests that might come up soon). For Windows Issues #25, I think apAVInputFormat[1] should contain all the options (including formats) which is passed to :iformat in AVFormatContext, but I will check again if this correct and how to use it later to actually set the format.
2. In the meantime, I will work on the AVOptions API with direct library calls as you suggest. Given as you say that the structs and fields in ffmpeg may disappear even with minor bug fixes, it makes a lot of sense to talk directly to the library API rather than creating immutable types in Julia and passing them around. I was actually wondering whether so many immutable declarations could be avoided. . .
   Cheers, Max

[kmsquire]

Sounds like a plan!

[maxruby]

@kmsquire. By the way, I think we should include in the VideoIO.jl file:
using Images
using ImageView
I understand that it may slow down the first time you import VideoIO, but it seems at least right now essential to play videos and do anything with the images captured. Anything against this?

[kmsquire]

The current situation was actually requested, to reduce startup times for
those who don't need or don't want to use those packages (e.g., someone
doing batch processing or using GLPlot).

I haven't followed all of the module related changes recently, however, so
there may be a middle ground, where, e.g., we can pass parameters when
loading a module. But I don't know if that functionality was merged yet.

Cheers,
Kevin

On Tuesday, September 30, 2014, Maximiliano Suster notifications@github.com
wrote:

@kmsquire https://github.com/kmsquire. By the way, I think we should
include in the VideoIO.jl file:
using Images

using ImageView

I understand that it may slow down the first time you import VideoIO, but
it seems at least right now essential to play videos and do anything with
the images captured. Anything against this?

—
Reply to this email directly or view it on GitHub
#27 (comment).

[timholy]

Even I can understand why you might want to support running without Images and ImageView 😄.

That said, the best solution is to make loading fast, and then make use of these two packages. I'm keeping a close eye on (and mildly helping out with) https://github.com/vtjnash/Speed.jl.

[maxruby]

@kmsquire. In addition to the av_opt_set and av_opt_get AVOptions we can also use the AVDictionary structure and set key,value pairs by calling av_opt_set_dict() on either AVFormatContext or AVCodecContext. One particular case where this is crucial is when we don´t know the input file format, i.e., before the file is actually opened. I found the AVDictionaryEntry declaration in livabutil_h.jl, but the declaration of AVDictionary fields in dict.c is not present in VideoIO. I would like to try to use AVDictionary (and it should not be difficult to use) - so we can have a a more global solution.

I can not find a declaration of AVDictionary like this in the AVUtil module, is there a reason for this?
Sorry for my ignorance.

   type AVDictionary               # struct in dict.c
      count::Int32
      elems:Ptr{AVDictionaryEntry}
   end

Cheers,
Max

[kmsquire]

Hi Max, making AVDictionary accessible sounds like a good plan. The reason that the type doesn't exist is precisely because it's defined in dict.c, and not in any of the header files. In the header files, it's just available as typedef struct AVDictionary AVDictionary;, which means its implementation is not part of the external api, and it's meant to be used as an opaque structure or pointer. So, all operations on AVDictionaries (creation, adding and removing keys/values, etc.) should be done using the public APIs.

Actually, I think AVDictionary, not AVOption is what I was thinking of when I proposed the dictionary-like interface above. So I'm glad you found and proposed to wrap it. ;-)

[maxruby]

Thanks for the explanation. Yes, I have used AVDictionary in VideoIO for the simple case above, used av_dict_set() and passed the AVDictionary to avformat_open_input. The AVOptions API also relies on key, value pairs like AVDictionary but has more functionality (e.g., we can use av_opt_find to search all the formats, codecs, etc in the AVFormatContext or AVCodecContext structures). And we can also use AVDictionary as input to av_opt_set_dict. . .

[kmsquire]

Hi Max, I'm on vacation this week and have limited internet access. I'll
take a look when I get the chance, but it possibly won't be until Sunday or
Monday.

On Wednesday, October 8, 2014, Maximiliano Suster notifications@github.com
wrote:

@kmsquire https://github.com/kmsquire. I have completed a draft version
of the AVOptions API wrapper for VideoIO.jl which should contain most of
the functionality needed to support AVOptions (I still need to do testing
and correct/refactor code). I tried to follow as closely as possible the
original ffmpeg library documentation.

Issues #25 #25 and #31
#31 should be addressed by
these new AVOptions functions. Currently, they are packaged in a single
file named avoptions.jl.

However, I would like to make sure that getting/setting options with
AVOptions functions is integrated in such a way that it works smoothly with
the current flow in avio.jl. In other words, it only makes sense to get/set
devices before we call opencamera -> AVInput(), and setting formats
(AVFormatConext) and codecs (AVCodecContext) of the input frames should be
done when calling
open_avinput(avin, source, input_format)

For example, avformat_open_input can take AVDictionary as
user_options = Dict{String,String}()

—
Reply to this email directly or view it on GitHub
#27 (comment).

[maxruby]

No problem, have a nice holiday!

I have finished the AVOptions wrapper and I will be testing it over the next couple of days. It has support for all essential functions with AVDictionary, AVDeviceCapabilitiesQuery. It is all contained in a new file called avoptions.jl. Hopefully I will have polished it further by Monday when you get a chance to look into it. Meanwhile, can you please suggest what is the best way for me to test the new module together with the VideoIO.jl module? In other words, how should I update the Package just for testing/debugging?

Thank you.
Max

[kmsquire]

I think this is what you're asking; if not, let me know.

What you'll want to do is create an test/avoptions.jl file, with simple tests that exercise the new functionality. Then you'll want to modify test/runtests.jl to include this file.

You'll probably want to move using Base.Test from test/avio.jl to test/runtests.jl, and you should use the test macros in test/avoptions.jl. See test/avio.jl (or the test directory in most other packages) for examples.

When you submit the PR, automated (Travis) testing runs test/runtests.jl and indicates whether or not the tests pass. This file is also run by @IainNZ's automated testing framework to see how well packages are tracking changes in mainline Julia. (VideoIO is currently broken in this regard because we haven't yet updated to the most recent ffmpeg libraries.)

Thanks again for taking this on. My intention is to add you as a collaborator in this repo after reviewing the PR.

[maxruby]

Thanks for the info. I will test thoroughly the new functions with test/avoptions.jl and get back to you as soon as I see the tests pass. . .

[maxruby]

@kmsquire. I am sorry for the delay - I wished to have this done by today, but I also had to spend time on a C++/OpenCV project. I will try to get back to you latest end of the week.

Is it possible at all to run my modified version of the VideoIO package this way?
julia > Pkg.rm("VideoIO")
-> include("avoptions.jl") in my modified VideoIO.jl
julia > "include("path/to/VideoIO.jl")
julia > using VideoIO

Then test any new functions in the REPL?

Max

[maxruby]

It seems it should be possible to rapidly test changes by adding files/modifying my ~/.julia/VideoIO folder and then:
reload("VideoIO")
Followed by
require("../src/VideoIO.jl")
include("../test/avoptions.jl")

Or do you have an alternative suggestion for rapid testing Pkg module scripts without committing changes/PR to github?

[kmsquire]

No worries, Max.

Regarding testing, you can run package tests with Pkg.test("VideoIO"). I'm not sure how this works if you change things. reload("VideoIO") between tests might work, but I haven't tested it.

In general, reload will work unless using VideoIO was called--in that case, you'll probably need to restart the REPL.

workspace() is supposed to help here by giving you a new Main module, but it's noisy when you the reload something, and I've found it often doesn't work as I need it to. But you might try it as well.

Sorry not to be able to give a canonical answer here!

[maxruby]

I have two questions now:

1. I may have read about this issue already somewhere before, but Pkg.test("VideoIO") gives errors even before I load my modified VideoIO package, e.g.,

julia> Pkg.test("VideoIO")
.... downloaded several files ....
Testing file reading...
Testing annie_oakley.ogg...
ERROR: test failed: img == first_frame
in error at error.jl:21
in default_handler at test.jl:19
in do_test at test.jl:39
in anonymous at no file:37
in include at ./boot.jl:245
in include_from_node1 at ./loading.jl:128
in include at ./boot.jl:245
in include_from_node1 at loading.jl:128
in process_options at ./client.jl:285
in _start at ./client.jl:354
in _start at /Users/maximilianosuster/julia/usr/lib/julia/sys.dylib
while loading /Users/maximilianosuster/.julia/v0.3/VideoIO/test/avio.jl, in expression starting on
line 19
while loading /Users/maximilianosuster/.julia/v0.3/VideoIO/test/runtests.jl, in expression starting on
line 1

failed process: Process(/Users/maximilianosuster/julia/usr/bin/julia /Users/maximilianosuster/.julia/v0.3/VideoIO/test/runtests.jl, ProcessExited(1)) [1]

However, if I simply use the example in your tutorial, VideoIO.playvideo("anne_oakley"), this works fine and I can see the video in the display window.

The fatal "ERROR: test failed: img == first_frame" refers to the macro @test
@test img == first_frame

I suppose this is to do with the swapping of .ogg to .png extension (which you do elegantly with swaptext)? which is indicated in the file. I would like to know if this is the only problem with the current version of Pkg.test("VideoIO") before I go on using this approach.

2. How can I load my fork of your VideoIO repo (now containing avoptions.jl) into the Julia environment and use it instead of the VideoIO package in the ./julia directory?

[kmsquire]

Regarding 2, the easiest way is to do the actual fork and development in .julia/v0.x/VideoIO. Alternatively, you can add the directory above your development version of VideoIO to LOAD_PATH:

julia> unshift!(LOAD_PATH, ".")
3-element Array{Union(ASCIIString,UTF8String),1}:
 "."                                                        
 "/Users/kevin/Source/julia/usr/local/share/julia/site/v0.4"
 "/Users/kevin/Source/julia/usr/share/julia/site/v0.4"      

Regarding the first issue: I'm not sure what the exact issue is. I'm running on a recently updated system, and I need to update VideoIO for the libraries available first.

For the tests, I previously had extracted the first (non-blank) image from each video. These exist as .png files in the test directory. The tests simply skip past any blank frames, read the first real image from each video, and compare those to the previously extracted images.

It may be that something is being read differently by a recent version of ffmpeg. I'll try to take a closer look once I update the VideoIO libraries.

[maxruby]

Simple testing doesn´t look straighforward . . .

I tried so far

1. changing, adding/modifying files, then reload and require/include in the REPL, but I get BinDeps errors related to init.jl so I am stucked at the very first step.

2. forking my latest VideoIO repo with AVOptions (originally forked from your master repo) to my /.julia/v0.3 directory (which is essentially identical your master repo but with my avoptions.jl file).
   Then I tried "using VideoIO" in Julia but errors came up again with BinDeps (see below).

3. Pkg.clone("VideoIO.jl.git my URL")
   I get the following error
   ERROR: could not open file /Users/maximilianosuster/.julia/v0.3/VideoIO/src/./deps/build.jl
   in include at ./boot.jl:245
   in include_from_node1 at ./loading.jl:128
   in include at ./boot.jl:245
   in include_from_node1 at ./loading.jl:128
   in include at ./boot.jl:245
   in include_from_node1 at ./loading.jl:128
   in reload_path at loading.jl:152
   in _require at loading.jl:67
   in require at loading.jl:51
   while loading /Users/maximilianosuster/.julia/v0.3/VideoIO/src/init.jl, in expression starting on line 9
   while loading /Users/maximilianosuster/.julia/v0.3/VideoIO/src/VideoIO.jl, in expression starting on l
   line 5

I am wondering whether there is a simple way to set up /.julia/v0.3/VideoIO with my modified VideoIO version without these type of complications with BinDeps and/or paths.

Sorry for the trouble.

[maxruby]

ok, I will just work directly with the files in /.julia/v0.3 and try to write over them (same as 1 above but using your original master obtained via the Pkg manager). I think this is the simplest since it preserves the original configuration of the folder/files.

[maxruby]

Good news is that I have several of the AVOptions functions tested now and they seem to be working.
However because C (unlike my favourite C++) does not have templated functions, it will be more tedious to work with all the individual av_opt_set functions to set values for image, pixel_format, etc. . . av_opt_get should be easier to handle. I am not sure if there is any workaround this? At least with av_opt_set input strings (value) are parsed to int or whatever based on the key string. Anyway, I will keep working on it and hopefully wrap it up by the end of the week.

[kmsquire]

I would start smile to get things working--maybe just focus on the string
versions for now. I think it's unlikely these need to to have high
performance.

That said, if the names are somewhat regular, it might be possible to use
meta programming to create them. The metaprogramming section in the docs
has more info, and there are some examples scattered throughout Base.

On Wednesday, October 15, 2014, Maximiliano Suster notifications@github.com
wrote:

Good news is that I have several of the AVOptions functions tested now and
they seem to be working.
However because C (unlike my favourite C++) does not have templated
functions, it will be more tedious to work with all the individual
av_opt_set functions to set values for image, pixel_format, etc. . .
av_opt_get should be easier to handle. I am not sure if there is any
workaround this? At least with av_opt_set input strings are parsed to int
or whatever based on the key string. Anyway, I will keep working on it and
hopefully wrap it up by the end of the week.

—
Reply to this email directly or view it on GitHub
#27 (comment).

[lucasb-eyer]

I would start smile to get things working

I often try that too 😄

[maxruby]

I know this is extremely basic but setting with Dictionary should also hopefully work soon too.
😄

julia> using VideoIO

julia> f = opencamera("Built-in iSight", VideoIO.DEFAULT_CAMERA_FORMAT)

julia> av_set_options_as_strings(f.avin, "probesize", "100000000")
probesize set to 100000000.

julia> av_set_options_as_strings(f.avin, "analyzeduration", "1000000") 
analyzeduration set to 1000000.

[maxruby]

@kmsquire. I now have some AVOption functionality for listing, finding and setting option values using key, value string pairs. For example, we can get a complete list of all options available that can be set to defaults or individually using the AVOptions API and like this:

julia> using VideoIO
julia> f = opencamera("Built-in iSight", VideoIO.DEFAULT_CAMERA_FORMAT)
julia> OptionsDictionary = avopt_view_all_options(f.avin)

------------------------------------------------------------------------------------------------------------------------
       LIST OF ALL OPTIONS FOR FORMAT, CODEC AND DEVICES
------------------------------------------------------------------------------------------------------------------------
aggressive                  =>  min: -2.147483648e9 , max: 2.147483647e9
analyzeduration             =>  min: 0.0 , max: 9.223372036854776e18
audio_preload               =>  min: 0.0 , max: 2.147483646e9
auto                        =>  min: -2.147483648e9 , max: 2.147483647e9
avioflags                   =>  min: -2.147483648e9 , max: 2.147483647e9
avoid_negative_ts           =>  min: -1.0 , max: 2.0
bitexact                    =>  min: 0.0 , max: 0.0
bitstream                   =>  min: -2.147483648e9 , max: 2.147483647e9
buffer                      =>  min: -2.147483648e9 , max: 2.147483647e9
careful                     =>  min: -2.147483648e9 , max: 2.147483647e9
chunk_duration              =>  min: 0.0 , max: 2.147483646e9
chunk_size                  =>  min: 0.0 , max: 2.147483646e9
compliant                   =>  min: -2.147483648e9 , max: 2.147483647e9
correct_ts_overflow         =>  min: 0.0 , max: 1.0
crccheck                    =>  min: -2.147483648e9 , max: 2.147483647e9
cryptokey                   =>  min: 0.0 , max: 0.0
direct                      =>  min: -2.147483648e9 , max: 2.147483647e9
disabled                    =>  min: -2.147483648e9 , max: 2.147483647e9
discardcorrupt              =>  min: -2.147483648e9 , max: 2.147483647e9
err_detect                  =>  min: -2.147483648e9 , max: 2.147483647e9
experimental                =>  min: -2.147483648e9 , max: 2.147483647e9
explode                     =>  min: -2.147483648e9 , max: 2.147483647e9
f_err_detect                =>  min: -2.147483648e9 , max: 2.147483647e9
f_strict                    =>  min: -2.147483648e9 , max: 2.147483647e9
false                       =>  min: 0.0 , max: 0.0
fdebug                      =>  min: 0.0 , max: 2.147483647e9
fflags                      =>  min: -2.147483648e9 , max: 2.147483647e9
flush_packets               =>  min: 0.0 , max: 1.0
formatprobesize             =>  min: 0.0 , max: 2.147483646e9
fpsprobesize                =>  min: -1.0 , max: 2.147483646e9
frame_rate                  =>  min: 0.1 , max: 30.0
genpts                      =>  min: -2.147483648e9 , max: 2.147483647e9
igndts                      =>  min: -2.147483648e9 , max: 2.147483647e9
ignidx                      =>  min: -2.147483648e9 , max: 2.147483647e9
ignore_err                  =>  min: -2.147483648e9 , max: 2.147483647e9
indexmem                    =>  min: 0.0 , max: 2.147483647e9
keepside                    =>  min: -2.147483648e9 , max: 2.147483647e9
latm                        =>  min: -2.147483648e9 , max: 2.147483647e9
list_devices                =>  min: 0.0 , max: 1.0
make_non_negative           =>  min: -2.147483648e9 , max: 2.147483647e9
make_zero                   =>  min: -2.147483648e9 , max: 2.147483647e9
max_delay                   =>  min: -1.0 , max: 2.147483647e9
max_interleave_delta        =>  min: 0.0 , max: 9.223372036854776e18
metadata_header_padding     =>  min: -1.0 , max: 2.147483647e9
nobuffer                    =>  min: 0.0 , max: 2.147483647e9
nofillin                    =>  min: -2.147483648e9 , max: 2.147483647e9
noparse                     =>  min: -2.147483648e9 , max: 2.147483647e9
normal                      =>  min: -2.147483648e9 , max: 2.147483647e9
output_ts_offset            =>  min: -9.223372036854776e18 , max: 9.223372036854776e18
packetsize                  =>  min: 0.0 , max: 2.147483647e9
pixel_format                =>  min: 0.0 , max: 2.147483647e9
probesize                   =>  min: 32.0 , max: 2.147483647e9
rtbufsize                   =>  min: 0.0 , max: 2.147483647e9
seek2any                    =>  min: 0.0 , max: 1.0
skip_initial_bytes          =>  min: 0.0 , max: 9.223372036854776e18
sortdts                     =>  min: -2.147483648e9 , max: 2.147483647e9
start_time_realtime         =>  min: -9.223372036854776e18 , max: 9.223372036854776e18
strict                      =>  min: -2.147483648e9 , max: 2.147483647e9
true                        =>  min: 0.0 , max: 0.0
ts                          =>  min: -2.147483648e9 , max: 2.147483647e9
use_wallclock_as_timestamps =>  min: 0.0 , max: 2.147483646e9
video_device_index          =>  min: -1.0 , max: 2.147483647e9

Setting options is as simple as:

julia> av_set_options_using_strings(f.avin, "frame_rate", "15")
frame_rate set to 15.

julia> av_set_options_using_strings(f.avin, "pixel_format", string(VideoIO.AV_PIX_FMT_UYVY422))
pixel_format set to 17.

julia> av_set_options_using_strings(f.avin, "analyzeduration", "10000000")
analyzeduration set to 10000000.

julia> av_set_options_using_strings(f.avin, "probesize", "100000000")
probesize set to 100000000.

However, when trying to retrieve values from an opaque pointer to AVDictionary or AVDeviceInfoList, e.g., avdevice_list_devices, I encouter repeatedly an "assertion" error (av_assert0) of AVDeviceInfoList** (goes back to line 187 of libavdev or libavformat.c). Any thoughts on what we could do to check where the problem originates?

For example,

julia> avdev_list_devices(f.avin)
Assertion device_list failed at libavdevice/avdevice.c:187

signal (6): Abort trap: 6
 __pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
 Abort trap: 6

Here is the avdev_list_devices function:

function avdev_list_devices(I::AVInput)

   # Select Ptr{AVFormatContext}
   pFormatContext = I.apFormatContext[1]

  # Initialize a Ptr{Ptr{AVDeviceInfoList}}
   pplist = Ptr{Ptr{AVDeviceInfoList}}[C_NULL]
  device_list = pplist[1]

   if (avdevice_list_devices(pFormatContext, device_list)<0)
     #avdevice_free_list_devices(device_list)
     error("Cannot list devices")
  else
    #get AVDeviceInfoList
    pdevice_list = unsafe_load(device_list)
    infolist = unsafe_load(pdevice_list) 
    #number of autodetected devices 
    nb_devices = infolist.nb_devices 
    #index of default device or -1 if no default 
    default_device =  infolist.default_device

    # get AVDeviceInfo
    pDeviceInfo = unsafe_load(infolist)
    DeviceInfo = unsafe_load(pDeviceInfo)
    # get device_name 
    device_name = bytestring(DeviceInfo.device_name)
    device_description = bytestring(DeviceInfo.device_description)
    println("|",device_name,"|"," "^10, "|",device_description,"|");

   #deallocate
   #avdevice_free_list_devices(device_list)
  end
end

[kmsquire]

Hi Max, nice! I think the error is because part of the struct needs to be initialized before passing it in to the function call. (See my comments in the relevant bug report.). For example, when calling this function through ffmpeg on the command line, one of the properties is initialized to "dummy". I don't know whether this is just part of the ffmpeg front end or if it's intrinsic to the function, but I suspect the latter.

Why don't you submit this as a PR, marking it "WIP" so it isn't merged until it's ready?

[maxruby]

OK. I created a PR but please keep in mind that this is still very rough and I need to do a lot of tidying up, presenting the documentation and adding as much functionality for AVDictionary/DeviceCapabilityQuery as possible. I look forward to making this as useful as it can be.

[maxruby]

Can you tell me which "comments in the relevant bug report" you refer to? I saw one where you comment on the dummy at the command line, but I am not so much wiser after reading this . . . I understand that AVDeviceInfoList must be initialized - so I will try again assigning the fields in AVDeviceInfoList (devices, nb_devices, default_device) appropriately and hopefully solve this too.

[kmsquire]

Thanks! The PR lets me (and others) test, comment on the design and offer
suggestions. Anything you push to the original branch is visible. It's
nice to do this earlier rather than later, and the "WIP" says not to merge
it.

On Friday, October 17, 2014, Maximiliano Suster notifications@github.com
wrote:

OK. I created a PR but please keep in mind that this is still very rough
and I need to do a lot of tidying up, presenting the documentation and
adding as much functionality for AVDictionary/DeviceCapabilityQuery as
possible. I look forward to making this as useful as it can be.

—
Reply to this email directly or view it on GitHub
#27 (comment).

[kmsquire]

Nope, that was it. Maybe this was something I was thinking about when
trying to debug this and didn't actually write.

On Friday, October 17, 2014, Maximiliano Suster notifications@github.com
wrote:

Can you tell me which "comments in the relevant bug report" you refer to?
I saw one where you comment on the dummy at the command line, but I am not
so much wiser after reading this . . . I understand that AVDeviceInfoList
must be initialized - so I will try again assigning the fields in
AVDeviceInfoList (devices, nb_devices, default_device) appropriately and
hopefully solve this too.

—
Reply to this email directly or view it on GitHub
#27 (comment).

[maxruby]

avopt_get_value() is also fixed now! (line 271 in avoptions.jl)
I had a lesson in how to pass double pointers correctly from Julia to C :)
Maybe this will help me correct the rest of the problems I encountered. . .

[maxruby]

Lots of progress since last night - most things are now working (both getting and setting) . . .
I will try to update in a couple of days.

[kmsquire]

Excellent!

On Saturday, October 18, 2014, Maximiliano Suster notifications@github.com
wrote:

Lots of progress since last night - most things are now working (both
getting and setting) . . .
I will try to update in a couple of days.

—
Reply to this email directly or view it on GitHub
#27 (comment).

[maxruby]

@kmsquire. I can document all AVOptions, make a Dict and print all of them, find, get and set AVOptions with strings and with the dictionary API. We can also get pixel formats compatible with the device with a new function called get_pixel_formats() in avio.jl. Interestingly, I had to change the declaration of AVDictionary for this API to work (it can not be declared simply as typealias Void in Julia).

These functions work:
document_all_options,
print_options,
is_option,
set_default_options,
set_option,
get_option,
create_dictionary,
set_options_with_dictionary,

However, I am struggling to get avdevice_capabilities_create() to work for the Device Capabilities API. It gives a negative (-78) error no matter how I initialize the AVDeviceCapabilitiesQuery structure. I am not sure whether there is something about this structure that needs to be initialized differently. Any thoughts?

Here is my code for for initializing AVDeviceCapabilitiesQuery

# Create a new device query structure
function create_device_query(I::AVInput, pDictionary)

  # Retrieve Ptr{AVFormatContext}
  pFormatContext = I.apFormatContext[1]
  FormatContext = unsafe_load(pFormatContext)

  pStream = unsafe_load(FormatContext.streams)
  stream = unsafe_load(pStream)
  codecContext = unsafe_load(stream.codec) #Ptr{AVCodecContext}
  frame_width, frame_height = codecContext.width, codecContext.height
  fps = AVRational(codecContext.time_base.den, codecContext.time_base.num)

  # Initialize a device query structure
  DeviceQuery = AVDeviceCapabilitiesQuery (FormatContext.av_class, #Ptr{AVClass}
                                       pFormatContext,      #device_context::Ptr{AVFormatContext}
                                       AV_CODEC_ID_MPEG4,   #codec::AVCodecID
                                       AV_SAMPLE_FMT_NONE,  #sample_format::AVSampleFormat
                                       AV_PIX_FMT_YUYV422,  #pixel_format::AVPixelFormat
                                       cint(0),            #sample_rate::Cint   
                                       cint(0),            #channels::Cint
                                       convert(Culonglong,0),#channel_layout::Int64  
                                       cint(640),         #window_width::Cint
                                       cint(480),         #window_height::Cint
                                       frame_width,            
                                       frame_height,
                                       fps)


   queries = avdevice_capabilities_create[pointer_from_objref(DeviceQuery)]
   #queries = Array(Ptr{AVDeviceCapabilitiesQuery},1) 
   queries[1] = C_NULL
   #pDictionary[1] = C_NULL

   # Can also set pDictionary => C_NULL
   if (avdevice_capabilities_create(queries[1],pFormatContext,pDictionary) < 0) 
     error("Can not create a device query. Try again")
   end

   #avdevice_capabilities_free(queries, pFormatContext)
   return queries
 end

[maxruby]

When trying to implement av_dict_get which is necessary for get_metadata, it turns out that it requires AVUTIL ver >= 53. Since we only have v52 it can not be implemented.

  AVDictionaryEntry* av_dict_get (FF_CONST_AVUTIL53 AVDictionary * m, 
  const char * key,
  const AVDictionaryEntry * prev,
  int flags 
  )     

[kmsquire]

I'm working on updating this now.

[kmsquire]

If you update to the latest version, AVUtil v54 should be available for ffmpeg, at least. I haven't updated the libav side of things (not sure what they've released recently).

[maxruby]

Thats ok. I have enough for now with ffmpeg. It should be relatively straightforward to implement the AVOptions in libavutil once everything works with ffmpeg. Thanks.

[kmsquire]

Of course, that doesn't make it easy to support earlier versions of ffmpeg/libav.

[maxruby]

I just need to solve the errors with AVDeviceCapabilitiesQuery and then I will be very happy. . .