-
Notifications
You must be signed in to change notification settings - Fork 323
OpenFX plugin programming guide (Basic introduction)
- 1st draft 08/24/2014 Alexandre Gauthier-Foichat
- Refined with Pierre Jasmin notes on 02/07/2014
This guide is intented for confirmed C++ developers who want to develop OpenFX plug-ins using the Natron fork of the OpenFX standard. Note that this fork is 100% compliant with the original repository and using this guide ensures that your plug-in(s) will work across all hosts.
First off, the main official specification should be followed very precisely when in case of doubt. This document is 100% true and answers almost all questions one might have. However this is a long document which can be quite hard to swallow the first time one has to read it.
I’ll skip all the details regarding packaging of OpenFX pug-ins and the philosophy and focus this document on the programming of OpenFX plug-ins using the Support layer. For an extensive explanation of the details of how the bundle should be setup, please refer to "Packaging OFX Plug-ins" in the official OFX specification.
OpenFX is in under the hood just a protocol for a plug-in and a host application to help communicate.
They do so with a C API using blind handles and properties which are uniquely identified by names, which all begins with kOfx*. The reason why C language is used is because the C++ API is sensitive to compiler versions.A plugin compiled using the base C API will work fine in Natron, this is about documenting the C++ wrapper to develop plugin for Natron.
To help using this API, the official guys from the OpenFX association have made a C++ wrapper around it so it is easier to use. The wrapper on the plug-in side is called the Support layer and the one on the Host side (the application, e.g: Natron) is called the HostSupport layer.
The official repository of the OpenFX association is actively maintained though their C++ layers contain bugs and missing implementations. That’s why the Natron dev team has forked it and continue maintaining it and fixing bugs whilst also incorporating new features of the newer versions of OpenFX.
The official repository of the OpenFX association can be found here: https://github.com/ofxa/openfx
The repository the Natron dev team maintains can be found here: https://github.com/devernay/openfx
On this repository you will find the 2 layers mentioned above in separate folders with their respective names:
- The Examples directory contains OpenFX plug-in examples which were programmed using directly the C API (thus not using the Support layer).
- The Support folder contains a Plugins folder which contains other examples. These examples are on the other hand programmed using the Support layer.
The Natron dev team has 2 separate repositories for its OpenFX plug-ins:
- openfx-io is to handle all plug-ins which do input/output operations and rely on external libraries (such as OpenImageIO, OpenColorIO,FFMPEG,OpenEXR, etc…).
- openfx-misc is for all other plug-ins which do image processing. They do not require linking to any other external library and are generally easier to compile and understand.
Depending on what kind of plug-in you implement you should base your work upon one of the 2 repositories above as they contain “state of the art” OpenFX plug-ins. They use the OpenFX API as one should do.
If you were to create a new reader plug-in to read whatever you need that isn’t supported already, I suggest that you fork openfx-io and derive the GenericReader class which does all the quite complex handling of what a fully-featured reader plug-in is expected to do (such as downscaling, color-space transformation). Same for writer plug-ins, I suggest you derive the GenericWriter class.
Note that originally, Readers and Writers are not part of the OpenFX standard, and to implement the I/O plug-ins, Natron is using the TuttleOFX Reader/Writer context extension. As we will see later a plugin specifies in which context it can run.
If on the other hand you were to write an image processing plug-in, I suggest you fork the openfx-misc repository and look at the plug-ins inside as examples. The Crop and Invert plug-ins are trivial plug-ins which should give you a fair understanding on how OpenFX works.
Each plug-in can be instantiated in different context by an application depending on its use.
They are well described in the OpenFX specification and won’t go through them here.
http://openfx.sourceforge.net/Documentation/1.3/ofxProgrammingReference.html#ImageEffectContexts
The OpenFX spec. defines functions that the host can call on the plug-ins to make it perform special actions. These functions are called actions and their specification is well described here:
http://openfx.sourceforge.net/Documentation/1.3/ofxProgrammingReference.html#id473661
The most important action to implement in general is the render action which is called when a plug-in needs to render its image. We will detail a bit more the important actions in the chapter dedicated to the plug-in object.
The plug-in can also call some functions on the host to query some informations. These functions are grouped in “suites“. Generally the type of things you would like to ask is “fetch that image”, “fetch that image’s size”, etc…
The idea of suite is to facilitate evolution, Over time a suite can be versioned and a version supported from an API version.. As such the Support Layer matches an API version.
A C++ plug-in is composed of 3 objects in the general cases:
The factory object which is used to instantiate the plug-in to the host application at the moment the binary is loaded.
The plug-in object which is used to communicate with the host application and do some work. Generally the processing is not done in this object, rather we do it in the last object…
The processor object which is used to do the processing in an optimised way: OpenFX offers a way to do multi-threading easily using this class.
The last class is not mandatory and one could also do the rendering in the plug-in class, though it would not be multi-threaded.
It basically serves 3 purposes:
- Instantiating the plug-in object (the Create Instance action)
- Declaring some parameters/clips (the Describe In Context action)
- Declaring some properties of the plug-in to the host application (the Describe action)
An instance of our plug-in class is created in the factory's createInstance function.
The plug-in parameters and clips are declared in the factory's describeInContext function.
The parameters are what the user could (generally can, but not if they are hidden/disabled) interact with in the user interface. They are several types of parameters and they are quite well described in the OpenFX specification. What you can control is whether the user can animate them, their name, default value, etc… Please check out the examples in the openfx-misc for implementation details.
The clips are the objects that refer the input images (in Natron a clip is the arrow between 2 nodes). This is a view of the plug-in of another input plug-in. The plug-in must always define an output clip: This is where the output image of this plug-in will be defined.
Plugin properties regarding the usage by the host application are declared in the factory's describe function. This is where the plug-in defines its name, which bit depths it can support,, whether it can support multi-resolution images, whether it will need fetching images at different times, whether it supports interlaced images, whether it supports multi-threading…. etc
See the invert plug-in describe function for an example of the function.
All properties defined in the describe function are well covered by the OpenFX specification though some need some extra caution:
-
kOfxImageEffectPluginRenderThreadSafety : This must be carefully set. If misused, then your plug-in might not be thread-safe.
Unsafe means that all instances of the same plug-in can only have 1 render-thread at once: they will all be synchronised.
InstanceSafe means that an instance can only have 1 render thread at a time, though several instances do not need special synchronisation.
FullySafe indicating that any instance of a plug-in can have multiple render threads running simultaneously.
On top of that the plug-in can also set the property kOfxImageEffectPluginPropHostFrameThreading : When set to 1, and if the thread-safety of the plug-in permits, the host will slice the render window and call it with several threads instead of calling the render function with only 1 thread.
Bear in mind that you as a plug-in do not need to set this to 1 since you can use the multi-thread suite and do the multi-threading yourself. This suite also includes locking abilities allowing you to properly control the thread-safety of your plug-in.
-
kOfxImageEffectPropSupportsMultiResolution : When set to 1 that means your plug-in is expected to work with arbitrary image rectangles in input and output and they may not necessarily be the same.
-
kOfxImageEffectPropSupportsTiles : When set to 1 that means your plug-in is expected to work with images that are not the “full image” but just a sub-rectangle of the full image. Any per-pixel process is a candidate for this
The following behavior is expected:
- If a plugin returns an error during
kOfxActionLoadit should prior to returning clean up any global memory it will have allocated using the OFX memory suite. The plugin will then not show in the UI. - If a plugin returns an error during Describe it expects the host to call
kOfxActionUnloadand the plugin will not show up in the UI. - In all other cases plugin should release everything during the
kOfxActionUnload. Otherwise during the system dynamic library destruction the OFX handles will probably be gone.
To create a group of plug-ins that will all show up under the same menu in the user interface, you need to set the kOfxImageEffectPluginPropGroupingproperty that is a string containing the menu group under which you want your plug-in appear, e.g: “Transform” or “Color/Transform”
This is the main object allowing to bridge with the application. When implementing your plug-in you will want to derive the OFX::ImageEffect class. There are a bunch of virtual functions which are the actions the host application can call. (The functions I mentioned above).
All communication occurs via an action handler which the mainEntryStr function in ofxsImageEffect.cpp.
The constructor of the plug-in is here to fetch a pointer to all the parameters/clip you defined previously in the describe function of the factory. These pointers represent the “instantiated” version of those parameters/clip, whereas in the describe function well…you just described them so the host would instantiate them correctly. This is with those pointers that you can get/set values and query informations.
The output clip (sometimes called dstClip_ in our examples) represents the output image.
The source clip represents the source image and this is from this object that you need to fetch the input image.
If you were to have several input clips, then you would fetch the input images from each of your source clips.
Some clips can be optional (such as a mask for example) and needs to be set so explicitly in the describe function of the factory.
In the following I will vaguely re-explain the main actions that are generally implemented by a plug-in. Though for more detailed explanation and how to report errors in these functions please check the OpenFX specification which does a full coverage on them.
virtual bool
isIdentity( const IsIdentityArguments &args,
Clip * &identityClip,
double &identityTime );This function must return true if the effect in its current state will not apply any change to the source image. This is called by the host to determine whether a call to the render action is necessary or not.
When true then the rendering pipeline is much faster as the host just skips this plug-in from the compositing tree.
The identityClip parameter must be set to a pointer of the input clip of which the effect is an identity of.
The identityTime parameter must be set to the time at which the effect is an identity of the input clip.
For example a gain effect whose “Scale” parameter would be set to 1 would be an identity.
virtual void
changedParam( const OFX::InstanceChangedArgs &args,
const std::string ¶mName );This function is called every time a parameter is changed. This function can be called either because you set the value of a parameter programmatically or because the user interacted with the parameter.
the args.reason parameter will inform you from what this function was called.
For example if you had a button parameter, when the user would press it, it would call this function with the args.reason == eChangeUserEdit.
This function is also a great place to show/hide and enable/disable other parameters according to special values of another parameter.
This function could be used to implement analysis effects (such as a tracker). Fetching an image is allowed in this action.
This is also where parameter writing is supposed to be done. For example one can fetch an image, find the most popular color and write it in the parameter, and the next render action will have that parameter value updated. Data can also be stored in the InstanceData Pointer, but they would have to be saved during the syncPrivateData action.
virtual bool
getRegionOfDefinition( const OFX::RegionOfDefinitionArguments &args,
OfxRectD &rod );This is called by the host to determine the size of the image (or region of definition) produced by this effect.
If your plug-in doesn’t apply any geometrical transformation to the image, then it is probably not modifying it’s size (e.g: an Invert plug-in doesn’t modify the image’s region of definition.) In that case you do not need to implement this function, this is the default behaviour to return the region of definition of the source clip.
On the other hand, a crop effect would return the size of the crop area as its region of definition.
virtual void
getRegionsOfInterest( const OFX::RegionsOfInterestArguments &args,
OFX::RegionOfInterestSetter &rois );Even though the name is close to the getRegionOfDefinition action, it doesn’t serve the same purpose at all!
This function is called by the host before the render action is called. This is called when it wants to pre-render the input images this effect might need. In order to do so, the host needs to ask this effect what is the rectangle of the source image we’re interested in. This is the purpose of this action. In general if your plug-in does need exactly the render window then you don’t have to implement this function. On the other hand, a blur plug-in might have to include the border padding to the region of interest of the input clip.
virtual void
getClipPreferences( ClipPreferencesSetter &clipPreferences );You rarely need to implement this action. This action is called by the host to allow the effect to perform modifications of the state of the clip, such as its premultiplication (is the image premultiplied or not?) or the image components (is it alpha, RGB or RGBA?) or image bit depth (is it byte, short or float ?)
For example the shuffle plug-ins uses it when the users chooses different output components or different bit depth.
virtual void
render( const OFX::RenderArguments &args );This is where the processing must be done. The args contain several parameters that define how the image should be rendered.
time: The time at which the render is taking place. This can be used to fetch the input images at the same time or at other times.
renderScale: When different than 1 this informs that the image is rendered at a lower resolution than the full resolution. For example this is used in Natron when the user zooms out. As a filter plug-in, this is merely an hint and doesn’t hold much value. On the other hand for a reader effect (such as the ones in openfx-io) this is clearly stating at which scale you should read the image. In this case you should explicitly downscale the image yourself (ONLY FOR READER PLUG-INS!!). If you don’t support downscaling the images, then in all the actions called by the host, you’re expected to check the render scale parameter and throw a kOfxStatFailed exception if the render scale is different than 1. This can in turn inform the host that you don’t support the downscaling of the images and it will take care of it for you. Make sure to also add this check to the various other action handlers that take renderScale as argument (such as getRegionOfDefinition() and isIdentity()). See the RunScript plug-in for example.
renderWindow: This is the portion of the image to render. If you specified that you support tiles in the describe function of the factory then it might not be the full region of definition of the image.
To fetch input images, call the fetchImage on the source clips at the desired time. You’re expected to check whether the input clip is connected before fetching the image (i.e: call getConnected() on the clip). A clip is connected in Natron when the arrow is connected to another effect.
If you cannot use the input image for any reason (bad bit depth, bad components, etc…) then your expected to throw a significant exception to indicate that the render failed (kOfxErrBadFormat).
You can only fetch an image during the render action or the changedParam action
Generally a plug-in is better if it can handle arbitrary bit depths and image components. To deal with that in our plug-ins we template the internal render function by the components and the bit depth.
For example in the Invert plug-in of the openfx-misc repository, the render function just instantiate the templated class ImageInverter with the good template parameters depending on the bit depth and the image components.
In this example we use the processor object (ImageInverter) to do the rendering because it enables the multi-threading offered by the host, but we could clearly do the processing in the render function, though, it wouldn’t be multi-threaded (unless we would have set the kOfxImageEffectPluginPropHostFrameThreading property to 1).
virtual bool
getTimeDomain( OfxRangeD &range );This action is called by the host to figure out what is the frame range over which the plug-in has an effect. For instance a Reader plug-in would return the length of the image sequence, or a rectangle generator could return a firstFrame-lastFrame pair.
Note that in most cases, the default implementation is fine, which is to have effect over the union of ranges of the input effects.
virtual void
getFramesNeeded( const OFX::FramesNeededArguments &args,
OFX::FramesNeededSetter &frames );If you were to write a plug-in that needs images at different times to do its processing, e.g: a Retiming plug-in that would need the image at T – 1 and T + 1 in order to produce the image at frame T, you need to implement this action.
This is called by the host to figure out what image you will need to render so it can pre-render them. You should in turn specify exactly what images will be fetched with the fetchImage(…) function in render.
Note that by default, when implementing a simple effect such as Invert, you only need the source image at the current time, the default implementation handles it for you so you don’t have to implement it.
The host application can implement the multi-thread suite. Remember, a suite is a set of functions that a host can implement, offering some functionalities to the plug-in. In this case this suite is designed to offer SMP style multi-processing.
The C++ processor class is just a wrapper around this suite so that it is easier for you, as a plug-in developer, to multi-thread your processing. You would need to derive the OFX::ImageProcessor to craft your own processor.
The only function you have to override is the multiThreadProcessImages function. This is the core render function which renders an image rectangle for a single thread. You can cycle through all the examples for inspiration on how the processing is generally written, though the inner part of the pixel processing is really up to the plug-in developer.
Bear in mind that it is more efficient to get all the values from the parameters before calling the processor multithread function. In order to do that we generally fetch all the values we want in the render function (or more specifically in the setupAndProcess function) and then give it in parameter to the processor class.
The getValue function of a parameter can be quite expensive and it’s better to call it once if you can.
Same applies for the fetchImage function.