Binary data Parameters

[fxtech]

Open Effects Proposal for Standard Change

Please read the contribution guidelines first.

Standard Change Workflow

• Create proposal as issue (you're doing this now!)
• Tag this issue with standard change tag
• Identify subcommittee: at least one plug-in vendor, and at least one host
• Discuss the idea in this issue
• Write new or updated code and doc
• Publish updates as a pull request (ideally on a feature/PROPOSAL-NAME branch)
  • Make sure that PR references this issue number to keep them in sync
  • Discuss and review code in the PR
  • Meet all requirements below for accepting PR
• When subcommittee signs off and other members don't have any further review comments,
  maintainer merges PR to master which closes PR and issue

Requirements for accepting a standard change:

• Header files updated
• Documentation updated
• Release notes added
• Compatibility review completed
• Working code demonstrated with at least one host and one plugin
• At least two members sign off
• No further changes requested from membership

Summary

There is currently no way to have parameters that store opaque binary.

Motivation

Currently, opaque binary data stored as a Custom parameter type must be encoded as a null-terminated string,
which can add overhead for encoding/decoding, as well as adds some ambiguity to hosts which know nothing
about what is in the data (is it binary or just a null-terminated string?), and therefore must also encode/decode
the value when saving their projects.

Problem

See Motivation above.

Impact

A new Parameter type (kOfxParamTypeBytes) is required, along with an associated OfxBytes struct for storing the relevant information.
If a host doesn't support kOfxParamTypeBytes, it can return kOfxStatErrUnsupported from paramDefine(), at which point a plugin can always fall back to using a traditional Custom parameter.

Documentation Impact

Descriptions of the new definition and struct must be written as part of the header documention, in addition to a sample.

Stakeholders

Both plugin writers and host vendors will benefit from this, since Bytes parameters will not need to go through potentially two extra encoding/decoding steps, thus speeding up hosts and plugins and reducing project data size.

Discussion

[MrKepzie]

See my comment in #142 (comment) with a PropertySuiteV2 I wrote 3 years ago that does the same

[MrKepzie]

Note that in my case, the plug-in "defined" metadata (e.g: a media reader), thus needed to create properties on its own.

[fxtech]

I prefer Alexandre's term of "Bytes" instead of "BinaryData"

[fxtech]

For Bytes parameters, we can avoid adding to or changing the API by doing it like this. Any comments?

#define kOfxParamTypeBytes "OfxParamTypeBytes"

typedef struct OfxBytes
{
    const uint8_t *data;
    size_t length;
} OfxBytes;

// query the bytes data of a parameter by passing a pointer to a OfxBytes instance:
OfxBytes data;
paramSuite->paramGetValue(param_handle, &data);
// now use the data

[garyo]

For Bytes parameters, we can avoid adding to or changing the API by doing it like this. Any comments?

#define kOfxParamTypeBytes "OfxParamTypeBytes"

typedef struct OfxBytes
{
    const void *data;
    size_t length;
} OfxBytes;

// query the bytes data of a parameter by passing a pointer to a OfxBytes instance:
OfxBytes data;
paramSuite->paramGetValue(param_handle, &data);
// now use the data

That seems simple enough. Is the ownership model the same as Strings (up to the next API call, so plugin must copy, as in https://openfx.readthedocs.io/en/main/Reference/ofxCoreAPI.html#strings)?

[fxtech]

Yes, that is what I was thinking. The plugin could do its own decoding directly off that data pointer if it's done within the same call.

[fxtech]

I removed some discussion around supporting a Bytes property type in the general sense.
That concept will be moved into the Metadata discussion regarding a Bytes metadata type.

[revisionfx]

Is this animatable? Often per-frame data.
Who owns the memory (for destuction)....?
Does param set value just pass new data, how is lenght updated?

[garyo]

As for animation, we can decide whether we'd like it to animate like custom params, using a custom callback similar to OfxCustomParamInterpFuncV1, or be non-animatable. Per the spec, Customs and Strings may be animatable, depending on the host. So we should probably do the same for Bytes.

As for memory, should be the same as string params (and this update should make that explicit to avoid any confusion). This was already discussed above.

As for length, if I understand the proposal correctly, that's what the OfxBytes struct is for -- data pointer + length. See @fxtech 's comment above.

[revisionfx]

doc note: it is highly recommended to use the Memory Suite for larger allocations to give a chance to host memory management wise.

The plugin is responsible to allocate and then to destroy that memory and reset pointer to null on action DestroyInstance
Host has pointer and sizeof (so could double check also :) )

Special Issue: Some hosts that have undo caching. We don't want each value change to trigger a sync private data etc
Does host check pointer value to create a prop reason (plugin has changed)? If so a plugin for paramSetValue should create new memory (new pointer), copy whatever they need and destroy old memory when changing.?(kOfxParamPropEvaluateOnChange) or is calling paramSetValue enough to force Evaluation - instance changed... say one just allocated a large fixed size table and just changed some data in it? Then that would not undo but fault of plugin.

Does this parameter if length is not 0 triggers a requirement to sync private data (save project including auto-save) - as is expected for custom data param maybe if one has set PropPersistent on that parameter?

[garyo]

The plugin is responsible to allocate and then to destroy that memory and reset pointer to null on action DestroyInstance
Host has pointer and sizeof (so could double check also :) )

Unless you're suggesting something different than usual, my understanding is that when the plugin calls paramSetValue, the host must copy the data it got from the plugin (not just the struct, but the pointed-to memory). The plugin may then free its copy if it wants. And when the plugin calls paramGetValue, it must copy the value it receives from the host (same: not just the struct, but the pointed-to memory). The spec says that data is only valid until the next API call, so the host might overwrite or free its copy right afterward. Do you agree?

So there is no data sharing here. If the plugin changes some values within the pointed-to memory, the host will never see those changes.

[revisionfx]

The spec says that data is only valid until the next API call, so the host might overwrite or free its copy right afterward. Do you agree?

Which specs? If I set to Persistent, I expect the host to hold that data, right? If you mean I have to use GetValue each time (e.g. each frame render) to get it back, ok. And you say that then I should after setValue like at exit of render call free that memory as the host will have copied it otherwise I will create a memory leak :)

[garyo]

Spec location ("Strings"): https://openfx.readthedocs.io/en/main/Reference/ofxCoreAPI.html#strings

Persistance just means the host will save & reload that param when the project is reopened. It's the default. It has nothing to do with sharing of data between plugin and host during a run.

As with any param value, I would expect you have to get it each frame during render().

And what the plugin does with its own copy of that param value data is up to it. Of course it should free whatever it allocates. (But for instance, a plugin might just copy the param bytes into a known fixed location, or send them to another process, or pass them to a function that processes them right then. Totally up to the plugin.)

[fxtech]

The Bytes param type really works exactly like String or Custom, just with a length.

[garyo]

The Bytes param type really works exactly like String or Custom, just with a length.

This makes sense to me. The major difference between String and Custom is that Customs have an animation callback kOfxParamPropCustomInterpCallbackV1. If we want Bytes to be animatable, this should probably include a similar kOfxParamPropBytesInterpCallbackV1 that the host calls in the same way as for customs.

I wonder how many hosts implement this callback property for customs though?

(It would be nice perhaps if String params had such a callback, for consistency. But that's another discussion.)

[revisionfx]

A use case is per frame data - say it's 50K per frame, and the clip is 1000 frames, do we want to transmit 50 Mb each render loop?
Also for custom data we need host to call sync private data, I presume this is needed too for this too (for save, and auto-save)?
We could also just assume/set NULL for per frame with gaps, 0 string length - KF.

[garyo]

It's just a param, like a string or custom. No sync private data is needed; it's not private data. The host should save & load it just like any other param. Am I missing something? SyncPrivateData is just telling the plugin to sync its private data into params. If you need that to be called to get your private data into a custom or string param today, you'll need to do the same for the Bytes param. But that doesn't affect this spec change.

As for your per-frame data, I assume you'll be calling paramSetValueAtTime(...) on it -- the host should save all values and give you the proper frame's data when you call paramGetValueAtTime(...) just like for any param. I'm not sure who would be transmitting 50Mb and to what? The host will have to store your 50Mb of data in its project file of course, and load it up when restoring the project, so that might get slow, but per frame it should just give you that frame's data. Just like a Custom or String param today.

[fxtech]

My use-case for Bytes is purely for binary "sequence data", like Mocha or Silhouette project data when running as a plugin, or for custom Lens Flare data, or anything else that is binary and "big". I wasn't thinking in terms of per-frame. I know Silhouette doesn't support custom param animation callbacks.

[garyo]

I see, I may have misunderstood Pierre. Pierre, perhaps you're saying you plan to store a single blob of data (not a value per frame, and no param animation expected), and this would be 50MB (it might represent an array of 1000 frames worth of data, but the API doesn't care about that). In that case yes, you'll have to memcpy the 50MB each frame, and similarly if you change it the host will have to memcpy it.

So should we just say that Bytes params are non-animatable, full stop, just to keep things simple? Any other host folks care to chime in on this?

[fxtech]

I don't think it would be a huge burden to make them animatable. They currently are NOT in Silhouette, but it would be possible. If we mandate that String, Custom and Bytes params should be able to be animatable then I can add that.

[garyo]

@fxtech there's a host param kOfxParamHostPropSupportsStringAnimation (& same for customs, and choice and booleans). I expect you return false for that.
The spec says that strings, customs, choice and booleans can animate, if the host allows it:

The following may animate, depending on the host. Properties exist on the host to check this. If the host does support animation on them, then they do not animate by default. They are…

kOfxParamTypeCustom ...

[revisionfx]

I don't think we need an animation callback for that. Just regular KF indexing (previous or equal...) could be useful. In that case a UI for that in KF editor could be a bit like an animated boolean and that would be ok. All one might want is to move time location, delete and copy-paste.

BTW another host that returned not animatable for choice just fixed it for next version. Almost cleared all hosts not supporting choice animation :) - down to 1 of 15 I think.

[garyo]

I don't think the spec is clear on what happens if the host returns kOfxParamHostPropSupportsCustomAnimation = true but then returns an error if the plugin tries to set the animation callback property. Just previous-or-equal as you suggest, Pierre? We could clarify that as part of this PR I suppose.
We could not allow the animation callback for Bytes params, just previous-or-equal as Pierre suggests. That's probably easiest for hosts and good enough for plugins, right? In a way, we'd be stealthily deprecating the custom animation callback which it seems like is not well supported (?)

[revisionfx]

Forgive me if I remember wrong, but I believe the original intent for custom param interact was for folks who wanted to have custom UI interaction in their effects controls. I never tried to implement that in OpenFX.

[garyo]

original intent for custom param interact

Not sure what you're talking about here? As far as I know there's no interacts for custom params, at least not in this context.
Are you talking about this? kOfxParamPropInteractV1 I don't think that has anything to do with this spec change.

[revisionfx]

Slightly drifting OT but Just saying way back then as we were listing all other API functionalities, this was also related to display custom UI in parameter control.

We at some point also said we could use the Overlay suite in there too (still documents to use OpenGL), but seems it would likely need a background texture addition (and is likely 8bit display)?
As many use Qt, it might be appropriate for that to have a special optional Qt widget for that (what RG suggested at some point when I discussed with them, e.g. displaying a color wheel).

https://github.com/AcademySoftwareFoundation/openfx/blob/main/Examples/Custom/custom.cpp
kOfxParamHostPropSupportsCustomInteract

https://openfx.readthedocs.io/en/main/Reference/ofxParameter.html
OfxStatus() OfxCustomParamInterpFuncV1 (OfxParamSetHandle instance, OfxPropertySetHandle inArgs, OfxPropertySetHandle outArgs)
"The interp value is a linear interpolation amount, however his may be derived from a cubic (or other) curve. "
I believe this was meant to display in curve editor a curve drawing - has anyone ever used that?

If no one implements this anyway, would likely be better to just use standard KF suite for all non numeric values.
And specialize custom parameter for task such as what I just described...

[garyo]

OK, understood -- let's try to keep this discussion topical. This is the official history of this spec change. :-) Further OT discussion to Slack please.
We just need to nail down the proposal:
(1) Should Bytes be animatable at all (protected by a host prop like String/Custom/etc.)?
(2) If so, should Bytes support a param interp function like Custom?

My vote is yes and no in that order, so it would be like String. In that case getParamValueAtTime returns the current-or-previous value for hosts that allow animation.

[revisionfx]

"In that case getParamValueAtTime returns the current-or-previous value for hosts that allow animation."

um... it returns data = NULL and length = 0 if you request AtTime where there is no KF/data maybe. It just uses the KF indexing as normal to find if 1) there is KF (num of it) and 2) where are the adjacent to current time ones... one can always get two if this implies some interpolation. In this case unlike choice, the only impact of not supporting anim is the possible size of transmitting back this at every frame render...

[garyo]

it returns data = NULL and length = 0 if you request AtTime where there is no KF/data maybe.

Is that what String and Custom params do? I think we should try to be consistent. (Except for host bugs of course!)

[revisionfx]

I never requested animated String and Custom here so I don't know
Nor do I know what host supports animating that... what you suggest should be OK for paramGetValue if this is legit to call that in render. It's fine for paramGetValueAtTime to return something as you suggest although for this as this is not a single value it's likely I would always use the KF interface to index which frame to use.

[fxtech]

I wasn't originally thinking of making this animatable, but it probably should be for parity with Custom (this is meant to be a Custom alternative). I imagine the main use will be a single blob of data used for the whole instance (ie. analysis data, lens-flare description, etc).
If it is animated, I would think it should use a default "hold" interpolation type, so if there are keys at 1 and 10 and you query 5, you get 1's data. Otherwise you'd always have to go query all the keys yourself.
Thoughts on this?
Is animation overkill, considering it's intended use? And if we support animation, shouldn't we also support interpolation like Custom?

[fxtech]

We agreed that Bytes animation is required, and the host will use step interpolation to derive values. There will be no OfxParamHostPropSupportsBytesAnimation property

[fxtech]

In order to provide a default value for Bytes, I'll need to rev PropertySuite and provide OfxBytes property support.
What if we disallow default values for now?

[garyo]

What if we disallow default values

100%. Since there's no UI, there's no need for plugin-specified default values. Empty byte array should be the default.

[revisionfx]

default is NULLPTR, 0

[fxtech]

PR #161

[revisionfx]

OK timely, we just tested custom data and about half the hosts at least don't support per frame data.

[fxtech]

I'll update the sample plugin to handle missing host support for Bytes parameters.
Phil asked that I also add code to test very large binary data support.

[barretpj]

The sample should also try setting and verifying (e.g. after save and relaunch) a value with null bytes followed by data, to ensure the host isn't erroneously using string functions on the param value.