EXT_property_animation extension proposal

[najadojo]

No description provided.

[donmccurdy]

Thanks for opening this!

I appreciate that your approach is infinitely flexible, but I'm worried that comes at the risk that there will be no complete implementation. E.g. there are some things we cannot animate in three.js without re-compiling a shader, and if we can't animate at 60-90fps we're probably not going to ship it.

Personally I'd feel better about including an explicit list of properties that may be animated, and adding things to that list after determining that they make sense and can practically be animated. This would make it vastly easier to test conformance. I realize we haven't established whether extensions should use versioning, which might be important here, but let's not have a logistical hurdle get in the way of designing this robustly.

[najadojo]

Would you help me out by describing what types of properties would need recompilation? I've attempted to avoid this particular issue by limiting the data types and not allowing index object references.

I'm not apposed to having this extension enumerate all the properties that can animate including those in the current accepted and draft extensions. Going forward extensions could reference this spec and indicate which properties are allowed. If there is a way to avoid this level of detail I think we'd benefit from it.

[donmccurdy]

A few likely-to-be-problematic examples that would seem to be allowed as written:

• accessors.byteOffset, .min, .max, .count
• accessors.sparse.count
• bufferView.byteStride
• buffer.byteLength
• extras.*
• material.*.texCoord
• node.matrix
• much of BLENDER_physics
• KHR_techniques_webgl uniform.count and uniform default values
• KHR_texture_transform offset/scale/rotation
  • This example is one I would like to see work, but (AFAIK) won't in three.js today

This feels like enough examples that new extensions are commonly going to run into special cases. What about a whitelist of properties in the core spec, and a provision that current and future extensions may "opt in" with their own whitelist? It does not necessarily need to be an exhaustively maintained whitelist. For example, material/* excluding texCoord would cover every case I care about in the core spec.

[lexaknyazev]

Is there any use of matrix interpolation?

[lexaknyazev]

At the current state of glTF 2.0 core spec, we could say that animation of all integer properties must be disallowed. This would cover the list above.

[lexaknyazev]

Empty lists are forbidden by the current schema. Why not just re-use the existing path property (since the schema allows arbitrary strings there)?

[najadojo]

I updated the sample such that it doesn't fall into this problem. We can't use the path as it exists because its defined as an enum value with only the four existing supported values.

There may be assets that wish only to animate some property values but not TRS or weights. Suggestions welcome for schema change or a best practice note for an identity animation that would go in core spec channels.

[lexaknyazev]

The schema allows any string in addition to pre-defined four values.

glTF/specification/2.0/schema/animation.channel.target.schema.json

Lines 27 to 29 in 09a404d

	{
	"type": "string"
	}

[najadojo]

It does but the validator and more than one engine makes this a strict enum value. Plus this schema requires a node reference which isn't required with the JSON Pointer.

[lexaknyazev]

Node reference is not required

glTF/specification/2.0/schema/animation.channel.target.schema.json

Line 35 in 09a404d

"required": [ "path" ]

The latest validator revision (wip) no longer treats unknown enum values as errors.

The documentation about extensions explicitly says that they can extend allowed enums.

[lexaknyazev]

I see how it may work although this distinction complicates promotion a bit (so schemas must be remade for minor version updates).

Looking again at the animation example above, it seems that target isn't referenced by any other property but inlined into channel, so it's neither case, right?

[bghgary]

schemas must be remade for minor version updates

I'm not sure what you mean. What schema changes are you talking about?

Looking again at the animation example above, it seems that target isn't referenced by any other property but inlined into channel, so it's neither case, right?

target is being referenced through animations[n].channels[m].target. Basically, any enum that will be parsed and interpreted by an implementation loading the core spec falls into case 2.

[lexaknyazev]

What schema changes are you talking about?

Updated properties reside in the extension object for now. If/when the same functionality goes into 2.x core version, the schema will be different. That would complicate implementations willing to support all combinations of core versions / extensions.

target is being referenced through animations[n].channels[m].target.

I'd say that target is contained within channel object, not referenced by it. So an engine may transitively treat the whole channel object as "unknown" as well as animation. Given that there're no external (indexed) references to any of animation-related objects, I don't see much difference with top-level image with updated mimeType.

[bghgary]

I don't see much difference with top-level image with updated mimeType.

I suppose it depends on implementation. If an implementation is a top-down loader, then I see a difference. Images referenced only by an extension will never be loaded if the extension is never read and thus the types in the image can be extended without fear of existing implementations falling over.

[lexaknyazev]

I suppose it depends on implementation.

Exactly. Given that the spec allows an asset to have only, e.g., images collection, we can't assume that an arbitrary implementation won't try to load them in advance unless we want to incorporate "top-down" approach into spec's compatibility strategy (which may be a good idea after all).

The existing schema prevents us from extending animation.channel.target with an extension object because animation.channel.target.path is required. With versioning concerns in mind, this means that the safest way of designing the extension schema is to use a new top-level collection.

[lexaknyazev]

I'd propose to provide a full list of allowed accessor types for each animatable core spec property.

[lexaknyazev]

Must all keyframe values comply with limitations specific to the animated property (e.g. min/max for material factors)?

[donmccurdy]

I had this initially but the engines I looked at need to collect and clone the materials as if the path implies the channel should only effect the single mesh primitive and not others that use the same material reference.

Engines already have to solve that; e.g. a material may be reused by a skinned and non-skinned mesh, meaning you need different shaders and so must collect and clone the materials (at least this is how it works in three.js).

I'm a bit worried about this... now if a top-level material is targeted we need to track down the nodes that use it, and possibly clones of the material created e.g. because of skinning. The three.js animation system generally targets nodes and their subproperties, and is based closely on Unity's so I'm guessing the situation is similar there.

Reusing materials is also important for an application, and having the authoring tool split identical materials in order to target them with different animations isn't ideal. On the other hand, if materials are reused it is possible to modify uniforms on a per-object basis while rendering. (This may be more complex for the client, but is achievable and likely to be worthwhile.)

I don't think easier implementation is a sufficient reason for targeting top-level objects with animation. It feels less natural conceptually, and the fact that it loses information about where materials could otherwise be reused worries me.

Top-level approach leaves the whole material targeted everywhere its used;

Potentially across multiple scenes, also?

[najadojo]

Extending Animations

Property animations can be added to an animation by adding the EXT_property_animation extension to any glTF animation. For example, the following defines an animation with two channels that modify the baseColorFactor and roughnessFactor of a mesh's material.

"animations" : [
    {
        "channels" : [ ... ],
        "extensions" : {
            "EXT_property_animation" : {
                "channels" : [
                    {
                        "sampler" : 0,
                        "target" : {
                            "node" : 3,
                            "path" : "mesh/primitives/0/material/pbrMetallicRoughness/roughnessFactor"
                        }
                    },
                    {
                        "sampler" : 1,
                        "target" : {
                            "node" : 3,
                            "path" : "mesh/primitives/0/material/pbrMetallicRoughness/baseColorFactor"
                        }
                    }
                ]
            }
        },
        "samplers" : [
            {
                "input" : 6,
                "interpolation" : "CUBICSPLINE",
                "output" : 7
            },
            {
                "input" : 8,
                "interpolation" : "LINEAR",
                "output" : 9
            }
        ]
    }
]

EXT_property_animation adds a channels list separate from the core specification's such that a different set of target values can be described.

target

The target object identifies the node or scene to animate using a node or scene property. Which property of the node or scene is identified by path.

path

In a typical engine at runtime glTF is transformed from a series of JSON object lists into a connected graph of engine objects. To represent the runtime instancing inherent in such a transform EXT_property_animation specifies animation binding at their runtime location.

For the following glTF:

{
    "materials" : [
        {
            "pbrMetallicRoughness" : {
                "baseColorFactor" : [
                    1.0,
                    0.0,
                    0.0,
                    0.2
                ],
                "roughnessFactor" : 0.0
            }
        }
    ],
    "meshes" : [
        {
            "primitives" : [
                {
                    "attributes" : {
                        "NORMAL" : 2,
                        "POSITION" : 1
                    },
                    "indices" : 0,
                    "material" : 0
                }
            ]
        }
    ],
    "nodes" : [
        {
            "mesh" : 0
        }
    ]
}

EXT_property_animation will interpret as if it were operating on the transformation:

{
    "mesh" : {
        "primitives" : [
            {
                "attributes" : {
                    "NORMAL" : 2,
                    "POSITION" : 1
                },
                "indices" : 0,
                "material" : {
                    "pbrMetallicRoughness" : {
                        "baseColorFactor" : [
                            1.0,
                            0.0,
                            0.0,
                            1.0
                        ],
                        "roughnessFactor" : 0.0
                    }
                }
            }
        ]
    }
}

Transformation follows each indexed object reference (i.e. "mesh": 0) and replaces it with the referred object and continues descending.

path is a JSON Pointer in the transformed JSON starting from the node or scene object and ending at a leaf property value. A path in the above document:

mesh/primitives/0/material/pbrMetallicRoughness/baseColorFactor

Both mesh and material are indexed object references that are followed from the original JSON. Circular references are acceptable in the transformed JSON but a path must always takes the shortest path from root to leaf.

This was a previous iteration description having node/scene based targeting. I think both approaches have their downsides; clarity of effect and difficulty of implementation in one place or another.

It'd be best to have a target description that is explicit where we can describe "everything with this material" or "just this instance of this material" but I've not encountered an animation system that is this descriptive. In three.js' animation system when targeting node.material.color if the material object is shared among multiple nodes the color changes on all even though that wasn't the target described.

The three.js proof of concept I did not attempt to collect up objects that would need to have clones based on an animation. Would the algorithm be simply if material is part of any animation assign a unique clone it unless it was previously cloned? It seems like it would be better to group them based on the key data and top level target (i.e. three blinking lights on my jet model, should be one animation on one material).

[scurest]

Perhaps it's worth advising what a producer should do for an animation that doesn't have any regular glTF channels since it's not immediately obvious. Current syntax requires a non-empty animation["channels"]. You can however use

"channels" : [
  {
    "sampler" : 0,
    "target" : {
      "path" : "scale"
    }
  }
]

because the spec says that channels without target["node"] properties should be ignored. (Both 0 and "scale" are arbitrary; scale is just the shortest path name).

edit: ah, I see this was discussed above.

[scurest]

The property need not be present in the original JSON; default or implicit properties can be animated.

Does this extend to animating KHR_texture_transform properties on a texture info object that doesn't already use KHR_texture_transform?

[najadojo]

Yes I think the language of the extension allows that. Though supplying a requires or uses extension should be necessary as those properties are undefined without the target extension.

[lexaknyazev]

@najadojo @bghgary @donmccurdy

Could you please elaborate a bit on the positioning and the life-cycle of this extension? I understand that this extension has EXT prefix so it can be defined and implemented as vendors wish. However it could become a de facto way of animating materials or lights much sooner than we could come up with a KHR extension or a new spec version.

I think that at some point, we'll want to have something very similar to this extension in glTF core.
The current fallback design is clearly not optimal and imposes additional restrictions like existence of "node ": 0 and an unused sampler with two accessors. The question whether definitions of certain objects are animated or only their instances (like materials, cameras, or lights) should still be open even if it falls outside of the scope of this extension for now.

Moreover, I'm still concerned about (from comments above):

An existing implementation will fail to load if a new path is now part of the channels list.

and

It also seems node is not optional for existing engines.

since such behavior doesn't follow the schema.

[najadojo]

This extension was a proposal for something we were thinking about internally, unfortunately we are no longer perusing those plans. I started with EXT because I was able to prototype functionality in multiple engines and tools. There has been pretty good engagement in the discussion for this PR but I don’t believe enough to solidify the proposal. It may make the most sense to archive this PR until future additions to glTF core are underway.

[donmccurdy]

It also seems node is not optional for existing engines.

since such behavior doesn't follow the schema.

Added this as a suggestion for KhronosGroup/glTF-Asset-Generator#414.

I think that at some point, we'll want to have something very similar to this extension in glTF core.

I think it would be useful to have a general syntax for extensions to use when adding animatable properties (the syntax in this PR seems reasonable, also see #1346) and if that requires changes to glTF core at some point we should consider it. But I do still have reservations about making things animateable-by-default, and would prefer that we consider allowing animation property-by-property as part of any new extension. For example, KHR_lights_punctual2 might add animation of punctual lights.

[lexaknyazev]

KHR_lights_punctual2 might add animation of punctual lights.

That's something I'd avoid, because it will likely lead to unmanageable fragmentation.

[donmccurdy]

@lexaknyazev could you explain further? In the long term, my opinion is that when we add any extension where animation is likely to be needed, that extension should define which of its properties can be animated, using syntax that is (a) part of glTF core, or (b) not in core but at least consistent across extensions. I only suggest KHR_lights_punctual2 here on the assumption that it's too late to add animation to KHR_lights_punctual.

Are you concerned about having extensions define animation, having inconsistent channel target syntax, or something else?

[donmccurdy]

/Bump. I think it would be helpful to have a general plan for (eventually) supporting additional animated properties. Example:

1. ???_property_animation extension is created, defining a syntax for targeting properties with animation, and listing properties in the existing spec and existing extensions for which implementations of the extension must support animation.
2. Future extensions should use the syntax defined by ???_property_animation to specify animation, if animation makes sense for that extension. For example, a hypothetical KHR_lights_area extension should declare which of its properties may be animated.

In retrospect, it would have been useful to have an animation syntax for arbitrary properties in place from the beginning, so that extensions like KHR_lights_punctual and KHR_texture_transform specs could include animation in a pre-defined way.

[donmccurdy]

A few more things I'd like to track here:

• Per Consider Using a more generic referencing system #1518, using JSON pointers (or some other generic reference system) seem like a step in the right direction.
• Per Should an animation channel be able to target multiple nodes at once? #1520, consider allowing an animation channel to have multiple targets.
• Consider allowing animation of individual morph targets. E.g. meshes/{}/weights/{} or nodes/{}/weights/{}.

[emackey]

To turn off visibility, it may be worth calling out that scale is allowed to become 0, 0, 0, and implementations are welcome to optimize this by turning off rendering of the entire child-nodes tree (as there's no recovering visibility from scale zero by any child nodes).

Implementations that don't support this optimization will still do the right thing when all the geometry becomes degenerate. This is better than for example rasterizing a ton of fragments that all fail an alpha test, as opacity animation would do.

[donmccurdy]

so are you implying that we will need a node extension for visibility just like khr_texture_transform was added for material?

No, I'm not suggesting an extension. Just wondering what the options are. There are other animations that would ideally be supported that aren't suitable to JSON pointers (e.g. animate one morph target at a time). Perhaps we can have animation shorthand for them without adding new JSON schema for the features.

implementations are welcome to optimize this by turning off rendering of the entire child-nodes tree

It seems like hack to support zero-scale, degenerate matrices throughout an engine just to disable a node tree, when engines already have visibility as a feature. 😕

Another option is to move the node (far) out of the probable camera frustum.

[scurest]

There are other animations that would ideally be supported that aren't suitable to JSON pointers (e.g. animate one morph target at a time).

Why aren't JSON pointers suitable for animating one morph target? As you said above, it'd just be /nodes/{}/weights/{}, right?

[emackey]

Another option is to move the node (far) out of the probable camera frustum.

Let's not do this. Some viewers will adapt the default view to include the entire scene, including distant nodes. And some renderers, including Cesium, will show much of the inner Solar System, making it unclear how far one would have to translate a node to get out of the way.

Scaling to zero isn't elegant, but it works even without hacky optimizations, and even on under-powered late-1990's era hardware (ehh... probably best not to ask me how I know this... 😅 )

[donmccurdy]

Why aren't JSON pointers suitable for animating one morph target? As you said above, it'd just be /nodes/{}/weights/{}, right?

Oops, yes that might be fine.

Scaling to zero isn't elegant, but it works even without hacky optimizations...

I wouldn't want to prescribe this as the official way to toggle visibility in glTF, but yeah it might be good enough. We could also introduce one (last?) pre-defined path, like the glTF 2.0 ones...

"target": {
  "node": 0,
  "path": "visibility"
}

... and use JSON pointers for everything else. I don't know what I prefer.

[bghgary]

Would visibility be a boolean or a float value?

[Jaswant99]

Would visibility be a boolean or a float value?

Accessor componentType does not allow boolean.

[bghgary]

I don't mean the actual type on disk. Another way to ask, is visibility expected to be (0 and 1) or (0 to 1)?

[donmccurdy]

I think 0 or 1.

[bghgary]

I updated the comment above to indicate that Babylon.js has an implementation of this extension from @najadojo. It is a bit out-of-date as it was implemented a while back.

[Jaswant99]

Here is a POC in Babylon.js I did to animate KHR_texture_transform properties using EXT_Property_animation proposal https://github.com/Jaswant99/Babylon.js/tree/ext_prop_anim_POC .

[elpie89]

Hi, I'm from Asobo Studio, I'm thinking to implement this in FlightSimulator
we need to add the possibility to animate material property

"animations" : [
    {
        "channels" : [
            {
                "sampler" : 2,
                "target" : {
                    "node" : 0,
                    "path" : "scale"
                }
            }
        ],
        "extensions" : {
            "EXT_property_animation" : {
                "channels" : [
                    {
                        "sampler" : 0,
                        "target_property" : "/materials/1/pbrMetallicRoughness/albedo"
                    },
                    {
                        "sampler" : 1,
                        "target_property" : "/nodes/0/mesh/primitives/0/material/pbrMetallicRoughness/roughnessFactor"
                    },
                    {
                        "sampler" : 2,
                        "target_property" : "/mesh/primitives/0/material/pbrMetallicRoughness/metallicFactor"
                    },
                    {
                        "sampler" : 3,
                        "target_property" : "/mesh/primitives/0/material/pbrMetallicRoughness/CUSTOM_EXTENSION/customValue"
                    }
                ]
            }
        },

what do you think?

[donmccurdy]

@elpie89 this extension is meant to allow animation of material properties, yes. However, it is just a proposal at this stage. EXT_property_animation is not implemented in current glTF tools. If you need a workflow within your own tooling, you may want to use it with a ASOBO_ prefix instead.

[arpu]

would be very helpful for UV scrolling/offset for gif like animations

[Alexeyt89]

Hello,
I'm working on an automotive project, for which I need to implement light animations. The format already supports Light sources with these properties: "intensity", "color", "innerConeAngle", "outerConeAngle". It would be nice to have the possibility to animate these properties. I use Blender to create animated models and on exporting models to glTF only "translation", "rotation" and "scale" animations are exported.

[donmccurdy]

This is a very high-level comment on an idea with a lot of moving parts, but I wonder if a some mechanism for procedural textures (see #1889) might not be a better alternative to the (difficult to fully scope, define, and implement?) approach we've been working on here. Procedural textures with vertex displacement potentially allow a much broader range of use cases (see Houdini VAT: https://github.com/keijiro/HdrpVatExample) than JSON pointers to specific glTF properties, and are easier to normatively define — see the open OSL implementations of MaterialX Standard Nodes.

One case this does not cover would be animation of lights, but that's much easier to define in a simple standalone extension.

[UX3D-nopper]

Merged this into #2033
Would this be an alternative?

[aaronfranke]

This is fully superseded by the glTF Object Model and KHR_animation_pointer, so this PR should be closed.

As for the discussion of node visibility, this is covered by KHR_node_visibility, which is close to complete.