[web-animations-1] Generalized Time Values and Timelines

[majido]

tl;dr

Generalize the time value in web animation to be a dictionary instead of a scalar real number. This allows us to have rich user inputs such as touch and scroll to be timelines and drive animations enabling sophisticated interactive effects.

Background

In Web Animations, the input to animation model is a time value which is a single real number that nominally represents a number of milliseconds from some moment. AnimationTimeline is the source of the time value and there is a hierarchy of timing nodes (e.g., animation, effect) through which the value cascades down and is potentially transformed before reaching the actual animation effect.

ScrollTimeline is a proposal that extends this model by mapping a single scroll axis to time enabling a certain kinds of scroll-linked animations. This is a great direction but we feel it does not go far enough. In particular, the web animation timing model has a fundamental limitation that the effect input has to be modeled by a single dimensional variable. This makes it impossible (or very awkward) to use when the input is inherently multi-dimensional which is the case for many interesting input types.

Here are a few example scenarios where a single dimensional timeline is not sufficient:

• Some shortcomings of current single dimensional ScrollTimeline proposal:
  • Effects that depend on multiple scroll axes.
  • Effects that want to differentiate between momentum vs non-momentum scrolling.
  • Effects that need to animate based on both scroll and time dimensions e.g., hidey-bar.
• Effects that depend on state and location of multiple touch pointers:
  • Effects that need to react to addition or removal of a pointer.
  • Multi-touch direct manipulation effects that depend on knowing the position of all pointers.
• Interactive gesture effects:
  • Drag and drop and swipe effects that use a single pointer but depend on movement in two axes or the pointer velocity.
  • Rotation and scaling effects that depend on multiple pointers.

These effects are currently impossible to do using web animation model. This forces developers to implement them using requestAnimationFrame.

While AnimationWorklet helps enable fast script driven effects but without a rich multi-dimensional timeline input, it has limited applicability to such usecases. We think combined with a rich multi-dimensional timing model, it can replace many of the rAF based usecases.

Proposal

Allow time value to be a map instead of (or in addition to) a scalar real number. Current timelines with single scalar value are simply a special case of this more generic model.

typedef (double or DOMString or record<DOMString, TimeValue>) TimeValue;

interface AnimationTimeline {  
   readonly attribute TimeValue? currentTime;
};

Key Properties

• Time value shape can change, for example a PointerTimeline can produce an empty map, a map with single entry when a single pointer is active, or one with two or more entries when more pointers are active.
• Non-integral values can be part of the time value map. For example, ScrollTimeline can include a ‘phase’ enum.
• We can introduce new types of timing nodes which allow grouping or filtering of time values. For example:
  • A UnionTimeline can union the output of a ScrollTimeline and DocumentTimeline to enable effects that depend on both scroll and time.
  • A FilterTimeline that takes in PointerTimeline and returns only the “primary” pointer to be consumed with existing single dimensional effects.

Detailed Proposal

This document has more details on how this can be incorporated in the current specification, provides examples, discusses backward compatibility and some initial ideas on potential new timelines and group effects that this proposal enables.

We aim for this to be a starting point of a discussion to evaluate the merit of this idea and how best
to enable rich interactive effects as part of web animation model.

[majido]

/cc @shans @flackr @birtles

BTW, It will be great if we could have chat about this idea in the sidelines of upcoming Houdini, CSSWG in Berlin.

[majido]

We discussed this Generalized Time Values proposal in Houdini Berlin F2F. The feedback was that it is complex and perhaps requires too much change in web-animation model.

Since then we have spent more time thinking about this which has resulted to the alternative design below. The new design avoids mixing the input and time concepts and sidesteps the need to alter concept of 'time' as used by web-animation. This keeps the web-animation timing model simple while exposing multi-dimensional user input to animation worklet where it can be used for creating rich interactive effects. See motivation in above comment.

AnimationInput Proposal

Introduce a new concept, AnimationInput, which represents an input source to an animation. A single animation can have many inputs and each input produces a 'current' value.

In addition to providing a value, AnimationInput can also be explicitly subscribed to (via listen/unlisten method pair). When a worklet animation is listening to an AnimationInput, changes to the input value invalidates the animation causing it to get executed and produce output in the next animation frame.

interface AnimationInput {
   readonly any value;

   listen();
   unlisten();
}

A worklet animation constructor is updated to accept a map of inputs. These inputs are then passed to the corresponding animator constructor when it is being initialized in the animation worklet global scope.

typedef record<DOMString, AnimationInput> AnimationInputMap;

dictionary Options {
  AnimationInputMap inputs;
  any data;
};

[  Constructor(DOMString animatorName,
               (AnimationEffect or sequence<AnimationEffect>) effects,
                optional AnimationTimeline timeline,
                optional Options options)
] interface WorkletAnimation {}

Explicit Input Subscription - Listen/Unlisten

By default an input source is passively observed ¹ by the animation. The animation can switch to actively listening to the input source by calling listen() on it. Listening to an input source means that the animation gets to execute and produce output every time ² that input value changes.

Some effects only need to observe an input but don’t need to be sampled when its value has changed. For example a touch driven effect that wants to calculate the velocity of a swipe gesture needs wall time as an input source, but should only be sampled when the touch input changes. Making the subscription step explicit allows the system to distinct between observing and listening. This ensures animations are sampled only when necessary.

Other effects may choose to observe multiple input sources but listen to only one at any given time. For example a hidey-bar effect is driven by scroll when that user is actively scrolling or time when scroll ends and the bar needs to complete its animation.

[1]: Default is passive observation to favor performance.
[2]: Every time is further limited to frame rate of the system. The animation is sampled every time there is a new frame and its inputs have changed.

Non-Scalar Value

Unlike AnimationTimeline, the AnimationInput value does not need to be a scalar number. This is important for cases where the underlying source is inherently a multi-dimensional variable, where the dimensions can only be represented by e.g. an array or a dictionary.

Open Questions

1. Should AnimationTimeline be an input type?
   It is possible to make AnimationTimeline interface extend the AnimationInput interface. In this case the value will be the same as currentTime. The main motivation is to re-use pre-existing timeline to represent time as an input which is nice to do.

2. Should WorkletAnimation continue to have a timeline?
   One option is to allow worklet animations to have both timeline and inputs. While these is overlap
   between timeline and inputs, they play a different functions. Timeline allows the animation to
   participate in coordinated playback with other web-animations while input is the mean to provide
   rich input to animation. Note that that animations are allowed to have a null timeline. So it is
   possible for a worklet animation to have a null timeline and only be driven by their input (e.g.,
   for touch-driven effects). Alternatively, timeline can be explicitly unlistened in constructor.

Example Inputs

Here are a few examples of non-scalar input sources with some speculation on how they can be modeled as AnimationInput. Note that these are just ideas to show the richness of the AnimationInput model. The the main proposal is AnimationInput itself and not these concrete implementations.

ScrollAnimationInput

A scroll input; exposes both x,y, and scroll phase as a dictionary.

enum ScrollPhase {
  "active",
  "momentum",
  "idle"
};

dictionary ScrollValue {
  double x;
  double y;
  ScrollPhase phase;
}

ScrollAnimationInput : AnimationInput {
   readonly ScrollValue value;
}

PointerAnimationInput

A pointer input that can be used to drive animations based on the position of pointers on an element. The pointer input value provides a snapshot of all active pointers during a possibly multi-touch user interaction. Its value is considered changed whenever a pointer is added, removed, or changes. A pointer input should be created with a source element and it is effectively equivalent to having a passive touch event listener on that element.

dictionary PointersValue {
integer pointerCount;
DOMString identifier → dictionary PointerInfo {
    // client coordinate of the equivalent touch event
    dictionary  coordinate: {
        double x;
        double y;
    };
    dictionary radius {
        double x;
        double y;
    };
    double rotationAngle;
    double altitudeAngle;
    double azimuthAngle;
    PinterType type; // direct | stylus
}
}

PointerAnimationInput : AnimationInput {
   readonly PointersValue value;
}

GestureAnimationInput

A GestureAnimation is a higher level input that allows worklet animation to be tied to gesture interaction such as rotation, scale, translation. See for example MSGestureEvent and WebKit GestureEvent for some precedent work on exposing gestures on the web.

Here is a strawman API on how this input value can look like:

dictionary GestureValue {
  double rotation;
  double scale;
  dictionary translation {
      double x;
      double y;
  };

  dictionary velocity {
      double x;
      double y;
      double angular;
      double scale;
  };
};

GestureAnimationInput : AnimationInput {
   readonly GestureValue value;
}

[majido]

Here are two examples that showcase how the above AnimationInput construct can be used to create interactive animated effects.

Gesture Driven Image Resizer

In this example we are creating a basic image scale, rotate animation that is linked to corresponding multi-touch gestures using GestureInput.

<img id='target'>

<script>

await CSS.animationWorklet.addModule('scare-rotate-animator.js');
const target = document.getElementById('target');

// Using individual transform properties
const rotateEffect = new KeyFrameEffect(
   target, {rotate: ['rotate(0)', 'rotate(360deg)']}, {duration: 100, fill: 'both' }
);
const scaleEffect = new KeyFrameEffect(
  target, {scale: [0, 100]}, {duration: 100, fill: 'both' }
);

// Note the worklet animation has no timeline but two inputs.
const animation = new WorkletAnimation(
  'image-manipulator', [rotateEffect, scaleEffect],  null,  
  {inputs: {'gesture': new GestureInput(target)}}
);
animation.play();
</script>

registerAnimator('image-manipulator', class {
  constructor(options) {
    // Always listen to gestures.
    this.options.inputs.gesture.listen(this);
  }
  animate(currentTime, inputValues, effects) {
    // Note that currentTime is undefined and unused.

    // Get current gesture value and update rotation and scale effects accordingly.
    const {rotate, scale} = inputValues.gesture;
    effect.children[0].localTime = rotate / 100;
    effect.children[1].localTime = Math.min(scale, 100);
  }
});

Mixing Scroll and Time inputs

This example recreates twitter hidey-bar effect that uses two animation inputs: scroll and time.
The animation is only attached to time input when it is actively animating this ensures that the animation is only executed when user is actively scrolling or when the effect is in transition and does not need to be sampled rest of the time.

<div id='scrollingContainer'>
  <div id='header'>Hidey-bar header</div>
  <div>Scrolling content</div>
</div>

<script>
await CSS.animationWorklet.addModule('hidey-bar-animator.js');
const $header = document.getElementById('header');
const $scroller = document.getElementById('scrollingContainer');

const headerHeight = .clientHeight;
const scrollRange = $scroller.scrollHeight - $scroller.clientHeight;
const effect = new KeyFrameEffect($header,
  [{transform: 'translateY(0)'}, {transform: `translateY(${scrollRange}px)`}],
  {duration: scrollRange, fill: 'both' });

const scrollInput = new ScrollInput($scroller);
const timeInput = document.timeline;

// Note the worklet animation has no timeline but two inputs.
const animation = new WorkletAnimation('hidey-bar', effect,  null,  {
  inputs: {'scroll': scrollInput, 'time': timeInput}
  data : {'headeHeight': headerHeight}
});
animation.play();
</script>

const MIN_HIDE_AMOUNT = -50;
const MAX_HIDE_AMOUNT = 0;
const HIDE_SPEED = 0.35; // hide animation speed in pixel per millisecond

function clamp(value, min, max) {
  return Math.max(min, Math.min(max, value));
}

function sign(value) {
  return value < 0 ? -1 : 1;
}

registerAnimator('hidey-bar', class  {
  constructor(options) {
    this.scrollInput_ = options.inputs.scroll;
    this.timeInput_ = options.inputs.time;
    // Always listen to scroll changes.
    this.scrollInput_.listen(this);

    this.headerHeight_ = options.data.headerHeight;

    this.hideAmount_ = 0;
    this.lastY_ = -1;
    this.lastTime_ = 0;
    this.lastPhase_ = 'idle';
    this.lastHideSpeed_ = 1;
  }

  animate(currentTime, inputValues, effect) {
    // Note that currentTime is undefined and unused.

    // Scroll value is in {x, y, phase} form.
    const {x, y, phase} = inputValues.scroll;
    const time = inputValues.time;

    var currentMinHideAmount = clamp(this.headerHeight_ - y,  MIN_HIDE_AMOUNT, 0);
    
    if (phase != 'idle') {
      // When actively scrolling hide in keep with scroll amount.
      var scrollDelta = this.lastY_ - y;
      this.lastHideSpeed_ = HIDE_SPEED * sign(scrollDelta) * scrollDelta;
      this.hideAmount_ += scrollDelta;
    } else {
      // When the scroll goes idle we animate based on time
      // determine if we need to keep sliding the header.
      bool isCompleted = this.hideAmount_ == currentMinHideAmount || this.hideAmount_ == MAX_HIDE_AMOUNT;
      bool isStarting = this.lastPhase_ == 'active';

      if (isCompleted) {
        this.timeInput_.unlisten(this);
      } else {
        if (isStarting)
          this.timeInput_.listen(this);

        // Continue hide/show animation following the direction and speed of last scroll.
        var timeDelta = time - this.lastTime_;
        this.hideAmount_ +=  lastHideSpeed_ * timeDelta;
      }
    }

    this.hideAmount_ = clamp(this.hideAmount_, currentMinHideAmount, MAX_HIDE_AMOUNT);
    // Position the hidey bar relative to the current scroll amount.
    effect.localTime = y + this.hideAmount_;

    this.lastY_ = y;
    this.lastTime_ = time;
    this.lastPhase_ = phase;
  }
});

[birtles]

Thank you for this. These concrete examples really help. The idea of using inputs as opposed to blessing a particular timeline seems good to me.

[css-meeting-bot]

The Houdini Task Force just discussed [web-animations-1] Generalized Time Values and Timelines.

The full IRC log of that discussion

<emilio> topic: [web-animations-1] Generalized Time Values and Timelines
<TabAtkins> github: https://github.com//issues/2493#issuecomment-422109535
<TabAtkins> majidvp: Early on we looked at things people needed for scroll, but we early knew we needed to solve the more general use-case of interactive animated effects.
<TabAtkins> majidvp: Mobile or touch-friendly interface, you see drag-and-drop, scaling/zooming/transforming etc.
<TabAtkins> majidvp: Touch-based interactions.
<TabAtkins> majidvp: Righ tnow only way to do this is with touch-event handlers on main thread and use rAF().
<TabAtkins> majidvp: These are not time-based animations, just input-driven.
<TabAtkins> majidvp: Scroll-linked animations are a subset of these.
<TabAtkins> majidvp: Scroll is easier; it's 1d mostly.
<TabAtkins> majidvp: So you can map it to time, easy. And can use with WebAnim.
<TabAtkins> majidvp: Touch and gestures are more complex: not single-dimenstional, and are stateful.
<TabAtkins> majidvp: Almost impossible to do this declaratively in a way that's expressive.
<TabAtkins> majidvp: This is why people use the manual scripting approach on the main thread.
<TabAtkins> majidvp: What we want to do is use AW to handle these use-cases as well.
<TabAtkins> majidvp: If we can pull this off, we can incentivize devs to move some critical UI rendering work off of main thread.
<TabAtkins> majidvp: It also lets browsers know it's an input-drive animation, so they can prioritize properly.
<TabAtkins> majidvp: We've all seen janky touch-based effects; it's very hard to pull off on the web.
<TabAtkins> majidvp: So if AW can handle these, it should help a lot with the responsiveness.
<TabAtkins> majidvp: And will have the same basic play/pause api we have for regular time-based animations.
<TabAtkins> majidvp: So a lot of goodness here.
<TabAtkins> majidvp: This is basically what we're going for scroll, but supercharged.
<TabAtkins> majidvp: Like to start with pointer inputs, you can do a lot of interesting things there.
<TabAtkins> majidvp: So why AW as a primitive?
<TabAtkins> majidvp: First it already runs off-main-thread. Designed so it can be stateful.
<TabAtkins> majidvp: And because it's JS you can express complex things that would be very hard to do declaratively.
<TabAtkins> majidvp: Main thing missing is having input other than scroll.
<TabAtkins> majidvp: We've been thinking about this for a little while. Initially thought we could fit everything into timelines; maybe multiple timelines for animations.
<TabAtkins> majidvp: I chatted with Brian and Antoine last time, and realized model doesn't work well. Multi-timeline doesn't fit well with animations, too complex.
<TabAtkins> majidvp: I think our new appraoch is reasonable.
<TabAtkins> majidvp: AniamtionInput is separate from AnimationTimeline
<TabAtkins> majidvp: But touch animations can have multiple effects: pointer-input, scroll-input, etc.
<TabAtkins> majidvp: More complex scroll too - 2d scrolling, maybe active scroll or momentum. Maybe know that finger is down, even if not actively moving.
<TabAtkins> majidvp: So this keeps timeline simple, focused on that use-case, and then have the wider thing.
<TabAtkins> majidvp: I want to build this model so later if there's a gesture exposed, maybe we can get access to those.
<TabAtkins> majidvp: Also want to make sure that you can listen/unlisten to a given input.
<TabAtkins> majidvp: Passively monitor something without it actually ticking you.
<TabAtkins> majidvp: Conceptual diagram:
<TabAtkins> majidvp: [on screen]
<TabAtkins> majidvp: This is the high-level IDL: [on screen]
<TabAtkins> majidvp: Example of PointerAnimationInput [on screen]
<TabAtkins> majidvp: Proposal has a lot more details.
<TabAtkins> majidvp: Here's an example of a more sophisticated scroll...
<TabAtkins> majidvp: I like to dream, here's how a gesture might look.
<TabAtkins> majidvp: [example code]
<TabAtkins> majidvp: [example of mixing scroll+time]
<TabAtkins> majidvp: Hidey-bar, for some portion you want to be linked to scroll, when user lifts their ifnger you want to complete animation based on time.
<TabAtkins> majidvp: Set up two inputs...
<TabAtkins> majidvp: Set actual timeline to empty, then provide the two inputs explicitly.
<TabAtkins> majidvp: This misses its timeline, even tho it takes a scroll as an input, so we can control precisely when scrolling will tick the animation. If provided as a timeline input, it would always tick no matter what.
<TabAtkins> majidvp: So this is a high-level example, showing general direction.
<TabAtkins> majidvp: So you might ask why not just use events?
<TabAtkins> majidvp: For one, bubbling happens. We don't want to necessarily expose DOM via bubbling.
<TabAtkins> majidvp: For two, pull vs push. We think pulling makes more sense.
<TabAtkins> majidvp: One possiblity - instead of this value concept, maybe have a list of events that occurred since last frame.
<TabAtkins> majidvp: Maybe use event interface without actually inheriting from EventListener.
<TabAtkins> Rossen: So what are next steps?
<TabAtkins> Rossen: Continue in WICG?
<TabAtkins> majidvp: No, it's in Houdini already, we'll continue to work there.
<TabAtkins> majidvp: I'm working with Brian now.
<TabAtkins> majidvp: SCrollTimeline, WA, and AW have interdependencies.
<TabAtkins> majidvp: Last time Simon told me to make the extension point clear in WA spec. We're currently patching WA.
<TabAtkins> majidvp: So working with Brian, joined their monthly sync, to bring these concepts to that event.
<TabAtkins> majidvp: Try to make progress on defining theswe.
<TabAtkins> majidvp: And having this patchwork be propertly introduced and integrated into WA.
<TabAtkins> majidvp: Also hoping that ScrollTimeline will push forward at same time.
<TabAtkins> majidvp: And hoping to get dev feedback on ergonomics.
<TabAtkins> majidvp: That's on spec side.
<TabAtkins> majidvp: For new things, I want to figure out this proposal.
<TabAtkins> majidvp: I think there's a lot of value.
<TabAtkins> majidvp: For moving interactive content off of main thread.
<TabAtkins> majidvp: I think our experiment with getting scroll to work from animationworklet has been informative.
<TabAtkins> majidvp: Want to push it so we're in good place.
<TabAtkins> majidvp: One last thing - people are thinking about introducing some form of input to Workers, so I'm gonna try to see if there's a general solution here for all these use-cases.
<TabAtkins> majidvp: I'll try to implement in the polyfill first.
<TabAtkins> majidvp: You can do all this with PointerEvents today.