Weird behaviour when document is not visible anymore

[binarykitchen]

Hello again!

This is probably a question, not sure if it's an animitter bug. The short version is: when the active window aka document looses focus (i.E. I switch to another browser tab), animitter seems to behave funny. Delta- and Elapsed times do not seem right.

The longer version, where it happens:

1. https://github.com/binarykitchen/videomail-client/blob/develop/src/wrappers/visuals/recorder.js#L794
2. When I am recording there and switch to another tab and come back, there is a difference in the delta time.
3. What I assume is that animitter pauses. But I don't want it to pause, to continue recording while the document is not visible. It was possible with my own code before I used animitter.

Thoughts?

[roboshoes]

Hi there, I think the issue is that window.requestAnimationFrame, which animitter relies on, does not run reliably when the window is not active or visible. Browsers do that in order to save CPU+GPU+Memory and all sorts of optimizations. (Same with CSS3 animations and transitions)

The best way to check this is by listeing for the "blur" and "focus" events on window. After/before which requestAnimationFrame won't run at 100% the speed it could.

I'm not 100% sure this is why your "bug" occures but it would explain it 😄

[hapticdata]

Yes, Mathias is correct. Since requestAnimationFrame is intended for animation and to cause a repaint on a tab that isn't visible can drain battery etc the spec says to abort calling the callback based on the hidden attribute that comes from the Page Visibility API.

You could listen for visibilitychange event on document and use that play()/stop() animitter if you want to negate the time on different tabs in your deltaTime and elapsedTime.

If you wish to make it record while your document is not visible you could use setTimeout, its less preferred for animations but does continue while page visibility is hidden.

You can even use animitter with setTimeout with just a little bit of additional complexity:

const loop = animitter();
//setTimeout requires a millisecond interval
//loop.getFPSLimit() will return Infinity unless you have explicitly set the FPS 
//this is because of VR headsets running at 90+
loop.setRequestAnimationFrameObject({
    requestAnimationFrame: (fn)=> setTimeout(fn, 1000 / Math.min(60, loop.getFPSLimit()),
    cancelAnimationFrame: (id)=> cancelTimeout(id)
});

[binarykitchen]

i see and thanks for the link to the specs - this explains all.

can you imagine building this feature into animitter, so that it also can deal with visibility changes? this with a new option using the above setTimeout trick? of course with a warning that this is not recommended for performance reasons but i need it for my videomail app (special case).

what do you think guys?

[binarykitchen]

@MathiasPaumgarten @hapticdata any thoughts on the above?

[roboshoes]

This is up to @hapticdata 🤷‍♂️

But as far as I am concerned, @hapticdata listed a possible solution for you:

You can even use animitter with setTimeout with just a little bit of additional complexity:

const loop = animitter();
//setTimeout requires a millisecond interval
//loop.getFPSLimit() will return Infinity unless you have explicitly set the FPS 
//this is because of VR headsets running at 90+
loop.setRequestAnimationFrameObject({
    requestAnimationFrame: (fn)=> setTimeout(fn, 1000 / Math.min(60, loop.getFPSLimit()),
    cancelAnimationFrame: (id)=> cancelTimeout(id)
});

I think adding setTimeout as a permanent option might be out of scope for animitter as its original idea was to avoid using setTimeouts as far as I understand.
Especially since your scenario seems to be a specific edge case.

[hapticdata]

yeah, I am mixed on this. Both seem a little outside the scope of animitter's concerns. I am considering integrating Page Visibility into an option for animitter such as:

animitter({ pauseWhenHidden: true })

but it could also be argued that it would be better to use something like VisibilityJs to avoid another API dependency + polyfill. Using that library you could just do:

const loop = animitter().start();
Visiblity.change(()=> loop[ Visibility.hidden() ? 'stop' : 'start']())

still debating setTimeout as well

[roboshoes]

As far as I understand @binarykitchen's problam, it's not that he wants animitter to pause, but rather to keep runner reliably even if the page is not visible, right?
So pausing would actually not solve his problem? I honestly think the best thing would be to set a setTimeout based callback using the already existing: setRequestAnimationFrameObject

[binarykitchen]

@MathiasPaumgarten correct matthias, i want animitter to continue even when not visible anymore.

i think this should be in the animitter code as a fallback regardless. because using the above setTimeout solution outside of the realm of animitter can become messy. animitter is supposed to solve one problem, regardless in what state the browser document is, isn't it?

btw, on your readme you say
"Animitter is a combination of an EventEmitter and a feature-filled animation loop. It uses requestAnimationFrame with an automatic fallback to setTimeout..." <- fallback to setTimeout?

when you already have such a fallback, then why not? ;)

[hapticdata]

The fallback occurs for browsers IE9 and older, in this scenario its not wanted as a fallback but to be forced.

In general it is not desirable for animitter to keep running in the background, if you are developing any sort of animation, webgl / canvas site, VR experience etc you would never want all of that work to be done on the users computer while they can't see it.

Its valid to want that behavior for projects that are performing client-side rendering. I have added this type of support in other ways with features like animitter({ fixedDelta: true }) and animitter.globalFixedDelta so that the timer always responds with the same millisecond delay (so that longer or asynchronous frame renders still run normally).

I am opposed to animitter({ useSetTimeout: true }) because it lacks elegance and requires the library to consider itself in one of two modes. The above code requires access to the animitter.Animitter instance so to use it would look like loop.setRequestAnimationFrameObject(animitter.createSetTimeout(loop)) which doesn't seem great either.

What is the short comings of just using the code I provided in the first comment?

[hapticdata]

considering this: loop = animitter.useSetTimeout(options)

exports.useSetTimeout = (options)=>{
    var loop = animitter(options);
    function raf(fn){
        setTimeout(fn, 1000 / Math.min(60, loop.getFPSLimit()));
    }
    function cancel(id){
        cancelTimeout(id)
    }
  
    loop.setRequestAnimationFrameObject({
        requestAnimationFrame: raf,
        cancelAnimationFrame: cancel
    });
    
    return loop;
}

need some time to think about it and am busy on some other projects for a little while

[binarykitchen]

worse, what if the user switched between tabs in the browser and comes back to the initial tab?

in other words:

1. anmitter has started the loop as usual, the normal way with animation frames
2. switched to a tab, visibility changes
3. we somehow have to tell animitter to continue with the loop but with the use of setTimeout
4. when user comes back to that tab, tell animitter to stop using setTimeout and to reuse animation frames again

getting tricky eh?

[binarykitchen]

also have experimented here if it's possible to set setRequestAnimationFrameObject in the middle of the loop. i am afraid not, it does not seem to respect the new request animation frame object.

[hapticdata]

it should work just fine to use setRequestAnimationFrameObject in the middle of it running and hot-swap between objects, I frequently do this with WebVR projects since that spec requires you to call VRDisplay#requestAnimationFrame in order to get new pose data.

What is the issue you are running into when hot-swapping with setRequestAnimationFrameObject?

The main benefit of requestAnimationFrame over setTimeout is that it does not loop while its tab is hidden. If you want the behavior of running when hidden then it doesn't make much sense to ever use requestAnimationFrame you should just use setTimeout at all times.

[binarykitchen]

@hapticdata yeah, about the main benefit, i already got that. thanks.

i tried that hot-swapping in the middle yesterday, but it didn't work. can you point me to the exact code line where the animitter loop is reading the new request animation frame object?

[hapticdata]

certainly,

the setRequestAnimationFrameObject is as expected and here is where I deal with a hot-swapped rAF object: https://github.com/hapticdata/animitter/blob/master/index.js#L96-L106

Is it correct that what you are doing is using requestAnimationFrame and then when you receive a hidden event you are changing to setTimeout? If so, that may not reliably work because requestAnimationFrame calls drawFrame asynchronously and could be ignored when hidden has already happened. You would need to switch to setTimeout one-tick before becoming hidden occurs which I don't believe is possible.

In WebVR if you try to access pose data from anything but VRDisplay#requestAnimationFrame an error gets thrown. Once I receive a VR display I do this:

loop
        .setRequestAnimationFrameObject(button.manager.defaultDisplay)
        .off('update', desktopUpdate)
        .on('update', vrUpdate);

where desktopUpdate renders my scene without VR and vrUpdate renders my scene with VR data. I do not get any errors from this because I ensure once it has been set there will never be another update on the old requestAnimationFrameObject

in case its useful, heres a gist of one quick WebVR prototype where it works

[binarykitchen]

Is it correct that what you are doing is using requestAnimationFrame and then when you receive a hidden event you are changing to setTimeout? If so, that may not reliably work because requestAnimationFrame calls drawFrame asynchronously and could be ignored when hidden has already happened. You would need to switch to setTimeout one-tick before becoming hidden occurs which I don't believe is possible.

exactly, that's the problem! can't do the switch before documents gets hidden. how about adding a new public function to call drawFrame() directly (i know it is ugly but probably the solution for this special case?) - so that it will continue with the new requestAnimationFrameObject

(re the VR thing, i dont see how this is relevant here, but good to know)

[hapticdata]

VR thing was just to show you that it does respect a new requestAnimationFrameObject in response your comment:

also have experimented here if it's possible to set setRequestAnimationFrameObject in the middle of the loop. i am afraid not, it does not seem to respect the new request animation frame object.

If you listen for a hidden event and do:

loop
    .stop()
    .setRequestAnimationFrameObject(useSetTimeoutObject)
    .start()

you should be able to keep it going because the first call in onStart is synchronous

[binarykitchen]

aaah, gotcha! will try that tonight ...

[binarykitchen]

ah, that worked @hapticdata but now we have the old problem that the frame rate is very wrong when using setTimeout .... oh well :(

is there a known trick how to make it work best with setTimeout?

[binarykitchen]

for now is is the best code i can come with

    function loopWithTimeouts() {
        debug('Recorder: loopWithTimeouts()')

        const wantedInterval  = 1e3 / options.video.fps // which is 15fps

        var processingTime = 0,
            start

        function raf(fn) {
            return setTimeout(
                function() {
                    start = Date.now()
                    fn()
                    processingTime = Date.now() - start
                },
                // reducing wanted interval by respecting the time it takes to
                // compute internally since this is not multi-threaded like
                // requestAnimationFrame
                wantedInterval - processingTime
            )
        }

        function cancel(id) {
            clearTimeout(id)
        }

        setAnimationFrameObject({
            requestAnimationFrame: raf,
            cancelAnimationFrame: cancel
        })
    }

but it turns out, that when i switch to another browser tab, animitter's deltaTime suddenly jumps from 67ms to 1000ms!

this can't be right, when the above timeout is triggered in average about 40ms. a bug?

[hapticdata]

It looks like setTimeout still gets de-prioritized by modern browsers.

I ran this code on this page in lastest Chrome:

last = Date.now();
next = ()=>{
    let now = Date.now(); 
    console.log(now-last); 
    last = now; 
    setTimeout(next, 40);
};

next();

and got this result, the higher numbers is when I went to another tab:

41
44
43
409
977
1063
1386
539
1499
748
753
740
41
40

doesn't look like theres much that can be done about it. If you need the deltaTime to pretend to be consistent you can use the fixedDelta option that I implemented for recording canvases in less-than realtime:

loop = animitter({
    fps: 15,
    fixedDelta: true
})

[binarykitchen]

to pretend? isn't that dangerous? and what disadvantages will fixedDelta come with?

[hapticdata]

It isn't dangerous but it also may not serve any utility in your use case.

Typically procedural animations such as those you find in games or visualizations will base their animation off of a delta time or a total elapsed time so that animations stay in sync. Say you have made some very nice procedural animation and you decide that you would like to render out the canvas for an image sequence or video; you might not be able to maintain your 60 (or any other) fps framerate while recording. The fixedDelta feature allows your code to stay the same and for your animations to still look smooth in your final recording even if each frame takes several seconds to save and render.