Inaccurate `Time<Real>`, missed `FixedUpdate`

[dubrowgn]

Bevy version and features

• v0.16.1 (main and older versions also look affected; the code is at least several years old)
• vsync disabled

What you did

While trying to implement a simple frame limiter system using thread::sleep, I noticed Update was running too often compared to FixedUpdate, causing persistent frame stutters.

Analysis

After adding some logging, I discovered the thread was correctly sleeping for the expected e.g. ~15ms, but Time<Real> (and thus all other Time<T>'s) were only moving forward e.g. ~500µs. I tracked Time<Real> updates back to time_system, where I added a line to log the received time:

sys_sleep -
    now: Instant { tv_sec: 5601, tv_nsec: 324402504 }
    Time<Real>: 7.652275701s (delta: 16.982329ms)
sys_sleep - slept 16.334885ms
time_system - received time: Instant { tv_sec: 5601, tv_nsec: 324451814 }
sys_sleep -
    now: Instant { tv_sec: 5601, tv_nsec: 341332892 }
    Time<Real>: 7.652884464s (delta: 608.763µs)

We can see above that actual wall clock time moved forward 16.93ms, but Time<Real> only moved forward 608.763µs.

This leads to FixedUpdate not running, as the requisite amount of time does not appear to have passed. On the next frame, everything catches back up, and FixedUpdate is run twice. Rinse and repeat, causing stutters.

What went wrong

Time<Real> does not track wall clock time. Instead, it consumes timestamps sent from the RenderApp, from render_system in the Render schedule. This causes the main app to consistently lag by the time it takes to process any load between RenderApp Render and main app First. A sleep in this time frame makes the issue more obvious.

Someone here probably knows, but it's not clear from the code why it was designed this way. It seems suspect to me that RenderApp effectively owns Time<Real>, and not the main app. If time really does need to be synchronized in this way, I would have expected it to be the other way around.

Additional information

Potentially related/impacted:

• Res<Time> is unreliable / jittery
• Variable frametime during user input

Caused by: #4744

[dubrowgn]

cc @hymm ; Sending time from RenderApp looks to have been added in a1d3f1b

[dubrowgn]

@aevyrie, I'm wondering if you have seen or experienced similar issues with https://github.com/aevyrie/bevy_framepace. Looking at the code, it seems like it would, but maybe nobody uses it with FixedUpdate?

[hymm]

This is a bit of a complex topic, but I'll try and give a quick summary.

Ideally the time that you want to use for rendering physics is the time between the last frame was rendered to when the current frame will be presented in the future. There are some graphics api's that try to provide this, but bevy doesn't have easy access to this yet. Someone needs to do the work to integrate it with wgpu first.

So what bevy does do is calculate the time between when success calls to present frame are called. This misses the time it takes the gpu to present the frame, but also because of pipelined render we get the n-2 frame's time. We use this time as a proxy for the n frame's time. But this only works when the frame times are relatively consistent and produces jittery movement when it is not. As long as you're hitting vsync it should be relatively consistent.

We can't calculate the time in the main world because there is no consistent place where the frame starts. It gets even worse because of input handling mentioned in the linked issue happens before the main world is run.

Where are you running your sleep? In framerate limiting you usually want to do it in the RenderAppExtract schedule. http://docs.rs/bevy/latest/bevy/render/pipelined_rendering/struct.PipelinedRenderingPlugin.html

If you want to play with how the time is being updated you can change the TimeUpdateStrategy to manual and mess with it that way. https://docs.rs/bevy/latest/bevy/time/enum.TimeUpdateStrategy.html

[dubrowgn]

Thanks for the quick reply. There is some excellent food-for-thought in there. Given the problem space and constraints, I can see how the current implementation came about.

Ideally the time that you want to use for rendering physics is the time between the last frame was rendered to when the current frame will be presented in the future.

This makes a certain kind of sense. You want to draw things where they will be by the time the frame is presented, but this doesn't appear to be what bevy is trying to do. Currently, RenderApp just sends Instant::now to the main app during Render. The main app receives this timestamp in First of the next frame, so the "current" simulation time is always in the past relative to real time. Potentially by a lot if timestamps are queuing in a channel (async render?), or anything interesting happens after Render.

Based on what you described, you would want simulation time to always be in the future relative to real time, right? Am I missing something?

We can't calculate the time in the main world because there is no consistent place where the frame starts.

I (maybe naively) expected it to be the very first thing to happen in the Main schedule. E.g.

1. Timestamp the frame
   • update Time<Real> with Instant::now()
   • set delta to previous frame time
2. Sample input
3. Game logic (e.g. FixedUpdate)
4. Render frame
5. Present frame
6. Clean up

It seems like the above does exactly the same thing you described, if you assume the previous frame time is a decent estimate for how long the current frame should take. Everything that happens in the frame is accounted for in the same frame time because the frame time calculation is the very first thing that happens every frame.

The problem #4744 solved (as I understand it) was that the timestamp happened in between input sampling and vsync-wait. #4744 moved the timestamp to after render, so matching input sampling and vsync-wait would be accounted for in the same frame time. But, it seems (to me at least) a similar result could have been achieved by moving timestamp to before input sampling and continuing to track real time as before. The benefits of tracking real time are: 1) reduced latency, and 2) accurate timing.

In framerate limiting you usually want to do it in the RenderAppExtract schedule.

Anything between input sampling and rendering affects so called "click-to-photon" latency (including vsync wait, which is a major reason why people turn of vsync). Instead, I put my frame limiter in Last, which ensures it runs after render but before input sampling.

[hymm]

Instead, I put my frame limiter in Last, which ensures it runs after render but before input sampling.

last doesn't run after rendering. In runs as the last schedule in the main world which runs in parallel with the render world.

I (maybe naively) expected it to be the very first thing to happen in the Main schedule. E.g.

This is sort of hard to do since input sampling happens inside the app runner. We could make it a concern of the runner. but that adds even more complexity to implementing a app runner. Besides which the time that happens in (6) cleanup belong to the next frame as the current frame has already been presented.

[dubrowgn]

I'm still just reading up on the context. It's interesting that @cart presented the same idea I did in #4728 (comment). The rationale for not doing that was, "Ideally we'd be setting the time as close to when the frame is presented as possible.", but I'm not necessarily clear why that would be ideal.

last doesn't run after rendering. In runs as the last schedule in the main world which runs in parallel with the render world.

Good to know. I don't think it matters much, but I'll have to think through the ramifications of that. (See "frame time" discussion next.)

Besides which the time that happens in (6) cleanup belong to the next frame as the current frame has already been presented.

Kind of critical to this discussion is, "what is a frame?" I probably should have been more explicit above, but when I say "frame time," I really mean the time it takes to run one full Main schedule. That's how long it takes to get back to preparing the next frame. Given any place in the schedule loop, how long until I get back to the exact same place? That's how long I need to simulate, that's the time I need to estimate to implement a frame limiter, etc. Cleanup is part of that time expenditure, so it should be accounted for in the same Main schedule execution you are measuring time for. Otherwise, you get the exact same issue #4744 is trying to fix, just somewhere else.

This is sort of hard to do since input sampling happens inside the app runner.

Maybe we need to define "input sampling" here? Do you mean by the underlying infrastructure? When bevy generates bevy input events? Something else? Worst case, because Main runs in a loop, can we just put it very last then? Or make a Time schedule that runs before the problematic input processing happens?

Sorry if these are dumb questions, but I haven't looked closely at this yet.

If you want to play with how the time is being updated you can change the TimeUpdateStrategy to manual and mess with it that way.

I'm running out of time today, but I'll take a closer look at doing that Monday. Thanks for the pointer.

Edit: in retrospect, #4728 looks exactly like what I proposed, and seemed to get good results. I'm curious why it was abandoned

[hymm]

Do you mean by the underlying infrastructure?

This. The structure of a bevy app is that winit is actually in control of the app loop. app.update() is called when RedrawRequested events come from the OS. Besides RedrawRequested there are all sorts of input events that happen. So what really happens looks more like

1. Input Events
2. Redraw Requested
   (1) Main schedule runs
   (2) Extract Main world data to render world
   (3) Render world runs on a separate thread

In practice you get something like:

|----------------|----------app.update() 1-------------------------------------------------------------|----------------|---------------app.update() 2--------------
| winit events 1 | main schedule 1 (Last runs here) | sync 1 | extract 1 | RenderExtractApp schedule 1 | winit events 2 | main schedule 2 |
|                                                                        | extract commands 1 | rendering schedule 1                      |

Note that the RenderExtractApp Schedule is where you'd want to add delay to so that input sampling runs closer to the presentation time.

I probably should have been more explicit above, but when I say "frame time," I really mean the time it takes to run one full Main schedule.

Oh right. You don't want to use any of the time that bevy provides. Just measure your own duration of the time between when the last sleep ends and the next begins. That'll be the most accurate and that's what bevy_framepace does.

[dubrowgn]

Just measure your own duration of the time between when the last sleep ends and the next begins.

I started with that, but the results leave a lot to be desired when using FixedUpdate. This is because Time<Fixed> is not directly coupled to Instant::now(), so the frame limiter doesn't know how long to sleep without missing the next FixedUpdate step. What you end up with is Update running more often than it needs to, wasted CPU cycles, and stuttering. The solution (theoretically) was to include Time<Fixed>::overstep() in the sleep calculation so you can wake up before the next FixedUpdate. That's the point when I discovered Time<Real> is not actually "real" time, so you still can't know how long to sleep. This is exacerbated by the fact that Time<Fixed> has a max delta, and can be dilated or paused.

You really need to be able to accurately predict Time<Fixed> to implement a good solution, but currently, that just does not seem to be possible.

Note that the RenderExtractApp Schedule is where you'd want to add delay

Maybe I'm missing something, but if timestamping happens in Render, and Render is async to the main Update loop, I don't see how sleeping anywhere after extract (including RenderExtractApp) wouldn't cause the same problem. We still don't know how long to sleep, because the timestamping is effectively decoupled from the main schedule. My understanding is rendering happens after extract "at some point," and does not wait for any load in the main schedule. Is that correct?