Support for non-60hz display rates

[quinton-ashley]

Goal

Support non-60hz display rates. This includes:

• the 50hz standard used in Europe, Australia, New Zealand, and parts of Africa, Asia, and South America.

• high refresh rate displays (many Androids and gaming monitors)

• ProMotion variable refresh rate displays (latest iPhone models)

I want p5play developers to be able to share their games with the whole world and have it run well on any capable device.

p5play game developers and players shouldn't have to worry about frame rates.

The Problem

p5.js sets the default _targetFrameRate to 60. It uses requestAnimationFrame but just avoids drawing anything at a higher rate than 60fps. So users with higher refresh rate displays are stuck at 60hz. But if the user's display rate is lower, like 50hz... umm Houston we got a problem! The physics simulation will run 16% slower than real time because by default, p5play updates the physics simulation by 1/60th of a second at the end of each draw call.

By default q5.js will run the draw loop using requestAnimationFrame based on the user's display rate. This would also make the physics simulation too slow on 50hz displays and way too fast on high refresh rate displays.

Personally, as a tech consumer, I've been a big fan of increasing visual fidelity over smoothness, opting for 4K over higher refresh rates. For me the difference between 4K and 1080p is HUGE, but with refresh rates higher than 60hz, I can't really tell the difference. Maybe my eyes are slow or something lol. All the devices I personally own can "only" do 60hz. So my personal biases led me to not consider this major problem until now.

Also it'd be nice if p5play could limit the physics simulation to 30hz if the user's device isn't capable of achieving 60fps.

I was quite conflicted on how to approach this problem, so I did research on Unity.

How Unity does it

Unity separates the game's physics updates from the frame rendering.

https://docs.unity.cn/520/Documentation/Manual/TimeFrameManagement.html#:~:text=Unlike%20the%20main%20frame%20update,the%20last%20physics%20update%20ended.

Unity uses a fixed timestep of 1/50th of a second for physics calculations to ensure consistent physics simulation, regardless of the frame rate.

Unity uses a variable timestep for rendering and general game logic, which is handled in the Update method. This method is called once per frame, so the frequency can vary depending on the display rate.

In between physics updates, Unity interpolates the positions of physics objects for rendering. This means that even if a physics update hasn't occurred for a particular frame, Unity will estimate the current position of the object based on its previous and next calculated positions. This allows the object to appear to move smoothly, even though its actual position is only being updated at the fixed physics timestep.

This approach allows Unity to provide consistent physics simulation while still rendering as smoothly as possible based on the performance of the device.

I think implementing something like this in p5play would be ideal, but how?

Research

After researching this, I made an exciting discovery: I was wrong!
processing/p5.js#6013

requestAnimationFrame DOES provide a much more accurate way to calculate refresh rate. The input arg timestamp records when the callback was scheduled to run!

In my defense though, the MDN documentation is not super clear on this point. I will write a PR to improve it.
https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame

So then the true fix for issue 6013 in p5.js (and q5.js) will be to use the timestamp to calculate _frameRate and deltaTime

Only two lines of code need to be edited to fix the issue!

p5.js/src/core/main.js starting from line 394

this._draw = (timestamp) => {
  const now = timestamp;

Using performance.now() inside the callback to requestAnimationFrame will always generate a time that's a wildly varying amount of milliseconds later than when the timestamp records when the callback was scheduled to run.

So that's the cause of frameRate() and deltaTime calculations varying wildly on a frame to frame basis, even if the sketch is running at a solid 60hz.

Visually for a typical p5.js sketch the difference is not that important since the averages would be the same, but greater precision is critical for p5play physics and animations on a per frame basis.

Using the requestAnimationFrame timestamp in deltaTime calculations makes average variance go down from 0.5-0.2 down to 0.07. That's good enough to really accurately estimate their display rate, or how badly the sketch is under-performing.

Hardware

Many modern smart phones nowadays have high refresh rate displays, so I've tagged this issue as part of my Mobile Dev milestone. Of course these changes will benefit PC gamers with high refresh rate monitors as well.

One thing that's different with computer monitors vs newer iPhones, is that monitors have a fixed refresh rate but newer iPhone's "ProMotion" refresh rates are dynamic, in attempt to strike a balance between smoothness and battery life.

https://developer.apple.com/documentation/quartzcore/optimizing_promotion_refresh_rates_for_iphone_13_pro_and_ipad_pro#//apple_ref/doc/uid/DTS40017657

More problems

If the goal is to not require developers or players to worry about frame rates, that means not having a default frame rate. That means frames shouldn't be used as a unit of time measurement anywhere in a game's code. Yikes!

Animations use the ani.frameDelay property to define how many frames an image in the animation is displayed before the next image is displayed.

The collision handling system stores how long players have been colliding in frames.

Input devices store how long the user has been doing an input action in frames.

Big yikes...

The case for abstracting frames

Switching from frames to seconds would require making tons of breaking changes to p5play.

Also p5.js primarily uses frameCount as its measure of time.

@davepagurek suggested an alternative solution:

I guess there's maybe a world where "frames" are like css pixels, and one "frame" as a unit might actually represent multiple real frames under the hood on high refresh rate displays?
not sure if that's more or less confusing though haha
like if everyone defined sketches as if it were 60fps, but sometimes the frame count is a fraction instead of a whole number

More info: https://www.quirksmode.org/blog/archives/2010/04/a_pixel_is_not.html

I do find using frames (smaller numbers) to be more convenient than milliseconds, a level of precision that's not required for typical user input handling.

I also agree that this issue with higher refresh rates is analogous to the challenge that high pixel density displays posed to developers over a decade ago.

When Apple first introduced high pixel density displays to consumers, I thought the abstract re-branding of pixels was confusing, now it seems perfectly natural. Yet, was that only acceptable because web developers no longer needed to care about real pixels? With retina displays users could zoom in and out without really compromising the visual appearance of text, which Apple had just a few years prior been boasting about always displaying pixel perfect on lower resolution displays.

Have we gotten to that same point with high refresh rate displays above 60fps? I think so.

The case for seconds

How has Apple dealt with these higher display rates?

Apple has opted to abandon the frame as a unit of measurement. They don't even provide an API in iOS for accessing the screen's current display rate, and emphasize that it's dynamic. They recommend that developers tie animation frame changes to the delta time between frame draws.

I also imagine Europeans might not feel too good about abstracting "frame" to be a 60fps frame haha.

Vanilla JS timing functions like performance.now return milliseconds, but Unity, Godot, and Unreal all use seconds stored in a float.

So in the name of increasing accessibility, should p5play follow Apple's and other game engines lead and abandon frame measurements?

Backwards compatibility

If p5play did use seconds, could backwards compatibility be achieved?

I could take Dave's advice and just convert frame times at 60fps to seconds.

It's not too difficult to just deprecate ani.frameDelay and add a new property, maybe ani.frameChange that could be defined in seconds.

But what about the properties of input objects storing frames? They're just going to need to change to use seconds. But how can I possibly do that without breaking backwards compatibility?

One solution is to make user's opt-in via a flag, p5play.useSeconds = true. But I don't like the idea of requiring new users set options. Perhaps I should do it in v3 though, so that I can do a soft roll out right now and update all the documentation, that way all users will be ready when I make it the default in version 4.

[quinton-ashley]

Unfortunately, p5.js just released v1.9.1 (EDIT: and later v1.9.2 as well) without the necessary bug fixes that p5play v4 will rely on. If you want to help, please let the Processing Foundation know that deltaTime needs to be fixed!

I will start work on p5play v4 soon using q5.

[quinton-ashley]

iOS Safari's default limit is 60fps

I did some testing on an iPhone 13 and to my surprise found that p5play was running at 60fps.

I learned that by default even on iOS devices with ProMotion displays, Safari and WKWebViews are limited to 60fps, assumedly to save battery. This can be changed in settings, although I wouldn't expect that many people do it. Low power mode can also limit some iPhone's display rate to 30hz to save battery.

So it turns out existing p5play v3 projects should just run fine on any current iPhone.

Second Thoughts about Seconds

I somehow didn't consider equivalence checks. lol

The thing is, equivalence checks with frames are so nice and simple. Like this example that checks if the user has been pressing the mouse for 10 frames.

if (mouse.pressing() == 10) {
  // run some code one time
}

How could an equivalent check be written using seconds?

if (mouse.pressing() == 0.16666666666) {
  // run some code one time
}

AHHH! It's awful. Also it's not even a solution to the problem with different refresh rates like PAL 50fps. 8 frames would be 0.16 seconds and 9 frames would be 0.18.

ChatGPT says in Unity that something like this would have to be done:

float mousePressedTime = 0f;
bool codeExecuted = false;

void Update() {
    if (mouse.pressing()) {
        mousePressedTime += Time.deltaTime;
        if (!codeExecuted && mousePressedTime >= 0.166f) {
            // run some code one time
            codeExecuted = true; // Set the flag to true after executing the code
        }
    }
    else {
        mousePressedTime = 0f;
        codeExecuted = false; // Reset the flag if the mouse is not being pressed
    }
}

oof!

In ye old days of retro gaming, no wonder developers choose to keep the code simple. They programmed for 60fps and just had the games run slower in Europe. But this problem really ought to be solved better in p5play.

Let's assume that we all like being able to check for equivalence with integers and that the time these numbers represent should be loosely equivalent, regardless of display rate. We also want a user's p5play program they developed using a 60fps display to run pretty much the same on any other display and vice versa.

Abstracting frames (revisited)

It seems abstracting frames is actually the way to go.

like if everyone defined sketches as if it were 60fps, but sometimes the frame count is a fraction instead of a whole number

That'd be fine for frame rates higher than 60hz but not lower. For example, Let's say p5play is running on a 50hz display, it'd need to convert 50fps frames to the 60fps equivalent by rounding.

for (let i = 0; i < 50; i++) {
    console.log(Math.round(i / 50 * 60));
}

0, 1, 2, 4, 5, 6, 7, 8, 10...

Obviously we're gonna be missing some integers, since 50 is less than 60. If when developing a game with my 60hz display, I check if the user has been pressing the mouse for 9 "frames", it would never be true if a player's draw loop rate is 50hz.

if (mouse.pressing() == 9) {
  // run some code one time
}

So this abstract frame would need to be at least as large as the lowest frame rate p5play will support.

Let's say we want to support 30hz and 25hz, even Nintendo still uses such a low frame rate in their biggest games in order for the Switch to keep up. But then our abstract frame would be 1/25th of a second (40ms). Is that precise enough for detecting user input at high levels of play?

Professional F1 drivers average reaction time is ~200ms, so all good on that front. But, the current record for human button presses per second is around 10-15, achieved via the rolling technique used to play tournament level Tetris. That's one button press every 6ms, so 2.8 button presses per 60hz frame (16ms).

Latency and responsiveness are more important factors to consider. As a musician I can attest that low latency between an action, for example playing a key on a keyboard, and hearing a response is important. Humans can time actions with 10-20ms of precision when they are prepared to act. Higher than expected latency could make players think their game is running slow, even if it maintains a solid high frame rate visually.

[quinton-ashley]

Latency and Responsiveness

Is handling user input in the draw loop even a good idea? Most Unity developers do, putting input handling logic in the Update loop that runs once per frame. ChatGPT says for the majority of games, handling input in the Update method provides a good balance between simplicity, performance, and responsiveness.

In p5play, checking user input in the draw loop also enables users to do cool stuff like this:

if (kb.pressing('space') || mouse.pressing()) {
	sprite.color = 'green';
}

Visually there's no point in polling for user input more frequently than the delay between frames. But what about rhythm based games where ~20ms of audio delay is a concern? Fortunately there are already event based functions like mousePressed and keyPressed for that. Perhaps contros should let users define a input handling function that would run on every controller poll. In rhythm games its when the input starts that's the most important factor so not a great concern for our "abstract frames" initiative.

Better Name for "Abstract Frame"

ChatGPT suggests "standard frame". It doesn't seem like a good idea to use the term "frame" because that's not what it's going to be on a device with a non-50hz display. As far as I'm aware there's no precedent for something like this.

ChatGPT suggests beat, pulse, tick, and quantum. beat and pulse are too generic. "tick rate" is already commonly associated with server updates. "quant" sounds cool but "quantum" It doesn't really have anything to do with time and the implication that it's a really small unit makes it a misnomer. I'll have to think about this more.

I'm open to suggestions!

[ShiMeiWo]

How about putting "base" or "ideal" at the beginning of the words?

ex)

• baseFrequency
• idealFPS

[quinton-ashley]

@ShiMeiWo I wouldn't want to use "ideal" cause it implies other fps rates are not ideal. I think baseFrequency is too long. It'd be nice to have a short name for the abstract frame.

[quinton-ashley]

Fixed vs Dynamic

Fixed-Time Step (Fixed Update Rate):

In this approach, the game's logic updates occur at a consistent interval, typically every frame. This means that regardless of the frame rate, the game's logic progresses at the same pace. It offers consistency in gameplay speed but may result in less smooth experiences during performance fluctuations.

Dynamic Frame Rate (Adaptive Time Scaling):

This approach adjusts the gameplay speed based on the frame rate. It ensures smoother gameplay experiences by synchronizing game mechanics with real-time rendering. By adapting to the frame rate, it prevents players from missing crucial frames and provides more consistent experiences across different hardware configurations.

Dropped frames

In the context of video games, dropped frames can occur when the hardware cannot keep up with the demands of rendering the game, resulting in skipped or missed frames in the animation sequence. This can lead to a less smooth and visually appealing gameplay experience, as the animation appears stuttery or choppy. Dropped frames are often an indication of performance bottlenecks that may need to be addressed through optimization techniques or hardware upgrades to ensure smoother gameplay.

If the goal is to separate the draw loop from the physics simulation, how should the abstract frame system handle dropped frames?

Currently p5play v3 uses a fixed update rare. If a game can't maintain 60fps, dropped frames are not counted in frameCount. Only after a frame is completed will p5play's post draw function run, which by default runs world.step and updates contact handling and input frame counters.

to be continued...