Brainstorming designs for data (lag) visualisation in TAStudio

[YoshiRulz]

After last week's lengthy discussion of just what lag means, really, we're still not clear on that point.
But what is clear is that IsLagFrame is holding us back. It's clearly not good enough and no-one likes it or can even agree on how to use it, but it's also integral to emulation and recording.
So since we'll have to live with IsLagFrame for the foreseeable future, I thought we could make some progress by thinking about the user experience of a hypothetical piano roll. (On Discord, I got the impression that the other devs considered UX more important than my concept of "correctness", and I have to admit that it is.)

As always, it was brought up that we have (at least) 2 concepts called 'lag', that a TASer might only find 1 useful, and that which one is more useful depends mostly on the console.
From the previous conversation we had the idea of colouring individual cells red or green, or using a colour in-between red and green for some rows. This lead us to quantify the problem as needing to represent > 1 bit of information but all we have is a bool, obviously just 1 bit.
This lead me to ask: what kinds of things could TAStudio do if it had, say, 2 bits' worth per frame? And so I quickly made a note of how 4 states could be represented with colours and promptly forgot about it for a week.

Concept Y1: As we're used to, background is green when inputs are polled, and red otherwise. When the virtual system "lags", an icon is shown in the gutter beside that row.

Concept Y2: The background now indicates "lag" per se (changing from the base white to red). The stroke colour of the frame index is red when inputs are not polled, and black otherwise.

[nattthebear]

I can't make sense of this issue. You refer to two different concepts here:

1. Whether inputs were polled in a frame
2. "lag"; in quotes with no further elaboration

What is the second one supposed to mean? Can you provide a complete description of the problem, understandable by anyone who was not part of the previous discussion?

[YoshiRulz]

By "lag per se" I mean when the virtual system is overloaded, which can mean video processing or game logic is delayed.

<feos> not all games lag gameplay and delay polling at the same time. what tasers need to know is when the game lags gameplay. whether it polls on that frame or not is useful but not critical
<CasualPokePlayer> some cores implement (at least a mode) where lag frames dont represent non-polling frames at all, but rather some other metric for lag ("gpu lag")
<CasualPokePlayer> 3DS just only does this since it's completely useless to track non-polling frames due to how that works (the console just checks inputs at a constant rate)
<CasualPokePlayer> GC/Wii are similar in that regard (completely useless to track non-polling frames when it's always polled at a constant rate)

(And Mupen64Plus has its alternate lag mode.)
Later:

<feos> polling is an accidental [sic] thing. if gameplay continues while no buttons are polled, it's mostly fine. if it's the other way around, it's worse
[snip]
<feos> lag is when the char stops moving (and most of the time the entire game freezes), which is why it's bad

[vadosnaprimer]

polling is an accidental [sic] thing

What I meant there, the concept of input polling is only a part of the lag concept accidentally: because it happened to be a common indication of gameplay lag in early consoles.

[CasualPokePlayer]

Even for consoles where input polling can be used as a good indicator of gameplay lag, simply trying to consider all input polling might end up being useless. Take the DS, where only its ARM9 CPU (which executes game code) doing input polls is anything useful for lag, if you look at the ARM7 CPU (which executes stock Nintendo code for any licensed game) polling the touch screen, you end up getting no "lag frames" since the stock Nintendo code just polls continously multiple times every frame.

[nattthebear]

Thanks for the context. I largely agree with what's been said above, but to sum up and clarify my viewpoints:

• "Inputs were not observed" will always be useful for TASing. After all, if the input won't be observed, then there's no point in changing it at this time, and good knowledge of this is very helpful for quickly constructing optimal input sequences, both for humans and bots. Usually, we can automatically infer the value of this flag from the application's behavior (whether it read certain ports, etc), but there are situations where this fails, both due to system quirks such as the NDS touchpad above, as well as application logic oddities.
• "Application logic ran slower than nominal" is useful, but in the past we haven't done much with it because on most systems, it can't be inferred automatically. Application code can react to timing interrupts and structure its main loop in an endless number of different ways. With some knowledge about program structure and some system-specific assumptions, sometimes you can do better.

I support splitting out to two flags so long as both are overrideable from lua or other scripting interfaces, and so long as there's enough community interest to actually build out "Application logic ran slower than nominal" detection for common use cases.

[CasualPokePlayer]

There is also a matter of what inputs were not observed. In theory, you could create way more bits to represent this, perhaps 1 bit per button (or per controller?). This could be fairly useful in say linked consoles, or that NDS example, possibly also with consoles with multiple controllers. It would be much more complex however, and somewhat less useful if we just have these different lag definitions used simultaneously.