Alternative CPU benchmark pt. 1

[Garanas]

An alternative way of tracking the CPU performance, over using a benchmark written in the lobby. With thanks to @KionX we can now retrieve the current maximum sim speed of each player, see also: FAForever/FA-Binary-Patches@44fd742

This will be a two part pull request. The next release (20th of August) will contain the first part, the next release (not sure when yet) will contain the second part. The idea is that in the first part everyone can start gathering data. In the second part we replace the CPU score and because everyone has data it is immediately useful.

The question is though, what data should we keep? At the moment it computes per sim rate (e.g., -10, -9, -8, ... +9, +10) the min / max unit and entity count. It constantly tries to converge this, allowing the data to update accordingly. See also the new file /lua/system/performance.lua . Note that due to the location of this file it can't be hooked by mods. See an example in the next post.

The idea is that instead of having a CPU score, we have a red / orange / yellow / green indication that you can hover over for additional information. The color of the indication is based on whether this player is expected to slow down the game based on the map, the number of players, the unit cap and the performance data we are gathering in this pull request. The additional information is a graph of the data that we covered: on the vertical axis the unit count, on the horizontal axis the sim rate.

But can we do better? I'm open to ideas.

Note that the performance tracking window is incomplete.

In order to test this, make sure that you have the executable that is part of the FAF Develop game type. You can get that by starting the game once on that game type, close it again and then start using your development environment. You can find the results in the preference file.

[Garanas]

PerformanceTrackingV1 = {
    ['Open Palms - FAF version - 2'] = {
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            EntityCount = {
                Min = 7461,
                Max = 8309
            },
            Samples = 13,
            UnitCount = {
                Min = 362,
                Max = 413
            }
        },
        {
            EntityCount = {
                Min = 6807,
                Max = 7867
            },
            Samples = 20,
            UnitCount = {
                Min = 293,
                Max = 390
            }
        },
        {
            EntityCount = {
                Min = 4401,
                Max = 6957
            },
            Samples = 150,
            UnitCount = {
                Min = 6,
                Max = 319
            }
        },
        Samples = 1
    },
    ['Adaptive Archsimkats Valley - 3'] = {
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            EntityCount = {
                Min = 8512.7998046875,
                Max = 10561.200195313
            },
            Samples = 680,
            UnitCount = {
                Min = 40.534248352051,
                Max = 734.15069580078
            }
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            EntityCount = {
                Min = 8488.0498046875,
                Max = 10108.94921875
            },
            Samples = 864,
            UnitCount = {
                Min = 46.000003814697,
                Max = 855.99993896484
            }
        },
        Samples = 10
    },
    ['Setons Clutch - FAF version - 4'] = {
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            EntityCount = {
                Min = 15308,
                Max = 15336
            },
            Samples = 4,
            UnitCount = {
                Min = 3431,
                Max = 3438
            }
        },
        {
            EntityCount = {
                Min = 15233,
                Max = 15299
            },
            Samples = 8,
            UnitCount = {
                Min = 3415,
                Max = 3433
            }
        },
        {
            EntityCount = {
                Min = 15302,
                Max = 15363
            },
            Samples = 4,
            UnitCount = {
                Min = 3431,
                Max = 3438
            }
        },
        {
            Samples = 0
        },
        {
            EntityCount = {
                Min = 15243,
                Max = 15956
            },
            Samples = 8,
            UnitCount = {
                Min = 3180,
                Max = 3444
            }
        },
        {
            EntityCount = {
                Min = 18333,
                Max = 19175
            },
            Samples = 4,
            UnitCount = {
                Min = 3675,
                Max = 3732
            }
        },
        {
            EntityCount = {
                Min = 13903,
                Max = 20017
            },
            Samples = 68,
            UnitCount = {
                Min = 3135,
                Max = 3882
            }
        },
        {
            EntityCount = {
                Min = 13863,
                Max = 20416
            },
            Samples = 630,
            UnitCount = {
                Min = 2704,
                Max = 3917
            }
        },
        {
            EntityCount = {
                Min = 13800,
                Max = 19375
            },
            Samples = 652,
            UnitCount = {
                Min = 1941,
                Max = 3787
            }
        },
        {
            EntityCount = {
                Min = 13821,
                Max = 40331
            },
            Samples = 352,
            UnitCount = {
                Min = 1396,
                Max = 3155
            }
        },
        {
            EntityCount = {
                Min = 15316,
                Max = 43210
            },
            Samples = 244,
            UnitCount = {
                Min = 1220,
                Max = 2282
            }
        },
        {
            EntityCount = {
                Min = 32583,
                Max = 43460
            },
            Samples = 44,
            UnitCount = {
                Min = 872,
                Max = 1535
            }
        },
        {
            EntityCount = {
                Min = 38182,
                Max = 43438
            },
            Samples = 36,
            UnitCount = {
                Min = 671,
                Max = 1132
            }
        },
        {
            EntityCount = {
                Min = 22502,
                Max = 37570
            },
            Samples = 12,
            UnitCount = {
                Min = 450,
                Max = 658
            }
        },
        {
            EntityCount = {
                Min = 13336,
                Max = 32287
            },
            Samples = 12,
            UnitCount = {
                Min = 334,
                Max = 576
            }
        },
        {
            EntityCount = {
                Min = 26138,
                Max = 28709
            },
            Samples = 4,
            UnitCount = {
                Min = 504,
                Max = 543
            }
        },
        {
            EntityCount = {
                Min = 9779,
                Max = 12246
            },
            Samples = 4,
            UnitCount = {
                Min = 274,
                Max = 315
            }
        },
        {
            Samples = 0
        },
        {
            EntityCount = {
                Min = 5420,
                Max = 9650
            },
            Samples = 90,
            UnitCount = {
                Min = 8,
                Max = 252
            }
        },
        Samples = 1
    }
}

[Garanas]

After discussing it with @Hdt80bro , we think it is better to:

Instead of an each for each map, we have only three entries:

• Regular skirmish maps (everything in the vault), with no AIs
• Regular skirmish maps (everything in the vault), with AIs
• Campaign maps (everything in the co-op tab)

And use the following formula to determine the average unit count for a setup in the lobby: numberOfPlayers * (0.5 * Unitcap)

[KionX]

I have two ideas(simplified):

1. maxSP / UnitCount * numberOfPlayers * Unitcap * 0.5
2. Lua benchmark * numberOfPlayers * Unitcap * 0.5

[MrRowey]

i think this is going to be a better approach. if you were to have one for each map that list is going to get very long xD

[Garanas]

PerformanceTrackingV2 = {
    skirmish = {
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 14,
            UnitCount = {
                Min = 1,
                Max = 2
            }
        },
        Samples = 2
    },
    ['skirmish-ai'] = {
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 0
        },
        {
            Samples = 16,
            UnitCount = {
                Min = 641,
                Max = 999
            }
        },
        {
            Samples = 61,
            UnitCount = {
                Min = 589,
                Max = 938
            }
        },
        {
            Samples = 39,
            UnitCount = {
                Min = 435,
                Max = 770
            }
        },
        {
            Samples = 16,
            UnitCount = {
                Min = 311,
                Max = 456
            }
        },
        {
            Samples = 46,
            UnitCount = {
                Min = 6,
                Max = 404
            }
        },
        Samples = 1
    }
}
CPUBenchmark = 140

See also the old score, which is very vulnerable to editing 😄