Testing Approaches for Timing and Statistics Output

[ivanaamit]

This discussion was made from comments from this PR.

Overall, the discussion revolves around the best testing approach for timing and statistics output of gem5, with @giactra and @powerjg providing their perspectives on the matter.

Giacomo:

• Expresses concern about cluttering the simulator with numerous output/stats files.
• Questions the guarantee provided by the simout file in demonstrating the proper functioning of the strided generator.
• Highlights previous issues with stats-based regression testing, suggesting that changes in certain statistics might occur for various reasons.
• Recommends implementing unit-tests for the strided generator, involving a simple configuration file and memory pattern verification, to ensure thorough testing.

Jason:

• Responds to Giacomo's concerns and emphasizes the importance of testing the timing and statistics output of gem5.
• Acknowledges that the current approach may not effectively test a specific component but advocates for testing statistics.
• Suggests having one test for each traffic generator connected to no cache and simple memory to check the timing of traffic generators.
• Suggests taking a few traffic patterns and connecting them with no cache to some DRAM models to check that the DRAM models are not changing their timing unexpectedly.
• Mentions the need for scripts to handle statistics updating in case changes lead to timing differences.

Giacomo:

• Clarifies that they are not against stats-based regression testing but believe that simple text-based reference matching is insufficient.
• Lists requirements for effective stats-based regression testing, including allowing small variations in statistics output and implementing visualization tools for tracking variations, with the capability to pinpoint outliers to PRs/nightly runs.
• Suggests using Monitors and Python to query relevant statistics such as average latency and bandwidth.