[Feature Request] Accuracy/Performance Benchmark Suites for System Cores

[Zenjir0]

Description

The idea for this would be a script or subroutine that you can run per system for the current loaded core.

Example: Nestopia Core is loaded and you run the benchmark for the NES/FC/FDS systems. It loads test ROMs that devs normally use to test for accuracy and performance. Much like how the Mesen and Sameboy Devs do for their testing. https://www.mesen.ca/#Accuracy
https://sameboy.github.io/automation/

Once complete it will give a report showing what passed/failed, top FPS, etc.

Users could run the same benchmark on each core for the appropriate system, that the core supports.

Pros:

• This can help users identify the best core for the system/hardware they are running on.
• While also allowing/enabling core devs to quickly and easily troubleshoot what is or is not working, and see expected or unexpected behavior.
• Possibly encourage emu devs to add libretro/retroarch support to their emulators to utilize this tool.

Cons:

• Drawbacks to adding this feature could lead to core/emulator bias. As users prefer one core over the other as benchmarks get published.
• I myself am unaware of test material for all systems. So only certain systems (most likely 8bit and 16bit systems) will benefit at first.

[RobLoach]

I'd love to see more automated testing and benchmarks across RetroArch. You'll find some tests on:
https://travis-ci.org/libretro/RetroArch

But some cores lack good unit tests.

[dankcushions]

i started trying to do this via shell script: https://github.com/dankcushions/retropie-auto-testing

i think the problem i hit was that i couldn't work out how to terminate retroarch cleanly via command line (timeout etc). you need it to exit cleanly for it to output the --verbose benchmarking information to command line.

a native implementation would be much better. even a --timeout X retroarch command line option (X = ms? number of rendered frames?) would be a start, after which it exits cleanly. that way you could run a game via command line for X seconds/frames, and then the --verbose information tells you how many dropped frames.

[Ferk]

@dankcushions maybe rather than a timeout option, Retroarch should close cleanly when it receives either SIGTERM and/or SIGINT signals, rather than just getting killed without saving, which is what I would expect from a SIGKILL.
This could also be useful for other purposes and integrations with Retroarch, not only automating benchmarks.

EDIT: in fact, according to the documentation it's supposed to work like that already, but I don't think it does. At the moment it's just triggering an exit(1)

RetroArch/frontend/drivers/platform_unix.c

Lines 2195 to 2205 in 33f0009

	static void frontend_unix_install_signal_handlers(void)
	{
	struct sigaction sa;
	sa.sa_sigaction = NULL;
	sa.sa_handler = frontend_unix_sighandler;
	sa.sa_flags = SA_RESTART;
	sigemptyset(&sa.sa_mask);
	sigaction(SIGINT, &sa, NULL);
	sigaction(SIGTERM, &sa, NULL);
	}

RetroArch/frontend/drivers/platform_unix.c

Lines 2182 to 2193 in 33f0009

	static void frontend_unix_sighandler(int sig)
	{
	#ifdef VALGRIND_PRINTF_BACKTRACE
	VALGRIND_PRINTF_BACKTRACE("SIGINT");
	#endif
	(void)sig;
	unix_sighandler_quit++;
	if (unix_sighandler_quit == 1) {}
	if (unix_sighandler_quit == 2) exit(1);
	/* in case there's a second deadlock in a C++ destructor or something */
	if (unix_sighandler_quit >= 3) abort();
	}

[hizzlekizzle]

How about the --max-frames=NUMBER CLI option?

[dankcushions]

@hizzlekizzle awesome! i will experiment with that

[orbea]

maybe rather than a timeout option, Retroarch should close cleanly when it receives either SIGTERM and/or SIGINT signals, rather than just getting killed without saving, which is what I would expect from a SIGKILL.

RetroArch already exits cleanly with SIGTERM and SIGINT signals, see:

# print list of signals
kill -l

# SIGINT
kill -2 $(pidof retroarch)

# SIGTERM
kill -15 $(pidof retroarch)

Adding a signal handler for SIGTERM might just make it harder to debug, crashes should ideally just be fixed instead.

[dankcushions]

@orbea my experience is that it doesn't. at least, it didn't output the --verbose benchmarking information to command line when terminated in that manner.

[orbea]

@dankcushions I'm not sure what you mean?

[dankcushions]

if you launch RA with --verbose command line argument, when you exit via the menu, it dumps a bunch of benchmarking info to the command line. if you terminate the RA process via SIGTERM/SIGINT, it doesn't do that.

[orbea]

I'm still not sure I understand, isn't that what should happen?

SIGINT is the interrupt signal. The terminal sends it to the foreground process when the user presses ctrl-c. The default behavior is to terminate the process, but it can be caught or ignored. The intention is to provide a mechanism for an orderly, graceful shutdown.

SIGTERM is the termination signal. The default behavior is to terminate the process, but it also can be caught or ignored. The intention is to kill the process, gracefully or not, but to first allow it a chance to cleanup.

https://www.quora.com/What-is-the-difference-between-the-SIGINT-and-SIGTERM-signals-in-Linux-What%E2%80%99s-the-difference-between-the-SIGKILL-and-SIGSTOP-signals

What is your desired goal here?

[dankcushions]

The intention is to provide a mechanism for an orderly, graceful shutdown.

sounds to me like it should do everything you would normally expect when exiting retroarch - eg, saving .cfg (assuming save on exit is set), saving .sram, etc. does it do this? it also follows to me that it should also do the --verbose output i described.

i don't believe there should be a difference between behaviour of SIGINT and behavior when exiting via the menu - do you not agree? currently there is a difference.

(i don't really have an opinion about SIGTERM, although i don't know why it should be any different for RA, either)

[orbea]

It seems to exiting normally here with both SIGINT and SIGTERM using the master including saving my settings and printing to the end of the log. This doesn't happen if I use SIGKILL.

[dankcushions]

and if you launch via command line with --verbose? do you get the benchmarking info if you exit via SIGINT/TERM?