Requirements For Public Engines

One of the primary purposes of OpenBench is to be able to provide a single testing framework capable of supporting the development of multiple engines, allowing authors to pool their resources for testing. OpenBench also allows users to play cross-engine games. In order to achieve this, all engines must have uniform compliance in regards to a few aspects of the engine. Those will be outlined and explained below. Afterwards, there will be some explanation of the cpuflags, and systems portion of an Engine's configuration, which need to be understood by authors hoping to have 3rd parties run their engines.

Basic Requirements

• All engines must support the UCI option for Hash and for Threads. All workloads require both of these options, in order for clients to figure out whether they are capable of running the workload, and if so, how many concurrent games they can play. If your engine does not support multi-threading, you may report option name Threads type spin default 1 min 1 max 1 in your UCI startup. The same could be done for Hash.
• All engines must be buildable by executing a makefile. One of the configuration settings for all public engines is the location of the makefile in your repository. This makefile is how your engine will be built. It is possible to create an OpenBench specific makefile, by putting the makefile in its own directory. Engines written in languages other than C/C++ are still built by executing a make command, although the makefile might be as simple as executing cargo, and passing along an option or two.
• All engine makefiles must support passing EXE= to control the output binary name. When building your engine, OpenBench will execute make EXE=Engine-ABCDEFGH. It is critical that the output file is indeed named Engine-ABCDEFGH, or Engine-ABCDEFGH.exe in the same directory as the makefile, otherwise OpenBench will not find your binary, and will report an error.
• All engines supporting multiple compilers must support CC/CXX= for C/C++ respectively. Engines that support multiple compilers need to allow OpenBench to communicate this information to the makefile. This is done conventionally by CC= for engines written in C, and CXX= for engines written in C++. The default compiler in the makefile should always match the compilers configuration, for engines with only one compiler.
• All engines must support being "benched" from the command line. In order to affirm that the build is correct, and to determine the relative speed of the clients CPU, engines will be executed from the command line with ./binary bench. This must report a final node count, and a final nodes-per second count, and then exit. A simple, acceptable format is 4712710 nodes 1323423 nps. Refer to Client/bench.py's parse_bench_output() function for specifics.
• All engines must be deterministic across multiple runs, on all supported machines. Every client will bench your engine, and they must produce the same nodes result every time. For example, Zobrist keys should be generated with an initial seed. Some very complex non-determinism can arise when mixing AVX and AVX2+ machines. For more information on that, ask in the OpenBench Discord's #openbench-support channel.
• All engines using the Networks feature on OpenBench must support compilation with EVALFILE= to embed Networks. Its considered a bad idea to commit large binary files to git repos, so OpenBench allows uploading Network files, and compiling them into the engine. If a Network file is specified, it will be downloaded, and the make command will include EVALFILE=/path/to/network-ABCEDFEGH. Your build process must embed the weights inside the engine binary.

Platform & Instruction Set Support