Skip to content

Configuring Your OpenBench Instance

Jump to bottom
Andrew Grant edited this page Apr 12, 2024 · 2 revisions

OpenBench contains three types of configuration .json files. Each type is explained below, and examples are present in the repo for reference. Any changes made to these files requires the server to be restarted. OpenBench will attempt to verify the configuration files on startup, and should exit with error messages in the event of a clearly incorrect configuration.

Primary Configuration: Config/config.json

The primary configuration file handles enabling books, engines, basic settings, as well as the client-server auto-updating relationship. Below is a minimal configuration file example, with commentary to follow.

{
    "client_version"  : 22,
    "client_repo_url" : "https://github.com/AndyGrant/OpenBench",
    "client_repo_ref" : "master",

    "use_cross_approval"          : false,
    "require_login_to_view"       : false,
    "require_manual_registration" : false,
    "balance_engine_throughputs"  : false,

    "books" : [
        "Pohl.epd",
        "UHO_4060_v2.epd"
    ],

    "engines" : [
        "Ethereal"
    ]
}

The first three fields, client_version, client_repo_url, client_repo_ref, are used to control the source code version used by any workers. Put simply, the server will tell the client that it needs to be using client_version. If the client does not have the correct version, then it will redownload the source code. It will do this by looking at the provided client_repo_ref, in the client_repo_url. Refs can be branch names, tags, or commit shas. It would not be a great idea to run your server, using AndyGrant/OpenBench and master. This is because when I push a change, you'll need to update your server. By forking the project, or using a commit sha as the client_repo_ref, you can ensure a stable platform.

The next three fields are quick settings:

Name Type Description
use_cross_approval bool Requires another user to approve tests, similar to Fishtest, when set to true. Superusers are exempt.
require_login_to_view bool Requires a login to view any page other than the login page. Useful for private instances.
require_manual_registration bool Disables the Register page, requiring it to be done manually through the database.
balance_engine_throughputs bool Scales throughput of tests, to evenly distribute workers to each Engine

Next we have a very simple list of books. Each entry here is expected to have a matching json configuration file. In my example, we expect that Books/Pohl.epd.json exists, and that Books/UHO_4060_v2.epd.json exists. On startup, the server will read the corresponding file, and check for errors.

Lastly, we have a very simple list of engines. This behaves in the exact same way as the list of books. We expect to find Engines/Ethereal.json, for the example config above. The server will attempt to validate the config, and exit with error messages if there are issues.

Book Configuration: Books/<book>.json

Book configurations are by far the most simple. Each opening book will have its own Books/<book>.json file. To enable the book, you'll simply include the name of the book in the primary congfiguration file (Config/config.json). Below is an example.

{
    "sha"     : "6bf81e1ada6a3306bbc8356f7bca1e2984a2828d658799992d5443b7179c934d",
    "source"  : "https://raw.githubusercontent.com/AndyGrant/openbench-books/master/3moves_FRC.epd.zip"
}

The books can be hosted anywhere on the internet, although all examples present in this repo are hosted on AndyGrant/openbench-books. Github is recommended, however, as other hosting platforms have not been tested. The sha is obtained through the following example process.

>>> import hashlib
>>> with open('3moves_FRC.epd') as fin:
...   content = fin.read().encode('utf-8')
...   print (hashlib.sha256(content).hexdigest())
...
6bf81e1ada6a3306bbc8356f7bca1e2984a2828d658799992d5443b7179c934d

Lastly, it is important that the books be stored in .zip format, as workers expect this. Books can be either PGN or EPD. Additionally, any book containing the case-insensitive text FRC or 960 is assumed to be a Fischer-Random book. Double-Fischer random books are still considered Fischer-Random when it comes to Cutechess.

Engine Configuration: Engines/<engine>.json

Engine configurations have five components. Basic settings, Build settings, Test Presets, and Tune Presets, Datagen Presets. We will break the file into a few parts and explain each one as we go, using a minimal version of Ethereal as the example. For full references, dozens of engines have configuration files in AndyGrant/OpenBench.

Basic Settings:

    "private" : false,
    "nps"     : 1128000,
    "source"  : "https://github.com/AndyGrant/Ethereal",
Name Type Description
private bool Whether the engine is open source (false), or closed source (true).
nps int Reference speed for the engine to be scaled to. This ensures disparate machines can be aligned.
source str Repo location. Not required, but is used for the sidebar links, and auto-filling workload creation.

Build Settings:

    "build" : {
        "path"      : "src",
        "compilers" : ["clang", "gcc"],
        "cpuflags"  : ["AVX2", "FMA", "POPCNT"],
        "systems"   : ["Windows", "Linux"]
    },
Name Type Description
path str Location of the makefile relative to the root of the repo. Not included for Private engines.
compilers list[str] Acceptable compilers. Optionally may include things like gcc>=8 to ensure version requirements.
cpuflags list[str] Minimal system requirement CPU instructions. All uppercase, without dots, spaces, or dashes.
systems list[str] List of valid Operating Systems. May include Windows, Linux, and Darwin.

Workload Presets:

Workload presets are used to auto-fill the various workload creation pages. Each non-default entry in the preset creates a preset button, which can be clicked to immediately fill fields in a default way. When loading a workload creation page, the default settings will first be used to initialize the fields. Afterwards, the STC settings will, if present. From that point on, manual changes, or manually clicking other presets will allow additional changes.

This is most commonly used for having and STC and LTC button, for 2-stage testing. It can also be used to speed up the creation of any number of tests, such as regression tests, tests against specific versions, tests with specific books, tunes with specific settings, etc. Beneath the json is a list of all supported options. The server will verify the correctness of any supplied config.

    "test_presets" : {

        "default" : {
            "base_branch"     : "master",
            "book_name"       : "UHO_4060_v2.epd",
            "test_bounds"     : "[0.00, 3.00]",
            "test_confidence" : "[0.05, 0.05]",
            "win_adj"         : "movecount=3 score=400",
            "draw_adj"        : "movenumber=40 movecount=8 score=10"
        },

        "STC" : {
            "both_options"      : "Threads=1 Hash=8",
            "both_time_control" : "10.0+0.1",
            "workload_size"     : 32
        },
    },

    "tune_presets" : {

        "default" : {
            "book_name"              : "UHO_4060_v2.epd",
            "win_adj"                : "movecount=3 score=400",
            "draw_adj"               : "movenumber=40 movecount=8 score=10",
        },

        "STC" : {
            "dev_options"      : "Threads=1 Hash=8",
            "dev_time_control" : "10.0+0.10"
        },
    },

    "datagen_presets" : {

        "default" : {
            "win_adj"       : "None",
            "draw_adj"      : "None",
            "workload_size" : 128
        },

        "40k Nodes" : {
            "both_options"      : "Threads=1 Hash=16",
            "both_time_control" : "N=40000"
        }
    }

Options for all Workload Types

Name Description Name Description
both_branch Sets both the dev_branch and base_branch dev_branch Sets the Branch for the Dev Engine
both_bench Sets both the dev_bench and base_bench dev_bench Sets the Bench for the Dev Engine
both_network Sets both the dev_network and base_network dev_network Sets the Network for the Dev Engine
both_options Sets both the dev_options and base_options dev_options Sets the Options for the Dev Engine
both_time_control Sets both the dev_time_control and base_time_control dev_time_control Sets the Time Control for the Dev Engine
book_name Desired Opening Book syzygy_wdl Syzygy WDL Tables. DISABLED, OPTINAL, or <N>-MAN
upload_pgns Either FALSE, COMPACT, or VERBOSE syzygy_adj Syzygy Adjudication. DISABLED, OPTINAL, or <N>-MAN
priority Refer to the Creating Tests wikipage win_adj None, or movecount=<X> score=<Y>
throughput Refer to the Creating Tests wikipage draw_adj None or movenumber=<X> movecount=<Y> score=<Z>

Additional options for SPRT Workloads

Name Description Name Description
base_branch Sets the Branch for the Base Engine test_bounds Sets the SPRT elo bounds. Implies an SPRT test type
base_bench Sets the Bench for the Base Engine test_confidence Sets the SPRT error bounds. Implies an SPRT test type
base_network Sets the Network for the Base Engine test_max_games Sets the total games to play. Implies a GAMES test type
base_options Sets the Options for the Base Engine workload_size Refer to the Creating Tests wikipage
base_time_control Sets the Time Control for the Base Engine

Additional options for SPSA Workloads

Name Description Name Description
spsa_reporting_type Either BULK or BATCHED. See the Creating Tunes wikipage spsa_alpha See the Creating Tunes wikipage
spsa_distribution_type Etiher SINGLE or MULTIPLE. See the Creating Tunes wikipage spsa_gamma See the Creating Tunes wikipage
spsa_pairs_per Number of game pairs to play, per point spsa_A_ratio See the Creating Tunes wikipage
spsa_iterations Number of points to evaluate

Additional options for DATAGEN Workloads

Name Description Name Description
base_branch Sets the Branch for the Base Engine datagen_max_games Sets the total number of games to generate
base_bench Sets the Bench for the Base Engine datagen_custom_genfens Additional arguments passed via genfens interface
base_network Sets the Network for the Base Engine datagen_play_reverses Repeat openings as White and Black for each engine
base_options Sets the Options for the Base Engine workload_size Refer to the Creating Tests wikipage
base_time_control Sets the Time Control for the Base Engine
Clone this wiki locally