Support for "time per move" matches

[kiudee]

Rationale

In some cases it can be useful to tune with fixed time per move. That way the time management is not a confounding variable during tuning. An example application is tuning an engine for analysis strength.

Pull request

This pull request simply passes the appropriate flags to cutechess-cli. The user has to take care to disable all time management/pruning mechanisms in the engine.

Impacted issues

Closes #69

[Claes1981]

Eager to try the time per move time control (hoping the possible time loss problem can be solved by Move Overhead and Slow Mover for Stockfish, and MoveOverheadMs for Lc0 if needed). However I can't manage to install the beta version with the feature yet:

pip install --upgrade --pre chess-tuning-tools==0.6.0b2 ERROR: Could not find a version that satisfies the requirement chess-tuning-tools==0.6.0b2 (from versions: 0.1.0, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.3.0, 0.4.0, 0.5.0b0, 0.5.0b1, 0.5.0b2, 0.5.0b3, 0.5.0b4, 0.5.0b5, 0.5.0, 0.5.1) ERROR: No matching distribution found for chess-tuning-tools==0.6.0b2

Installing from the git source does not seem to work either unfortunately:

pip install --upgrade --editable "git+https://github.com/kiudee/chess-tuning-tools.git#egg=chess-tuning-tools"
Obtaining chess-tuning-tools from git+https://github.com/kiudee/chess-tuning-tools.git#egg=chess-tuning-tools
  Updating ./env/src/chess-tuning-tools clone
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied, skipping upgrade: scikit-learn<0.23,>=0.22 in ./env/lib/python3.8/site-packages (from chess-tuning-tools) (0.22.2.post1)
Requirement already satisfied, skipping upgrade: numpy<2.0.0,>=1.19.1 in ./env/lib/python3.8/site-packages (from chess-tuning-tools) (1.19.1)
Requirement already satisfied, skipping upgrade: emcee<4.0.0,>=3.0.2 in ./env/lib/python3.8/site-packages (from chess-tuning-tools) (3.0.2)
Requirement already satisfied, skipping upgrade: bask<1.0.0,>=0.8.0 in ./env/lib/python3.8/site-packages (from chess-tuning-tools) (0.8.0)
Requirement already satisfied, skipping upgrade: atomicwrites<2.0.0,>=1.4.0 in ./env/lib/python3.8/site-packages (from chess-tuning-tools) (1.4.0)
Requirement already satisfied, skipping upgrade: scikit-optimize<0.8.0,>=0.7.4 in ./env/lib/python3.8/site-packages (from chess-tuning-tools) (0.7.4)
Requirement already satisfied, skipping upgrade: Click<8.0.0,>=7.1.2 in ./env/lib/python3.8/site-packages (from chess-tuning-tools) (7.1.2)
Requirement already satisfied, skipping upgrade: scipy<2.0.0,>=1.5.2 in ./env/lib/python3.8/site-packages (from chess-tuning-tools) (1.5.2)
Requirement already satisfied, skipping upgrade: joblib>=0.11 in ./env/lib/python3.8/site-packages (from scikit-learn<0.23,>=0.22->chess-tuning-tools) (0.16.0)
Requirement already satisfied, skipping upgrade: matplotlib<4.0.0,>=3.3.0 in ./env/lib/python3.8/site-packages (from bask<1.0.0,>=0.8.0->chess-tuning-tools) (3.3.1)
Requirement already satisfied, skipping upgrade: tqdm<5.0.0,>=4.48.2 in ./env/lib/python3.8/site-packages (from bask<1.0.0,>=0.8.0->chess-tuning-tools) (4.48.2)
Requirement already satisfied, skipping upgrade: arviz<0.10.0,>=0.9.0 in ./env/lib/python3.8/site-packages (from bask<1.0.0,>=0.8.0->chess-tuning-tools) (0.9.0)
Requirement already satisfied, skipping upgrade: pyaml>=16.9 in ./env/lib/python3.8/site-packages (from scikit-optimize<0.8.0,>=0.7.4->chess-tuning-tools) (20.4.0)
Requirement already satisfied, skipping upgrade: cycler>=0.10 in ./env/lib/python3.8/site-packages (from matplotlib<4.0.0,>=3.3.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (0.10.0)
Requirement already satisfied, skipping upgrade: pillow>=6.2.0 in ./env/lib/python3.8/site-packages (from matplotlib<4.0.0,>=3.3.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (7.2.0)
Requirement already satisfied, skipping upgrade: certifi>=2020.06.20 in ./env/lib/python3.8/site-packages (from matplotlib<4.0.0,>=3.3.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (2020.6.20)
Requirement already satisfied, skipping upgrade: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in ./env/lib/python3.8/site-packages (from matplotlib<4.0.0,>=3.3.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (2.4.7)
Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in ./env/lib/python3.8/site-packages (from matplotlib<4.0.0,>=3.3.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (1.2.0)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.1 in ./env/lib/python3.8/site-packages (from matplotlib<4.0.0,>=3.3.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (2.8.1)
Requirement already satisfied, skipping upgrade: packaging in ./env/lib/python3.8/site-packages (from arviz<0.10.0,>=0.9.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (20.4)
Requirement already satisfied, skipping upgrade: pandas>=0.23 in ./env/lib/python3.8/site-packages (from arviz<0.10.0,>=0.9.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (1.1.0)
Requirement already satisfied, skipping upgrade: netcdf4 in ./env/lib/python3.8/site-packages (from arviz<0.10.0,>=0.9.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (1.5.4)
Requirement already satisfied, skipping upgrade: xarray>=0.11 in ./env/lib/python3.8/site-packages (from arviz<0.10.0,>=0.9.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (0.16.0)
Requirement already satisfied, skipping upgrade: PyYAML in ./env/lib/python3.8/site-packages (from pyaml>=16.9->scikit-optimize<0.8.0,>=0.7.4->chess-tuning-tools) (5.3.1)
Requirement already satisfied, skipping upgrade: six in ./env/lib/python3.8/site-packages (from cycler>=0.10->matplotlib<4.0.0,>=3.3.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (1.15.0)
Requirement already satisfied, skipping upgrade: pytz>=2017.2 in ./env/lib/python3.8/site-packages (from pandas>=0.23->arviz<0.10.0,>=0.9.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (2020.1)
Requirement already satisfied, skipping upgrade: cftime in ./env/lib/python3.8/site-packages (from netcdf4->arviz<0.10.0,>=0.9.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (1.2.1)
Requirement already satisfied, skipping upgrade: setuptools>=41.2 in ./env/lib/python3.8/site-packages (from xarray>=0.11->arviz<0.10.0,>=0.9.0->bask<1.0.0,>=0.8.0->chess-tuning-tools) (49.6.0)
Installing collected packages: chess-tuning-tools
  Attempting uninstall: chess-tuning-tools
    Found existing installation: chess-tuning-tools 0.5.1
    Uninstalling chess-tuning-tools-0.5.1:
      Successfully uninstalled chess-tuning-tools-0.5.1
  Running setup.py develop for chess-tuning-tools
    ERROR: Command errored out with exit status 1:
     command: /home/claes/Chess-tuning-tools/env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/claes/Chess-tuning-tools/env/src/chess-tuning-tools/setup.py'"'"'; __file__='"'"'/home/claes/Chess-tuning-tools/env/src/chess-tuning-tools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /home/claes/Chess-tuning-tools/env/src/chess-tuning-tools/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib64/python3.8/tokenize.py", line 392, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/home/claes/Chess-tuning-tools/env/src/chess-tuning-tools/setup.py'

[kiudee]

Yes, sadly the myst-nb dependency we now use for the documentation was not released yet. This prevents PyPI from accepting our deployments of the beta version (#73).

What currently works to build a pip installable version (needs poetry):

cd chess-tuning-tools
poetry build
# creates a .whl file in ./dist

# In the environment you want to install chess-tuning-tools to:
pip install --upgrade dist/chess_tuning_tools-0.6.0b2-py3-none-any.whl

I will check, if I can set up Travis CI to also produce whl files to Github.

[Claes1981]

Thanks for the help, I managed to install it when doing as you suggested. :)

[kiudee]

The main dev of myst-nb said the release is 1-2 days away, so soon we can have regular betas again.

[kiudee]

If you want to try the engineX_st options, make sure to also give the engines a time margin using the latest commit (I tested a time margin of 1000ms successfully).
Also turn off any safety buffers the engines might have. For example:

"MoveOverheadMs": 0,
"SmartPruningFactor": 0

for Lc0.

[Claes1981]

Sorry, I do not understand why I should set MoveOverheadMs to 0. Shouldn't a greater value (while still less than the time control) make the engine less likely to go over the time limit?

Update: Regarding Stockfish it seems that running on more than 3 threads cause the time loss problem (discovered thanks to Chess Tuning Tools :) )

Trying to tune with Threads set fixed to 8 threads (and no time margin in Cutechess-cli), no values of Move Overhead and Slow Mover seems to help against time losses...
(Fixed time per move: 1 second/move tested)

[kiudee]

Sorry, I do not understand why I should set MoveOverheadMs to 0. Shouldn't a greater value (while still less than the time control) make the engine less likely to go over the time limit?

The rationale is to have the engine try to play as close to the time limit as possible. Combining a time margin (avoiding time losses) with no safety buffer does that.

[Claes1981]

Alright, I assume that using a time margin might be okay when tuning an individual engine for analysis strength.

However, when comparing the strength between different engines under fixed time per move, for example in a Cutechess match, I suspect an engine actually following the time limit, is disadvantaged if the other engine is allowed to exceed the limit(, and also do exceed it).
Unfortunately, Stockfish does not seem to manage fixed time per move while using many threads. That is of course not a problem of Chess Tuning Tools, but of Stockfish.

Lc0, on the other hand, seems to handle the fixed time per move very well without any time margin in my short test so far! (Currently testing Lc0 version 0.24.1 in Chess Tuning Tools, 5 seconds per move. Only 64 games/32 iterations, but not a single time loss)