Skip to content

Kestrel 3.0

Choose a tag to compare

@Alon-Berg Alon-Berg released this 05 Nov 18:43
· 29 commits to main since this release

Kestrel 3.0

A much stronger network, and Syzygy tablebases. Against 2.3 it scores +130 ± 8 Elo over six time controls (3,800 games, one thread), and +104 ± 11 at 5 s + 0.1 s per game or longer (1,600 games).

What is new compared with 2.3

  • A new network, trained on about nine times as many positions of the same kind. It has the shape of the 2.x network (39.1 million parameters, 24,576 inputs per side, 1,536 neurons, 16 -> 32 -> 1 with 8 output buckets). The 2.x network was trained on 238 million positions of the Lichess evaluation database and overfitted: after about 50 hours the error on held-out positions stopped falling while the error on the training positions kept falling (0.0129 against 0.0089), because the database had no more positions to offer. Every evaluation in that database comes with the best lines that Stockfish found (up to 5 lines of 10 moves). 3.0 also learns from the positions along those lines: the positions after 1 to 9 moves of the best line and after 1 to 6 moves of the other lines, each labelled with the evaluation of its line; only quiet positions (not in check, no capture or promotion as the next move), duplicates removed, and the held-out test positions and the positions of the older training set left out. That gives 2.36 billion new positions (mined in two batches of 1.08 and 1.28 billion), and with them the overfitting is gone. The network was trained in two stages on one laptop CPU: 650,000 steps of 16,384 positions (54 hours) and a continuation of 90,000 steps (8 hours) on the second batch; the shipped network is the best snapshot of the second stage. On held-out positions its error is 17 % lower than the 2.3 network's on Lichess positions and 32 % lower on positions from Kestrel's own games. Stockfish is used only as the teacher of the data; no Stockfish code or network is included.
  • Syzygy tablebases. Set the new option SyzygyPath to the folders with the .rtbw / .rtbz files (the 3-4-5-men files are 940 MB; larger sets use the same code but were not tested). Inside the search a position with at most as many men as the largest table is looked up right after a capture or a pawn move; at the root, the DTZ files choose the moves that keep the best result and make progress under the fifty-move rule. New options SyzygyProbeDepth, Syzygy50MoveRule, SyzygyProbeLimit; info lines carry tbhits. Without tablebase files nothing changes. Details and tests: docs/SYZYGY.md. The probing code is Fathom (MIT licence, Ronald de Man, basil00, Jon Dart); it is part of the single source file and its licence is in THIRD_PARTY_NOTICES.txt.
  • Speed. The network is as large as before, so the cost of a node is almost the same (1.07 million nodes per second in the depth-12 bench against 1.13 million for 2.3), but the stronger network makes the search tree smaller: the bench needs 859,322 nodes instead of 1,051,844 and takes about 808 ms instead of 935 ms. On Windows the engine now asks the system not to treat it as a background process (efficiency mode): a GUI that starts the engine hidden could make Windows slow it down; opting out gave +13 to +17 % nodes per second at 2 to 8 threads in a test. info nodes and info nps no longer add the stale counters of idle helper threads when MultiPV is above 1 or fewer threads than requested could be started.

Strength

Kestrel 3.0 against Kestrel 2.3, packaged x86-64-v3 builds, fastchess, 1 thread, 64 MB hash, no book, no learning, no adjudication, the same openings for both sides (colours swapped); Elo with its 95 % interval.

Time per game (base + increment, seconds) Elo of 3.0 against 2.3 Won Lost Drawn
1 + 0.01 +192 ± 23 379 77 144
2 + 0.02 +150 ± 23 350 106 144
3 + 0.03 +146 ± 17 563 166 271
5 + 0.1 +105 ± 14 475 181 344
10 + 0.1 +102 ± 22 181 67 152
20 + 0.5 +102 ± 28 92 35 73

Pooled over all time controls: +130 ± 8 Elo against 2.3 (3,800 games, inverse-variance weights). The gain is largest at the shortest time controls and levels off with more time: from 5 s + 0.1 s per game on it is +104 ± 11 Elo (1,600 games). The gain comes from the network alone: the search and the time management are those of 2.3, and in a match of the two networks inside the same engine at 20,000 nodes per move the 3.0 network is +138 ± 11 Elo ahead (2,000 games). The network has the same size as before, so the cost of one node is almost the same (1.07 million nodes per second in the depth-12 bench on a performance core against 1.13 million for 2.3, 5 % fewer), but the stronger network makes the search tree smaller: the bench needs 859,322 nodes instead of 1,051,844 and takes about 808 ms instead of 935 ms (14 % less time).

In my own round robin against 100 other engines (5 s + 0.1 s per game, one thread, this laptop, every engine at full speed) 3.0 scored 76.0 of 200 games; fitted onto the CCRL Blitz scale through the engines whose ratings did not change that is about 3576 (95 % bootstrap interval 3552 to 3597), which would put it 71 of 101 in that table. This is an estimate under my own conditions, not a rating-list result.

Raw results: logs/fast_matches_3_0.txt.

What did not change

The search, the time management and the options of 2.3 are the same, and so is Lazy SMP with up to 8 threads. No rating list has tested Kestrel.

Which file do I need?

File For
Kestrel-3.0-windows-x86-64-v3.zip Windows 64-bit, AVX2 + BMI2 CPUs (recommended)
Kestrel-3.0-windows-x86-64-v2.zip Windows 64-bit, SSE4.2 + POPCNT CPUs
Kestrel-3.0-windows-x86-64.zip Windows 64-bit, any CPU
Kestrel-3.0-linux-x86-64-v3.tar.gz Linux x86-64, AVX2 + BMI2 CPUs
Kestrel-3.0-linux-x86-64-v2.tar.gz Linux x86-64, SSE4.2 + POPCNT CPUs
Kestrel-3.0-linux-x86-64.tar.gz Linux x86-64, any CPU
Kestrel-3.0-linux-arm64.tar.gz Linux ARM64
Kestrel-3.0-macos-x86-64.tar.gz macOS, Intel Macs
Kestrel-3.0-macos-arm64.tar.gz macOS, Apple Silicon (M1 and newer)

The three Windows builds are also attached as plain .exe files (same program as in the zips; a plain exe has no opening book: put claude_book.txt next to it and switch OwnBook on if you want it). kestrel_net_3_0.bin is the network alone (needed only to build from source or to load with EvalFile). The 3.0 builds can also load the network of 2.x with EvalFile (the same format, width 1,536); that is how the two networks were compared.

Only the x86-64-v3 builds have the AVX2 evaluation code (CPU with AVX2 and BMI2: most PCs since 2015). The v2 and generic builds work on older CPUs but are about 37 % slower.

Status of each build

  • Windows: all three builds were run here; each gives the same depth-12 bench (859,322 nodes), so the plain code and the AVX2 code agree exactly. Both black-box audits pass (83 of 83 and 31 of 31; one check of the second was relaxed for 3.0, see the changelog). The 30-check verification with GCC and clang AddressSanitizer + UBSan builds (also with 4 threads) passed 30 of 30. The Syzygy test against python-chess (3-4-5 men): 1,500 positions, 0 mismatches; root filter 1,483 positions, 0 problems; conversion 30 of 30 won endgames.
  • Linux and macOS: compiled with Zig and checked for file format. The workflow in .github/workflows/verify-release.yml runs the published files on GitHub's Linux, macOS and Windows machines (bench, a UCI session, the Syzygy test with the 3- and 4-men files, and on Linux the whole test suite) and builds the source with AddressSanitizer and UBSan; before the release the same workflow ran on this version's source (built with the 2.3 network) and passed everywhere; its run on the published 3.0 files is on the Actions page. The macOS Intel build is run by the workflow on GitHub's Intel Mac machine after the release is published (I have no Mac myself).
  • The exe is about 88 MB and is not code-signed; some virus scanners are suspicious of big unsigned programs (1.0 had a false positive, see docs/BUILDING.md).

Building from source

src/kestrel_3_0.cpp embeds the network with .incbin "kestrel_net.bin", which the assembler looks up in the current directory: download kestrel_net_3_0.bin from this release, save it as kestrel_net.bin in the folder where you run the compiler, then for example

g++ -std=c++17 -O3 -march=x86-64-v3 -static -pthread -DNDEBUG -o kestrel.exe path/to/src/kestrel_3_0.cpp

The network is not stored in the git history (76 MB per version); see docs/BUILDING.md for the other targets.