Kestrel 3.2

Kestrel 3.2

A smoother time management (+9 ± 5 Elo against 3.1 at short time controls) and four small changes: huge-page advice for the hash table (Linux), a memory limit for Hash (macOS), a time reserve that grows with the thread count, and release builds that no longer pay for switched-off options. The network, the search and the evaluation are those of 3.1: whenever a search is limited by depth, nodes or move time, Kestrel 3.2 plays exactly like 3.1 (the same depth-12 bench, 859,322 nodes; 24 test positions searched to depth 14 and with go nodes 30000 give identical nodes, scores, best moves and principal variations); only a search with a clock (go wtime ...) can differ.

What is new compared with 3.1

• Smoother time management. After every iteration of its iterative deepening the engine decides whether to start the next one. 3.1 used three fixed thresholds (60 % of the planned time for a normal move, 90 % when the best move had just changed or the score fell, 40 % when the best move had been the same for four iterations). 3.2 multiplies the planned time by factors that change continuously: it falls with every iteration that keeps the best move (150 %, 130 %, 110 %, 90 %, 70 %, 60 % ... of the base of 60 %), it grows when the score of this iteration is lower than the one before and when it is lower than the score of the previous move of the game, and it is shortened by 20 % when the best move got more than 90 % of the nodes of the iteration. The planned time itself (about 1/28 of the remaining time plus 3/4 of the increment) and the hard limits are unchanged. Packaged 3.2 against packaged 3.1 with one thread (the Windows x86-64-v3 builds, Hash 64, no book): +11.2 ± 6.2 Elo at 3 s + 0.03 s per game (4,000 games) and +6.2 ± 7.5 at 5 s + 0.1 s (2,000 games). Together +9.2 ± 4.8 Elo (6,000 games). The speed is that of 3.1 (the same depth-12 bench in the same time), so this is the effect of the time management; the same executable with the new rule on or off scored +7.3 ± 6.1 and +0.9 ± 7.5 in two development matches. Games lost on time: 3 s + 0.03 s: 3.2 5, 3.1 9; 5 s + 0.1 s: 3.2 0, 3.1 0. With a fixed move time, a depth or a node limit nothing changes.
• Linux: transparent huge pages for the hash table. Tables of 256 MB or more are advised (madvise) to use 2 MB pages, so that the random probes of many threads miss the TLB less often. It is only an advice and ignored where huge pages are off. I could not measure its effect (I have no Linux machine); Stockfish does the same.
• macOS: the limit of a Hash request. A request above three quarters of the installed memory is cut to that size (with an info string line), as in 3.1 on Windows and Linux; 3.1 did not know the memory size on macOS.
• Release builds compile the switches as constants. The development options (K22Flags, K31Flags, K32Flags, V31Flags, CorrFlags, T_...) are compile-time constants in the release source, so the compiler removes every switched-off idea from the search: with the new switches as ordinary variables the 3.2 exe was 4 % slower than 3.1 in the depth-12 bench (720 ms against 690 ms), with constants it is as fast as 3.1 and 0.5 MB smaller. The options that earlier versions accepted but did not list in uci are now ignored.
• Many threads and Move Overhead. The time that is held back for the communication with the GUI grows by 1 ms for every 8 threads above 8 (15 ms with 128 threads), on top of the Move Overhead option, because stopping dozens of threads takes longer. With 8 threads or fewer nothing changes.

What did not change

The network (the file of 3.0 and 3.1), the search, the evaluation, Syzygy support, every option and the strength with a fixed number of nodes. With one thread and a fixed depth, node count or move time Kestrel 3.2 and 3.1 give the same moves. The strength of 3.0 against 2.3 (+130 ± 8 Elo) and the gain of more threads (+108 to +123 Elo with 8 threads against 1) hold for 3.2; see the notes of 3.0 and 3.1.

Ideas that were tested and are not in 3.2

A history indexed by the pawn structure, a correction history indexed by the previous move, and skipping ProbCut when the transposition table says it cannot work were tested at 20,000 nodes per move against the same build without them (1,300 to 1,700 games each, stopped early): -1.4 ± 13.3, -1.9 ± 14.0 and -7.8 ± 12.5 Elo. The code of the first two and of more correction histories (by the minor and by the major pieces) is in the source as switched-off options of the development builds. A build with Zig/clang was 7 % slower than the g++ build.

Which file do I need?

File	For
Kestrel-3.2-windows-x86-64-v3.zip	Windows 64-bit, AVX2 + BMI2 CPUs (recommended)
Kestrel-3.2-windows-x86-64-v2.zip	Windows 64-bit, SSE4.2 + POPCNT CPUs
Kestrel-3.2-windows-x86-64.zip	Windows 64-bit, any CPU
Kestrel-3.2-linux-x86-64-v3.tar.gz	Linux x86-64, AVX2 + BMI2 CPUs
Kestrel-3.2-linux-x86-64-v2.tar.gz	Linux x86-64, SSE4.2 + POPCNT CPUs
Kestrel-3.2-linux-x86-64.tar.gz	Linux x86-64, any CPU
Kestrel-3.2-linux-arm64.tar.gz	Linux ARM64
Kestrel-3.2-macos-x86-64.tar.gz	macOS, Intel Macs
Kestrel-3.2-macos-arm64.tar.gz	macOS, Apple Silicon (M1 and newer)

The three Windows builds are also attached as plain .exe files (same program as in the zips; a plain exe has no opening book: put claude_book.txt next to it and switch OwnBook on if you want it). kestrel_net_3_2.bin is the network alone (the same file as kestrel_net_3_0.bin and kestrel_net_3_1.bin; needed only to build from source or to load with EvalFile).

Only the x86-64-v3 builds have the AVX2 evaluation code (CPU with AVX2 and BMI2: most PCs since 2015). The v2 and generic builds work on older CPUs but are about 37 % slower.

Status of each build

• Windows: all three builds were run here; each gives the same depth-12 bench (859,322 nodes), so the plain code and the AVX2 code agree exactly, and the x86-64-v3 build plays exactly like the build of the same source with 8 search copies and, for searches without a clock, like the 3.1 build with one thread (24 positions to depth 14, and with go nodes 30000: identical nodes, scores, best moves and principal variations). Both black-box audits pass (83 of 83 and 31 of 31). All 20 test scripts of the repository (tests/) pass on the packaged x86-64-v3 exe, among them tests/breakers.py (807 checks) with 1, 8 and 64 threads and with 16 threads and tablebases (1207 checks), and tests/stress_stop.py with 16, 64 and 128 threads; tests/depth1_test.py passes too (128 threads, go movetime 60: every search searches, no stalemating move). The Syzygy test against python-chess (3-4-5 men): 1,500 positions, 0 mismatches; root filter 1,483 positions, 0 problems; search 60 positions, 0 problems; conversion 30 of 30 won endgames. The sanitizer verification of the release source (GCC and clang AddressSanitizer + UBSan builds, also with 4 threads) passed 30 of 30 checks; an AddressSanitizer + UBSan build of the release source with up to 128 threads passed 16 of 17 steps (thread stress at 2 to 128 threads, hash tests, the UCI fuzzer, tests/breakers.py --quick with 8 and 64 threads, ...); the one step that stops, tests/edge_test.py, stops at the two FEN-parser spots listed under Known limits.
• Linux and macOS: compiled with Zig and checked for file format, but not run where they were made (I have no machine to run them on). The workflow in .github/workflows/verify-release.yml runs the published files on GitHub's Linux, macOS and Windows machines (bench, a UCI session, 128 threads with a 2 GB hash, the Syzygy test with the 3- and 4-men files, on Linux the whole test suite including 64 and 128 threads) and builds the source with AddressSanitizer and UBSan; the 3.1 files passed all 15 jobs there. The macOS Intel build is run by the workflow on GitHub's Intel Mac machine after the release is published (I have no Mac myself).
• The exe is about 82 MB and is not code-signed; some virus scanners are suspicious of big unsigned programs (1.0 had a false positive, see docs/BUILDING.md).

Known limits

• The threads are started and stopped for every search (a thread pool is not implemented): with 128 threads on a 22-processor laptop go movetime 60 took 50 ms in the median but once 328 ms in 60 searches, and about 0.8 s (median) when 24 other programs kept all processors busy; on a computer with 128 processors starting 127 threads takes about 14 ms. With a clock and many threads use a Move Overhead of at least 100 ms.
• The hash size is a power of two (rounded down). Tables of 32 GB and more (2^31 entries) could not be run on the 31 GB computer where this was made; the index arithmetic is 64-bit everywhere.
• Windows before Windows 11 / Server 2022 confines a process to one group of at most 64 logical processors; on a computer with more, threads beyond 64 would share those 64. Not tested. Windows large pages are not used.
• Two undefined-behaviour spots of the FEN parser exist in 3.0 to 3.2 (a FEN with a rank that does not add up to eight files; a halfmove clock of 2^31 or more); they are not changed and matter only to a sanitizer build fed with malformed input.

Building from source

src/kestrel_3_2.cpp (4.6 MB, 128 copies of the search) is generated from src/kestrel_3_2_base.cpp (the file to read: 8 copies) by scripts/make_threads.py; g++ -O3 needs about two and a half minutes and 1.1 GB for it. It embeds the network with .incbin "kestrel_net.bin", which the assembler looks up in the current directory: download kestrel_net_3_2.bin from this release, save it as kestrel_net.bin in the folder where you run the compiler, then for example

g++ -std=c++17 -O3 -march=x86-64-v3 -static -pthread -DNDEBUG -o kestrel.exe path/to/src/kestrel_3_2.cpp

The network is not stored in the git history (76 MB per version); see docs/BUILDING.md for the other targets.