Skip to content

Commit

Permalink
Add NNUE evaluation
Browse files Browse the repository at this point in the history
This patch ports the efficiently updatable neural network (NNUE) evaluation to Stockfish.

Both the NNUE and the classical evaluations are available, and can be used to
assign a value to a position that is later used in alpha-beta (PVS) search to find the
best move. The classical evaluation computes this value as a function of various chess
concepts, handcrafted by experts, tested and tuned using fishtest. The NNUE evaluation
computes this value with a neural network based on basic inputs. The network is optimized
and trained on the evalutions of millions of positions at moderate search depth.

The NNUE evaluation was first introduced in shogi, and ported to Stockfish afterward.
It can be evaluated efficiently on CPUs, and exploits the fact that only parts
of the neural network need to be updated after a typical chess move.
[The nodchip repository](https://github.com/nodchip/Stockfish) provides additional
tools to train and develop the NNUE networks.

This patch is the result of contributions of various authors, from various communities,
including: nodchip, ynasu87, yaneurao (initial port and NNUE authors), domschl, FireFather,
rqs, xXH4CKST3RXx, tttak, zz4032, joergoster, mstembera, nguyenpham, erbsenzaehler,
dorzechowski, and vondele.

This new evaluation needed various changes to fishtest and the corresponding infrastructure,
for which tomtor, ppigazzini, noobpwnftw, daylen, and vondele are gratefully acknowledged.

The first networks have been provided by gekkehenker and sergiovieri, with the latter
net (nn-97f742aaefcd.nnue) being the current default.

The evaluation function can be selected at run time with the `Use NNUE` (true/false) UCI option,
provided the `EvalFile` option points the the network file (depending on the GUI, with full path).

The performance of the NNUE evaluation relative to the classical evaluation depends somewhat on
the hardware, and is expected to improve quickly, but is currently on > 80 Elo on fishtest:

60000 @ 10+0.1 th 1
https://tests.stockfishchess.org/tests/view/5f28fe6ea5abc164f05e4c4c
ELO: 92.77 +-2.1 (95%) LOS: 100.0%
Total: 60000 W: 24193 L: 8543 D: 27264
Ptnml(0-2): 609, 3850, 9708, 10948, 4885

40000 @ 20+0.2 th 8
https://tests.stockfishchess.org/tests/view/5f290229a5abc164f05e4c58
ELO: 89.47 +-2.0 (95%) LOS: 100.0%
Total: 40000 W: 12756 L: 2677 D: 24567
Ptnml(0-2): 74, 1583, 8550, 7776, 2017

At the same time, the impact on the classical evaluation remains minimal, causing no significant
regression:

sprt @ 10+0.1 th 1
https://tests.stockfishchess.org/tests/view/5f2906a2a5abc164f05e4c5b
LLR: 2.94 (-2.94,2.94) {-6.00,-4.00}
Total: 34936 W: 6502 L: 6825 D: 21609
Ptnml(0-2): 571, 4082, 8434, 3861, 520

sprt @ 60+0.6 th 1
https://tests.stockfishchess.org/tests/view/5f2906cfa5abc164f05e4c5d
LLR: 2.93 (-2.94,2.94) {-6.00,-4.00}
Total: 10088 W: 1232 L: 1265 D: 7591
Ptnml(0-2): 49, 914, 3170, 843, 68

The needed networks can be found at https://tests.stockfishchess.org/nns
It is recommended to use the default one as indicated by the `EvalFile` UCI option.

Guidelines for testing new nets can be found at
https://github.com/glinscott/fishtest/wiki/Creating-my-first-test#nnue-net-tests

Integration has been discussed in various issues:
#2823
#2728

The integration branch will be closed after the merge:
#2825
https://github.com/official-stockfish/Stockfish/tree/nnue-player-wip

closes #2912

This will be an exciting time for computer chess, looking forward to seeing the evolution of
this approach.

Bench: 4746616
  • Loading branch information
nodchip authored and vondele committed Aug 6, 2020
1 parent 9587eee commit 84f3e86
Show file tree
Hide file tree
Showing 59 changed files with 2,470 additions and 241 deletions.
31 changes: 19 additions & 12 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
language: cpp
dist: xenial
dist: bionic

matrix:
include:
- os: linux
compiler: gcc
addons:
apt:
sources: ['ubuntu-toolchain-r-test']
packages: ['g++-8', 'g++-8-multilib', 'g++-multilib', 'valgrind', 'expect', 'curl']
env:
- COMPILER=g++-8
Expand All @@ -17,23 +16,23 @@ matrix:
compiler: clang
addons:
apt:
sources: ['ubuntu-toolchain-r-test', 'llvm-toolchain-xenial-6.0']
packages: ['clang-6.0', 'llvm-6.0-dev', 'g++-multilib', 'valgrind', 'expect', 'curl']
packages: ['clang-10', 'llvm-10-dev', 'g++-multilib', 'valgrind', 'expect', 'curl']
env:
- COMPILER=clang++-6.0
- COMPILER=clang++-10
- COMP=clang
- LDFLAGS=-fuse-ld=lld

- os: osx
osx_image: xcode12
compiler: gcc
env:
- COMPILER=g++
- COMP=gcc

- os: osx
osx_image: xcode12
compiler: clang
env:
- COMPILER=clang++ V='Apple LLVM 9.4.1' # Apple LLVM version 9.1.0 (clang-902.0.39.2)
- COMPILER=clang++
- COMP=clang

branches:
Expand All @@ -48,26 +47,34 @@ script:
- git log HEAD | grep "\b[Bb]ench[ :]\+[0-9]\{7\}" | head -n 1 | sed "s/[^0-9]*\([0-9]*\).*/\1/g" > git_sig
- export benchref=$(cat git_sig)
- echo "Reference bench:" $benchref

#
# Compiler version string
- $COMPILER -v

#
# Verify bench number against various builds
- export CXXFLAGS="-Werror -D_GLIBCXX_DEBUG"
- make clean && make -j2 ARCH=x86-64 optimize=no debug=yes build && ../tests/signature.sh $benchref
- make clean && make -j2 ARCH=x86-32 optimize=no debug=yes build && ../tests/signature.sh $benchref
- make clean && make -j2 ARCH=x86-32 build && ../tests/signature.sh $benchref
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then make clean && make -j2 ARCH=x86-32 optimize=no debug=yes build && ../tests/signature.sh $benchref; fi
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then make clean && make -j2 ARCH=x86-32 build && ../tests/signature.sh $benchref; fi

#
# Check perft and reproducible search
- export CXXFLAGS="-Werror"
- make clean && make -j2 ARCH=x86-64 build
- ../tests/perft.sh
- ../tests/reprosearch.sh

#
# Valgrind
#
- export CXXFLAGS="-O1 -fno-inline"
- if [ -x "$(command -v valgrind )" ]; then make clean && make -j2 ARCH=x86-64 debug=yes optimize=no build > /dev/null && ../tests/instrumented.sh --valgrind; fi
- if [ -x "$(command -v valgrind )" ]; then ../tests/instrumented.sh --valgrind-thread; fi

#
# Sanitizer
#
# Use g++-8 as a proxy for having sanitizers, might need revision as they become available for more recent versions of clang/gcc
- if [[ "$COMPILER" == "g++-8" ]]; then make clean && make -j2 ARCH=x86-64 sanitize=undefined optimize=no debug=yes build > /dev/null && ../tests/instrumented.sh --sanitizer-undefined; fi
- if [[ "$COMPILER" == "g++-8" ]]; then make clean && make -j2 ARCH=x86-64 sanitize=thread optimize=no debug=yes build > /dev/null && ../tests/instrumented.sh --sanitizer-thread; fi
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then make clean && make -j2 ARCH=x86-64 sanitize=undefined optimize=no debug=yes build > /dev/null && ../tests/instrumented.sh --sanitizer-undefined; fi
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then make clean && make -j2 ARCH=x86-64 sanitize=thread optimize=no debug=yes build > /dev/null && ../tests/instrumented.sh --sanitizer-thread; fi
17 changes: 15 additions & 2 deletions AUTHORS
Original file line number Diff line number Diff line change
@@ -1,10 +1,17 @@
# List of authors for Stockfish, as of March 30, 2020
# List of authors for Stockfish, as of August 4, 2020

# Founders of the Stockfish project and fishtest infrastructure
Tord Romstad (romstad)
Marco Costalba (mcostalba)
Joona Kiiski (zamar)
Gary Linscott (glinscott)

# Authors and inventors of NNUE, training, NNUE port
Yu Nasu (ynasu87)
Motohiro Isozaki (yaneurao)
Hisayori Noda (nodchip)

# all other authors of the code in alphabetical order
Aditya (absimaldata)
Adrian Petrescu (apetresc)
Ajith Chandy Jose (ajithcj)
Expand Down Expand Up @@ -36,6 +43,7 @@ Dariusz Orzechowski
David Zar
Daylen Yang (daylen)
DiscanX
Dominik Schlösser (domschl)
double-beep
Eduardo Cáceres (eduherminio)
Eelco de Groot (KingDefender)
Expand Down Expand Up @@ -115,7 +123,8 @@ Nick Pelling (nickpelling)
Nicklas Persson (NicklasPersson)
Niklas Fiekas (niklasf)
Nikolay Kostov (NikolayIT)
Nguyen Pham
Nguyen Pham (nguyenpham)
Norman Schmidt (FireFather)
Ondrej Mosnáček (WOnder93)
Oskar Werkelin Ahlin
Pablo Vazquez
Expand All @@ -135,6 +144,7 @@ Richard Lloyd
Rodrigo Exterckötter Tjäder
Ron Britvich (Britvich)
Ronald de Man (syzygy1, syzygy)
rqs
Ryan Schmitt
Ryan Takker
Sami Kiminki (skiminki)
Expand All @@ -143,6 +153,7 @@ Sergei Antonov (saproj)
Sergei Ivanov (svivanov72)
sf-x
Shane Booth (shane31)
Shawn Varghese (xXH4CKST3RXx)
Stefan Geschwentner (locutus2)
Stefano Cardanobile (Stefano80)
Steinar Gunderson (sesse)
Expand All @@ -155,9 +166,11 @@ Tom Vijlbrief (tomtor)
Tomasz Sobczyk (Sopel97)
Torsten Franz (torfranz, tfranzer)
Tracey Emery (basepr1me)
tttak
Unai Corzo (unaiic)
Uri Blass (uriblass)
Vince Negri (cuddlestmonkey)
zz4032


# Additionally, we acknowledge the authors and maintainers of fishtest,
Expand Down
125 changes: 86 additions & 39 deletions Readme.md → README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,13 @@
[![Build Status](https://ci.appveyor.com/api/projects/status/github/official-stockfish/Stockfish?branch=master&svg=true)](https://ci.appveyor.com/project/mcostalba/stockfish/branch/master)

[Stockfish](https://stockfishchess.org) is a free, powerful UCI chess engine
derived from Glaurung 2.1. It is not a complete chess program and requires a
derived from Glaurung 2.1. It features two evaluation functions, the classical
evaluation based on handcrafted terms, and the NNUE evaluation based on
efficiently updateable neural networks. The classical evaluation runs efficiently
on most 64bit CPU architectures, while the NNUE evaluation benefits strongly from the
vector intrinsics available on modern CPUs (avx2 or similar).

Stockfish is not a complete chess program and requires a
UCI-compatible GUI (e.g. XBoard with PolyGlot, Scid, Cute Chess, eboard, Arena,
Sigma Chess, Shredder, Chess Partner or Fritz) in order to be used comfortably.
Read the documentation for your GUI of choice for information about how to use
Expand All @@ -22,20 +28,19 @@ This distribution of Stockfish consists of the following files:
* src, a subdirectory containing the full source code, including a Makefile
that can be used to compile Stockfish on Unix-like systems.

To use the NNUE evaluation an additional data file with neural network parameters
needs to be downloaded. The filename for the default set can be found as the default
value of the `EvalFile` UCI option, with the format
`nn-[SHA256 first 12 digits].nnue` (e.g. nn-c157e0a5755b.nnue). This file can be downloaded from
```
https://tests.stockfishchess.org/api/nn/[filename]
```
replacing `[filename]` as needed.

## UCI parameters

Currently, Stockfish has the following UCI options:

* #### Debug Log File
Write all communication to and from the engine into a text file.

* #### Contempt
A positive value for contempt favors middle game positions and avoids draws.
## UCI options

* #### Analysis Contempt
By default, contempt is set to prefer the side to move. Set this option to "White"
or "Black" to analyse with contempt for that side, or "Off" to disable contempt.
Currently, Stockfish has the following UCI options:

* #### Threads
The number of CPU threads used for searching a position. For best performance, set
Expand All @@ -44,50 +49,51 @@ Currently, Stockfish has the following UCI options:
* #### Hash
The size of the hash table in MB. It is recommended to set Hash after setting Threads.

* #### Clear Hash
Clear the hash table.

* #### Ponder
Let Stockfish ponder its next move while the opponent is thinking.

* #### MultiPV
Output the N best lines (principal variations, PVs) when searching.
Leave at 1 for best performance.

* #### Skill Level
Lower the Skill Level in order to make Stockfish play weaker (see also UCI_LimitStrength).
Internally, MultiPV is enabled, and with a certain probability depending on the Skill Level a
weaker move will be played.
* #### Use NNUE
Toggle between the NNUE and classical evaluation functions. If set to "true",
the network parameters must be availabe to load from file (see also EvalFile).

* #### UCI_LimitStrength
Enable weaker play aiming for an Elo rating as set by UCI_Elo. This option overrides Skill Level.
* #### EvalFile
The name of the file of the NNUE evaluation parameters. Depending on the GUI the
filename should include the full path to the folder/directory that contains the file.

* #### UCI_Elo
If enabled by UCI_LimitStrength, aim for an engine strength of the given Elo.
This Elo rating has been calibrated at a time control of 60s+0.6s and anchored to CCRL 40/4.
* #### Contempt
A positive value for contempt favors middle game positions and avoids draws,
effective for the classical evaluation only.

* #### Analysis Contempt
By default, contempt is set to prefer the side to move. Set this option to "White"
or "Black" to analyse with contempt for that side, or "Off" to disable contempt.

* #### UCI_AnalyseMode
An option handled by your GUI.

* #### UCI_Chess960
An option handled by your GUI. If true, Stockfish will play Chess960.

* #### UCI_ShowWDL
If enabled, show approximate WDL statistics as part of the engine output.
These WDL numbers model expected game outcomes for a given evaluation and
game ply for engine self-play at fishtest LTC conditions (60+0.6s per game).

* #### Move Overhead
Assume a time delay of x ms due to network and GUI overheads. This is useful to
avoid losses on time in those cases.

* #### Slow Mover
Lower values will make Stockfish take less time in games, higher values will
make it think longer.

* #### nodestime
Tells the engine to use nodes searched instead of wall time to account for
elapsed time. Useful for engine testing.
* #### UCI_LimitStrength
Enable weaker play aiming for an Elo rating as set by UCI_Elo. This option overrides Skill Level.

* #### UCI_Chess960
An option handled by your GUI. If true, Stockfish will play Chess960.
* #### UCI_Elo
If enabled by UCI_LimitStrength, aim for an engine strength of the given Elo.
This Elo rating has been calibrated at a time control of 60s+0.6s and anchored to CCRL 40/4.

* #### UCI_AnalyseMode
An option handled by your GUI.
* #### Skill Level
Lower the Skill Level in order to make Stockfish play weaker (see also UCI_LimitStrength).
Internally, MultiPV is enabled, and with a certain probability depending on the Skill Level a
weaker move will be played.

* #### SyzygyPath
Path to the folders/directories storing the Syzygy tablebase files. Multiple
Expand All @@ -114,6 +120,47 @@ Currently, Stockfish has the following UCI options:
Limit Syzygy tablebase probing to positions with at most this many pieces left
(including kings and pawns).

* #### Move Overhead
Assume a time delay of x ms due to network and GUI overheads. This is useful to
avoid losses on time in those cases.

* #### Slow Mover
Lower values will make Stockfish take less time in games, higher values will
make it think longer.

* #### nodestime
Tells the engine to use nodes searched instead of wall time to account for
elapsed time. Useful for engine testing.

* #### Clear Hash
Clear the hash table.

* #### Debug Log File
Write all communication to and from the engine into a text file.

## classical and NNUE evaluation

Both approaches assign a value to a position that is used in alpha-beta (PVS) search
to find the best move. The classical evaluation computes this value as a function
of various chess concepts, handcrafted by experts, tested and tuned using fishtest.
The NNUE evaluation computes this value with a neural network based on basic
inputs (e.g. piece positions only). The network is optimized and trained
on the evalutions of millions of positions at moderate search depth.

The NNUE evaluation was first introduced in shogi, and ported to Stockfish afterward.
It can be evaluated efficiently on CPUs, and exploits the fact that only parts
of the neural network need to be updated after a typical chess move.
[The nodchip repository](https://github.com/nodchip/Stockfish) provides additional
tools to train and develop the NNUE networks.

On CPUs supporting modern vector instructions (avx2 and similar), the NNUE evaluation
results in stronger playing strength, even if the nodes per second computed by the engine
is somewhat lower (roughly 60% of nps is typical).

Note that the NNUE evaluation depends on the Stockfish binary and the network parameter
file (see EvalFile). Not every parameter file is compatible with a given Stockfish binary.
The default value of the EvalFile UCI option is the name of a network that is guaranteed
to be compatible with that binary.

## What to expect from Syzygybases?

Expand Down
17 changes: 10 additions & 7 deletions appveyor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,9 @@ clone_depth: 50
branches:
only:
- master
- appveyor

# Operating system (build VM template)
os: Visual Studio 2017
os: Visual Studio 2019

# Build platform, i.e. x86, x64, AnyCPU. This setting is optional.
platform:
Expand Down Expand Up @@ -36,8 +35,11 @@ before_build:
$src = $src.Replace("\", "/")
# Build CMakeLists.txt
$t = 'cmake_minimum_required(VERSION 3.8)',
$t = 'cmake_minimum_required(VERSION 3.17)',
'project(Stockfish)',
'set(CMAKE_CXX_STANDARD 17)',
'set(CMAKE_CXX_STANDARD_REQUIRED ON)',
'set (CMAKE_CXX_EXTENSIONS OFF)',
'set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_SOURCE_DIR}/src)',
'set(source_files', $src, ')',
'add_executable(stockfish ${source_files})'
Expand All @@ -51,10 +53,11 @@ before_build:
$b = git log HEAD | sls "\b[Bb]ench[ :]+[0-9]{7}" | select -first 1
$bench = $b -match '\D+(\d+)' | % { $matches[1] }
Write-Host "Reference bench:" $bench
$g = "Visual Studio 15 2017"
If (${env:PLATFORM} -eq 'x64') { $g = $g + ' Win64' }
cmake -G "${g}" .
Write-Host "Generated files for: " $g
$g = "Visual Studio 16 2019"
If (${env:PLATFORM} -eq 'x64') { $a = "x64" }
If (${env:PLATFORM} -eq 'x86') { $a = "Win32" }
cmake -G "${g}" -A ${a} .
Write-Host "Generated files for: " $g $a
build_script:
- cmake --build . --config %CONFIGURATION% -- /verbosity:minimal
Expand Down
Loading

0 comments on commit 84f3e86

Please sign in to comment.