Creating my first test 3

Contributor Guidelines

• Be respectful: Always interact politely and kindly with others. See the Open Source Etiquette Guidebook.
• Keep changes small: Submit focused, atomic changes. Large changes are harder to review and test. Aim for small, self-contained patches that are easy to understand and review.
• One test per idea: Adding to the previous point, each test should focus on a single idea or change. Avoid bundling multiple ideas into one patch. Submit each distinct idea as a separate test.
• Passing tests ≠ Automatic merge: Even if your change passes the tests, it needs review. Complex patches require significant benefits to be considered.
• Join the Discord: Participate in the Stockfish Discord server. This is the place to communicate with moderators and other developers.

Getting Started (One-Time Setup)

1. Prerequisites

To write patches for Stockfish and test them in the framework, you will need:

• A recent C++ compiler
• A working git on your system
• A GitHub account
• A git client on your computer (we recommend GitHub Desktop, for its simplicity)

2. Fork the Repository

Create your own copy (a "fork") of the official Stockfish repository on GitHub.

• Navigate to: https://github.com/official-stockfish/Stockfish
• Click the "Fork" button.
• GitHub Help: Fork a Repo

3. Clone Your Fork

Create a local copy of your forked repository on your computer using your git client.

git clone https://github.com/YOUR-USERNAME/Stockfish.git
cd Stockfish

Important

It is important to fork the repository. Creating a new repository which is a copy of the Stockfish repository will not work, and cloning directly from the official Stockfish repository will not allow you to push your changes.

The Contribution Workflow

Step 1: Sync Your Fork

Before starting any new work, ensure your local master branch is synchronized with the official Stockfish master branch. This prevents conflicts and ensures you're working on the latest code.

You can use Git commands, or the script provided below.

# Add the official repository as a remote named "official" (only needs to be done once)
git remote add official https://github.com/official-stockfish/Stockfish.git

# Fetch the latest changes from the official repository
git fetch official

# Switch to your local master branch
git checkout master

# Reset your master branch to match the official master
# WARNING: This discards any local changes on your master branch!
git reset --hard official/master

# Push the updated master branch to your GitHub fork
git push origin master --force

Alternatively, use this script

1. Save the script below as sync-with-official.sh in a convenient location.
2. Important: Edit the cd ./chess/stockfish/src line inside the script to point to the src directory of your local Stockfish clone.
3. Open a terminal or command prompt, navigate to where you saved the script, and run: sh sync-with-official.sh

#!/bin/sh

# Change directory to the path of the script
# Ensures the script knows where it is relative to your setup
cd "${0%/*}"

# !!! EDIT THIS LINE !!!
# Go to the src directory for Stockfish on your hard drive
# Example: cd /path/to/your/Stockfish/src
cd ./chess/stockfish/src

echo
echo "Syncing local master branch with official-stockfish/master..."
echo

echo "--> Setting up 'official' remote..."
# Add the official repo as a remote named 'official' if it doesn't exist
git remote add     official https://github.com/official-stockfish/Stockfish.git
# Ensure the URL is correct
git remote set-url official https://github.com/official-stockfish/Stockfish.git

echo "--> Switching to local 'master' branch..."
git checkout master

echo "--> Fetching latest changes from 'official'..."
git fetch official

echo "--> Resetting local 'master' to match 'official/master'..."
# WARNING: This discards any local changes on your master branch!
git reset --hard official/master

echo "--> Pushing updated 'master' to your GitHub fork ('origin')..."
# Force push is needed because the history was rewritten by the reset
git push origin master --force

echo "--> Recompiling Stockfish..."
make clean
make build -j
make net

echo
echo "Sync complete."

Step 2: Make Your Changes

Create a new branch for your feature or idea. From there, you can make your changes. The exact process depends on the type of contribution.

Standard Code Changes

This is the most common workflow for functional changes, simplifications, or bug fixes.

1. Create a Test Branch: On your local machine, create a new git branch for your changes.

   # Make sure you are on the master branch first
   git checkout master
   # Create and switch to a new branch
   git checkout -b my-new-idea

2. Edit the Code: Modify the Stockfish source files in the src directory as needed.
3. Compile: Follow the Compiling from source guide. The basic command from the src directory is make build -j.
4. Get the Bench Signature: Run the bench command to get an identifier for your changes.

   ./stockfish bench

   The output will contain a line like Nodes searched : 4190940. The number 4190940 is your signature.
5. Commit and Push:

    git commit -am "Describe your changes here"
    git push origin my-new-idea

Testing a New NNUE Net

1. Upload the Net: Upload your .nnue file to Fishtest. You will need an account. By uploading, you license your network under CC0.
2. Follow Naming Convention: The network file must be named nn-SHA.nnue, where SHA is the first 12 characters of its sha256sum.

   # Example command to get the SHA
   sha256sum my-net.nnue | cut -c1-12

3. Create a Test Branch: On your local machine, create a new git branch for your changes.

   # Make sure you are on the master branch first
   git checkout master
   # Create and switch to a new branch
   git checkout -b my-new-nnue

4. Edit the Code: In src/evaluate.h, change the default value of EvalFileDefaultNameBig or EvalFileDefaultNameSmall to the name of your uploaded net.
5. Compile: Follow the Compiling from source guide. The basic command from the src directory is make build -j.
6. Get the Bench Signature: Run the bench command to get an identifier for your changes.

   ./stockfish bench

   The output will contain a line like Nodes searched : 4190940. The number 4190940 is your signature.
7. Commit and Push:

    git commit -am "Describe your changes here"
    git push origin my-new-nnue

[!WARNING]

Do not add an EvalFile or EvalFileSmall to the Fishtest "Test Options"; Fishtest does not support it.

Tuning with SPSA

SPSA is used to automatically tune parameters.

How to Prepare Your Code for SPSA

1. Create a Test Branch: On your local machine, create a new git branch for your changes.

   # Make sure you are on the master branch first
   git checkout master
   # Create and switch to a new branch
   git checkout -b my-new-spsa

2. Expose Variables:
   • Move the variables you want to tune to the global scope of the Stockfish namespace.
   • Remove the const qualifier from these variables.
3. Flag Variables with TUNE: Use the TUNE macro to mark the variables for tuning. You can add multiple TUNE invocations in different places.
   • Variables must be of type int, Score, or Value. They can be arrays of arbitrary dimensions.
   • Example:

     namespace Stockfish {
       // Original variables
       int myKing = 10;
       Score myBonus = S(5, 15);
     
       // Add this line after their definition
       TUNE(myKing, myBonus);

4. Set Custom Ranges (Optional): By default, a variable v is tuned in the range [0, 2 * v]. You can override this.
   • Set a fixed range: TUNE(SetRange(-100, 100), myKing, SetRange(-20, 20), myQueen); (here myKing is tuned in [-100, 100] and myQueen in [-20, 20]).
   • Use a lambda for complex ranges: auto myFunc = ...; TUNE(SetRange(myFunc), myVar);
   • Return to default: TUNE(SetRange(-100, 100), myKing, SetDefaultRange, myQueen); (here myQueen uses the default range). Note: You can also change the range of each parameter manually as you input them in the Fishtest web form, but it must be within the allowed range.
5. Add Post-Update Functions (Optional): If a function needs to run after parameters are updated, add its name to the TUNE macro. You can add multiple functions and they will be called in the order you add them.

   void my_post_update() { /* ... */ }
   TUNE(myKing, my_post_update);

6. Compile and Get Parameters: Compile your code and run ./stockfish. It will print a comma-separated list of parameters. Copy this list.
7. Get the Bench Signature: Run the bench command to get an identifier for your changes.

   ./stockfish bench

   The output will contain a line like Nodes searched : 4190940. The number 4190940 is your signature.
8. Commit and Push:

    git commit -am "Describe your changes here"
    git push origin my-new-nnue

Technical Details of SPSA in Fishtest

The simultaneous perturbation stochastic approximation (SPSA) algorithm implementation in Fishtest can be (extremely) simplified in a loop based on these two steps:

• Evaluation step: play a mini match using 2 different values of the parameter to be optimized:
  • parameter_value - ck
  • parameter_value + ck
• Update step: update the parameter according to the mini match result:
  • parameter_value = parameter_value + (ck * rk) * (wins - losses)

Where:

• ck: is the variation applied to the parameter_value to detect the right direction of the update from the mini match result. The value must be large enough to have a measurable Elo difference during the mini match

• rk: is the fraction of ck used to update the parameter_value

  ck and rk decrease with the number of iterations (1 iteration = 2 games played) in order to have bigger updates of the parameter_value at the start of the tuning and to settle to a final value at the end of the tuning:

  ck = c0 / (1 + k)**gamma
  ak = a0 / (A + 1 + k)**alpha
  rk = ak / ck**2
  

The Fishtest page for a new SPSA test requires for each parameter:

• The starting value of parameter_value
• The clipping min value, applied on parameter_value and parameter_value - ck
• The clipping max value, applied on parameter_value and parameter_value + ck
• The final value of ck (i.e. when k = total_games / 2 - 1)
• The final value of rk (i.e. when k = total_games / 2 - 1)

The min and max clipping values should establish the broadest feasible range within which the parameter is meaningful. If this range is set too narrowly, it could lead to unwarranted clipping, thereby diminishing the sensitivity of Elo during the mini match. To confine the evolution range for a parameter, simply set a smaller rk value.

You can read more about SPSA in Fishtest at the Issue #535.

Step 3: Run Your Test on Fishtest

Warning

Do not run too many tests at the same time. Having too many active tests running will reduce their internal throughtput (ITP) significantly.

Standard tests

Please follow our Testing Methodology when choosing the test parameters.

1. Go to https://tests.stockfishchess.org/tests/run

2. Test repository: https://github.com/YOUR-USERNAME/Stockfish.

3. Test branch: The name of your branch (e.g., my-new-idea).

4. Test signature: The bench number you got earlier. (If you add Bench: [number] to your commit message, you don't need to insert this information.)

5. Info: A short but clear description of your change.

6. Click "Submit test".

   

SPSA tests

1. Go to https://tests.stockfishchess.org/tests/run

2. Nodestime allows SF to use a budgeted node count management, removing noise introduced by inconsistent hardware speed. If the value you're tuning is not susceptible to change significantly the nps, change Test options to read Hash=128 nodestime=600. With nodestime set to 600 the workers should be able to search at least 600 nodes per milliseconds (that is 0.6 Mnps) per thread which is above the fishtest minimum (540 nodes per millisecond per thread) for a node to be considered too slow.

3. For search parameters, normal TC tuning is usually preferred.

4. Choose the appropriate TC. A short TC is appropriate to get faster approximations or for checking the tune's correctness and parameters. A long TC (60+0.6) is best to get better scaling values. If using regular time control, set the appropriate TC and hash (Hash=16 for 20+0.2, Hash=64 for 60+0.6). If using nodestime, set the time control to 160+1.6 to get the equivalent of 60+0.6 with normal TM. The nodestime games will usually finish long before the time is up, provided that the workers are able to search more nodes per milliseconds than specified in the nodestime parameter; otherwise your games will end on time.

5. Paste the list that you copied into the SPSA Parameters list. This is comma-separated data for parameter name, initial value, minimum, maximum, ck final value, rk final value in that order. Here you can also make manual changes to min and max values for parameters.

   

6. A good tune run exhibits a significant change in the tuned values while minimizing random change. As a general rule, tuning many values at once (like a PSQT table) generates random change; while values that are only rarely used to score a position can have a hard time moving at all. Tweaking the ck value often helps to get a better result, with a higher ck forcing value change and a lower one reducing random change.

7. Make sure your test repo is correct (e.g., https://github.com/yourname/Stockfish)

8. Fill in the info describing your change

9. Click "Submit test".

10. If after a few thousand games the values are barely changing at all, the tuning run is useless and should be stopped. This usually happens when the ck value is too low for the parameter changes to influence the tuning result more than random noise.

Note

If you use the default range with an initial value of 0, the parameter will not be tuned since 2 * 0 and 0 / 2 are both 0, and empty intervals are not tuned.

Note

You cannot modify the number of games for SPSA tests, SPSA hyperparams are based off the initial number of games.

Step 4: Create a Pull Request (PR)

Once you are ready, once the tests with your nice idea have passed and/or you have enough speed data to support your improvements, congratulations: you can now open a pull request against the master branch of official-stockfish.

Guidelines for a great pull request:

• Is my pull request up-to-date with current master?
  • The first thing to do before opening a pull request is to synchronize your master branch with the official master branch.
• Does my pull request consist of a single commit?
  • If your branch has a long history, you may squash the commits and force push to your development branch or consider creating a new branch where you squash your changes together, then open the pull request from this new branch.
• Is my code really really clean?
  • Employ a coding style that is similar to surrounding Stockfish code, remove all spaces on empty lines and trailing spaces everywhere, etc.
• Can I improve the quality of my commit message?
  • Your git commit message should have a high-level description of the patch, explaining the reasoning behind the patch and why it improves on the current code. The pull request comment will automatically be filled with your commit message.
• Do I provide easy ways to check my data?
  • Most changes should also report the results obtained in Fishtest at STC and LTC with links to the test pages.
• What will be the next signature of Stockfish if my patch is accepted?
  • Both the commit message and the pull request comment on GitHub must mention if the patch is a 'No functional change' or changes the search. The last line of the commit message should be either 'No functional change' or 'Bench: XXXXXXX' where 'Nodes searched : XXXXXXX' can be found in the output of a ./stockfish bench invocation.
• Is my patch portable?
  • We have continuous integration testing, which will check for standard conformance and reproducibility of search using various compilers. Test your code on your GitHub repository continuous integration pushing your code on the helper branch github_ci with the command git push -f origin HEAD:github_ci and analyze the continuous integration logs in case of failure.
• How could we continue after the patch?
  • Write a few words in the commit message to offer perspectives for future work. This is often a nice way to get momentum for continuing research on this subject in Stockfish, and maybe somebody else in the community will pick-up the challenge.

Examples:

Simplify away nnue scale pawn count multiplier
Removes 2x multipliers in nnue scale calculation along with the pawn count term that was recently reintroduced.

Passed non-regression STC:
https://tests.stockfishchess.org/tests/view/64305bc720eb941419bdf72e
LLR: 2.95 (-2.94,2.94) <-1.75,0.25>
Total: 38008 W: 10234 L: 10021 D: 17753
Ptnml(0-2): 96, 4151, 10323, 4312, 122

Passed non-regression LTC:
https://tests.stockfishchess.org/tests/view/6430b76a028b029b01ac9bfd
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 91232 W: 24686 L: 24547 D: 41999
Ptnml(0-2): 30, 8721, 27986, 8838, 41

bench 4017320

Set the length of GIT_SHA to 8 characters
Previously, the length of git commit hashes could vary depending on the git environment.

No functional change

You can find more examples in the commit history.

Testing methodology

Definitions

• Hippopotamus: A patch that adds at least 10 lines of code. Please avoid them, if possible, it is often better to make a minimal version of your idea.
• Parameters tweak: Changing the value of some constants in the code. The generated machine code is the same complexity (aiming at the same number of processor instructions).
• Simplification: A way to make the code and the algorithms clearer. Most of the time, a necessary condition is that the number of lines in the source code of Stockfish goes down, or the number of processor instructions in the generated code goes down -- but this is by no means a sufficient condition because it is of course unfortunately possible to lower the number of lines in the code while obfuscating it.
• Bug: A bug that has been discussed in Discord or as an issue in GitHub and confirmed as such by the maintainers. Potential bug-fix solutions shall be first discussed in the server, then tested in the framework.
• TC: Time control, should almost always be STC or LTC, but exceptionally (e.g., testing time management) can be 40/10 or 40/10+0.1 (40 moves in 10s or 10s + increment), or 10+0 (sudden death).
• STC: Short time control (10+0.1)
• LTC: Long time control (60+0.6)
• SMP: Multi-threaded (symmetric multiprocessing) tests
• SPRT(x,y): Sequential probability ratio test (SPRT) test with elo0 = x and elo1 = y

The following recommendations to choose the right parameters can be best understood with the graphical SPRT calculator, which draws nice curves displaying the pass-rate and the average length of runs for various values of SPRT(x,y). When in doubt, stick with the standard test parameters.

There are single-threaded tests (e.g., STC, LTC) and multi-threaded (symmetric multiprocessing, "SMP") tests (e.g., STC SMP, LTC SMP) that run exactly on 8 threads. SMP testing is performed monthly as part of regression testing or pre-release testing of Stockfish.

Functional changes

Standard

We use these for almost all our tests. It's our workhorse, designed to commit only robust patches that almost surely work. Our goal is to reduce to the minimum possibility of regression and to avoid adding unnecessary complexity.

1. Run your test with the "Test Type": STC

   

2. If your STC test passes, create a new test with the "Test Type": LTC

   

   Note: If there have been no new commits to the Stockfish master branch since you created your first test, you can click the "Reschedule" button inside your test so most of the information will be automatically filled for you in the new test.

   

3. If your LTC test passes, congratulations! Create a pull request against the official-stockfish repository, so your changes can be reviewed. Please remember that it is not guaranteed to be committed.

Simplifications

These must be used for all functional changing simplifications, even one-liners, to test if the removal of the code is detrimental to Stockfish's strength.

We try to reject an Elo loss and even a neutral patch can fail -- nevertheless because the code under test is simpler/smaller than the original, we don't require the stricter standard mode. These tests are also used for bug fixes and other special cases, but only after being discussed in Discord and approved in advance to avoid people testing with non-regression mode becoming their preferred toy, instead of using the stricter standard mode.

For the most part, follow the same procedures as for a standard test but changing the SPRT bounds:

Scalers

If you think that your patch will perform better in very long time controls than in shorter ones you might be able to test it at longer time controls than our standard LTC.

• Time control: 180s + 1.8s, Threads: 1.
• Time control: 60s + 0.6s, Threads: 8 or 70s + 0.7s, Threads: 7. Depending on the availability of 8-thread workers.

Unions

Among parameter tweaks, a special sub-case is the so-called union patch or combo patch, which is a bundling of patches that failed SPRT but with positive or near positive scores. Sometimes retesting the union as a whole passes SPRT. Due to the nature of the approach and because each patch failed already, a union has some constraints:

• Maximum 2 patches per union.
• Each patch shall be trivial, like a parameter tweak. Patches that add/remove a concept/idea/feature shall pass individually.

Non-functional changes

If your patch is a non-functional change, you might still need to run it through Fishtest, but there are exceptions:

Cleanups

Usually, there is an open PR to collect small cleanups. They are trivial, don't change the bench, and are generally non-controversial. This might be typos, variable names, dead code, and similar.

Refactoring and non-functional simplifications

Code refactoring and non-functional simplifications are a very wide family of patches and, by their nature, more subjective than other kinds of patches. So also the acceptance guidelines rely more on maintainer knowledge, experience, and sensibility.

Anything not fitting the above small cleanup category should typically be tested on Fishtest unless the code is not really exercised on Fishtest (e.g. syzygy) and doesn't change bench. This approach verifies code correctness (no crashes), and makes sure there are no unexpected side effects or slowdowns.

• Regression tests will use non-regression SPRT bounds or exceptionally other bounds according to maintainer judgment.
• Rejects are always possible if the patch is worse than the original.

These kinds of patches, although very important for long-term code quality, are also the ones that can raise discussions because code style is subjective in large parts. So be prepared to accept a negative judgment by the maintainer: it is not easy for him, indeed it is a hard job for him to judge on these, so please do not take it personally or start an endless discussion in case your patch is rejected. Simply move on to your next Elo-winning idea.

Speedups

Test it on your machine by running ./stockfish bench. It is recommended to run it several times and compute a mean and stdev to see if the improvement is statistically significant. For Linux/Windows you can use psybench, for Windows you can use the specialized tools FishBench and BuildTester, both excellent.

In case you have access to Linux, the most reliable way is to use the amazing perf tool:

$ sudo perf stat -r 5 -a -B -e cycles:u,instructions:u ./stockfish bench > /dev/null

This command will run the bench 5 times, counting instructions and cycles and averaging. At the end, a report will be printed:

 Performance counter stats for 'system wide' (5 runs):

    22.747.981.856      cycles:u                                     ( +-  0,05% )
    28.409.592.052      instructions:u     #    1,25  insn per cycle ( +-  0,06% )

      4,331400608 seconds time elapsed                               ( +-  0,11% )

Important

Note the very small error, in case you get a sensibly bigger one, please run the test twice. If you run Linux behind a VM, like VMWare, you have to enable performance counters in virtual machine settings.

If the code is a trivial change, send us a pull request. Speedups need to be further verified, for at least 2 reasons:

1. The speedup needs to be statistically significant and not just random noise.
2. The speedup needs to be confirmed on different machines. Sometimes a speedup on one machine is a slowdown on another.

To be considered for inclusion, the speedup should be around 0.5%. If the patch is more complex, then the patch will go under normal STC+LTC Fishtest tests. This will require about 0.25% speedup at STC and about 0.7% speedup at LTC for a 50% passing chance. 1% speedups pass with an 85% chance. The rationale is that a speed-up is totally comparable to a normal patch: it adds complexity with the aim to improve Elo, so it makes sense to test under the same conditions. Some data and discussion in the following issue: https://github.com/official-stockfish/Stockfish/issues/2593

Advanced options

The section advanced options in the test creation page contain options that should be toggled on/off only by advanced users. Currently, the advanced options are:

• Auto-purge: Toggles auto-purge on and off. Having it off can be beneficial when testing for time-management patches or for patches affecting different OS in different manners.
• Time odds: Use different time controls for the test and the base.
• Custom book: Use a custom opening book for tests.
• Disable adjudication: Disables adjudication for testing.

Useful Resources

• Zugzwang Test Suite must be used on patches that affect verification search.
• More Bench Positions

Git Hooks

Place the following the file into .git/hooks/commit-msg and make it executable with chmod +x .git/hooks/commit-msg. This can automatically build and fill bench if a keyword (default: autobench) is found inside the commit message.

#!/bin/sh

set -e

if [ -z "$1" ]; then
    exit 1
fi

if ! git diff --exit-code --quiet; then
    echo "Working directory is not clean; cannot generate bench" >&2
    exit 0
fi

# Keyword to replace with bench message
ACTIVATION="autobench"

# Look for keyword in commit message to replace
if ! grep -q -- "$ACTIVATION" "$1"; then
    exit 0
fi

# Build Stockfish
cd src
make -j build debug=yes
cd ..

# Obtain signature
signature=`eval "./src/stockfish bench 2>&1" | grep "Nodes searched  : " | awk '{print $4}'`

# Replace keyword with bench message
sed -i "/^${ACTIVATION}\$/c\Bench: $signature" "$1"