Skip to content

Creating my first test 2

Jump to bottom
dav1312 edited this page Oct 16, 2025 · 1 revision

Contributor's Guide

Welcome! This guide outlines the process for contributing code, network updates, and parameter tuning to Stockfish. Following these steps will help make the process smooth for both you and the project maintainers.

Core Principles

Before you start, please familiarize yourself with our core principles.

  • Be Respectful: Always interact politely and kindly with others. See the Open Source Etiquette Guidebook.
  • Join the Community: Participate in the Stockfish Discord server. This is the primary place to communicate with maintainers and other developers.
  • Keep Changes Small: Submit focused, atomic changes. Small, self-contained patches are easier to review, test, and understand.
  • One Idea Per Test: Each test should focus on a single idea. Avoid bundling multiple changes into one patch.
  • Passing Tests ≠ Automatic Merge: Even if your change passes all tests, it still requires a review from the maintainers. Complex patches must provide significant benefits to be considered for merging.

Getting Started (One-Time Setup)

First, you need to set up your development environment. You only need to do this once.

1. Prerequisites

Ensure you have the following installed:

  • A recent C++ compiler
  • Git command line tools
  • A GitHub account
  • A Git client (e.g., GitHub Desktop, GitKraken, or the command line)

2. Fork the Repository

Create your own copy (a "fork") of the official Stockfish repository on GitHub.

  1. Navigate to httpshttps://github.com/official-stockfish/Stockfish.
  2. Click the Fork button in the top-right corner. fork
  3. For more help, see GitHub's guide: Fork a Repo.

3. Clone Your Fork

Create a local copy of your forked repository on your computer.

git clone https://github.com/YOUR-USERNAME/Stockfish.git
cd Stockfish

Important

You must fork the repository first and then clone your fork. Cloning the official repository and trying to push changes will not work.

The Contribution Workflow

This is the main loop you will follow for every contribution.

Step 1: Sync Your Fork

Before starting new work, always sync your local master branch with the official Stockfish master branch. This prevents merge conflicts.

You can do this manually with Git commands, which is the recommended approach:

# Add the official repository as a remote named "upstream" (only needs to be done once)
git remote add upstream https://github.com/official-stockfish/Stockfish.git

# Fetch the latest changes from the official repository
git fetch upstream

# Switch to your local master branch
git checkout master

# Reset your master branch to match the official master
# WARNING: This discards any local changes on your master branch!
git reset --hard upstream/master

# Push the updated master branch to your GitHub fork
git push origin master --force
Alternatively, use this automated script
  1. Save the script below as sync-with-official.sh.
  2. Edit the cd /path/to/your/Stockfish/src line to point to the src directory of your local Stockfish clone.
  3. Run it from your terminal with sh sync-with-official.sh.
#!/bin/sh

# Change directory to the path of the script
cd "${0%/*}"

# !!! EDIT THIS LINE to point to your local Stockfish src directory !!!
cd /path/to/your/Stockfish/src

echo "Syncing local master branch with official-stockfish/master..."

# Add 'upstream' remote if it doesn't exist
git remote add upstream https://github.com/official-stockfish/Stockfish.git 2>/dev/null || true

echo "--> Switching to local 'master' branch..."
git checkout master

echo "--> Fetching latest changes from 'upstream'..."
git fetch upstream

echo "--> Resetting local 'master' to match 'upstream/master'..."
git reset --hard upstream/master

echo "--> Pushing updated 'master' to your GitHub fork ('origin')..."
git push origin master --force

echo "--> Recompiling Stockfish..."
make clean
make build -j
make net

echo "Sync complete."

Step 2: Make Your Changes

Create a new branch for your feature or idea. From there, you can make your changes. The exact process depends on the type of contribution.

Standard Code Changes

This is the most common workflow for functional changes, simplifications, or bug fixes.

  1. Create a New Branch: 
    # Make sure you are on the master branch first
git checkout master
# Create and switch to a new branch
git checkout -b my-new-idea
  2. Edit the Code: Modify the source files in the src directory.
  3. Compile: Follow the Compiling from source guide. The basic command from the src directory is make build -j.
  4. Get the Bench Signature: Run the bench command to get a unique identifier for your changes. 
    ./stockfish bench
    The output will contain a line like Nodes searched : 4190940. The number 4190940 is your signature.
  5. Commit and Push: 
    git commit -am "A clear and descriptive commit message"
git push origin my-new-idea
Testing a New NNUE Net
  1. Upload the Net: Upload your .nnue file to Fishtest. You will need an account. By uploading, you license your network under CC0.
  2. Follow Naming Convention: The network file must be named nn-SHA.nnue, where SHA is the first 12 characters of its sha256sum. 
    # Example command to get the SHA
sha256sum my-net.nnue | cut -c1-12
  3. Create a Branch: Create a new branch for the test.
  4. Update the Code: In src/evaluate.h, change the default value of EvalFileDefaultName to the name of your new net (e.g., "nn-d123456789ab.nnue").
  5. Commit and Push: Commit and push your changes as you would for a standard code change. Do not add an EvalFile UCI option; Fishtest does not support it and will only use the default net.
Tuning with SPSA

SPSA is used to automatically tune evaluation or search parameters.

How to Prepare Your Code for SPSA

  1. Create a Branch: git checkout -b my-spsa-tuning
  2. Expose Variables:
    • Move the variables you want to tune to the global scope of the Stockfish namespace.
    • Remove the const qualifier.
  3. Flag Variables with TUNE: Use the TUNE macro to mark the variables for tuning.
    • Variables must be of type int, Score, or Value.
    • Example: 
      // Original variables
int myKing = 10;
Score myBonus = S(5, 15);

// Add this line after their definition
TUNE(myKing, myBonus);
  4. Set Custom Ranges (Optional): By default, a variable v is tuned in the range [0, 2*v]. You can override this.
    • Set a fixed range: TUNE(SetRange(-100, 100), myKing);
    • Use a lambda for complex ranges: auto myFunc = ...; TUNE(SetRange(myFunc), myVar);
    • Return to default: TUNE(SetRange(-100, 100), myKing, SetDefaultRange, myQueen);
  5. Add Post-Update Functions (Optional): If a function needs to run after parameters are updated, add its name to the TUNE macro. 
    void my_post_update() { /* ... */ }
TUNE(myKing, my_post_update);
  6. Compile and Get Parameters: Compile your code and run ./stockfish. It will print a comma-separated list of parameters. Copy this list.
  7. Get Bench Signature, Commit, and Push: Follow the same final steps as a standard code change.
Technical Details of SPSA in Fishtest

The SPSA algorithm in Fishtest works in a loop of two steps:

  • Evaluation: A mini-match is played using two different values for a parameter: value - Ck and value + Ck.
  • Update: The parameter's value is updated based on the match result: value = value + (Ck * Rk) * (wins - losses).

The Ck and Rk values decrease over time to allow for large adjustments at the beginning and fine-tuning at the end. The Fishtest UI requires you to provide the starting value, min/max clipping range, and the final Ck and Rk values for each parameter. The list you copied in step 6 contains all this information.

Step 3: Run Your Test on Fishtest

Warning

Do not run too many tests at once. Having many active tests significantly reduces the testing throughput for everyone.

Please follow our Testing Methodology when choosing test parameters.

  1. Go to https://tests.stockfishchess.org/tests/run.
  2. Test repository: https://github.com/YOUR-USERNAME/Stockfish
  3. Test branch: The name of your branch (e.g., my-new-idea).
  4. Base repository / branch: Leave these as the defaults (official-stockfish/Stockfish, master).
  5. Test signature: The bench number you got earlier. (If you add Bench: [number] to your commit message, this will be auto-filled).
  6. Info: A short but clear description of your change.
  7. For SPSA Tests: Paste the comma-separated list you copied earlier into the SPSA Parameters box.
  8. Click Submit test.

Step 4: Create a Pull Request (PR)

If your test passes STC and LTC, congratulations! You are ready to create a pull request.

  1. Sync First: Ensure your branch is up-to-date with the official master branch to avoid conflicts.
  2. Go to Your Fork: Navigate to your forked repository on GitHub.
  3. Open the PR: GitHub will usually show a prompt to "Compare & pull request". Click it.
  4. Checklist for a Great PR:
    • Single Commit: Is your PR a single, clean commit? If not, please squash your commits.
    • Clear Commit Message: The message should explain the reasoning behind the patch. This will become the PR description.
    • Link Test Results: Include links to your passed STC and LTC tests on Fishtest.
    • Provide Bench Signature: The last line of your commit message must be either No functional change or Bench: XXXXXXX.
    • Ensure Portability: Push your branch to a helper branch named github_ci on your fork (git push -f origin HEAD:github_ci) to trigger continuous integration checks.
Click to see example commit messages

Example 1: Functional Change

Simplify away nnue scale pawn count multiplier

Removes 2x multipliers in nnue scale calculation along with the pawn count term that was recently reintroduced.

Passed non-regression STC:
https://tests.stockfishchess.org/tests/view/64305bc720eb941419bdf72e
LLR: 2.95 (-2.94,2.94) <-1.75,0.25>
Total: 38008 W: 10234 L: 10021 D: 17753

Passed non-regression LTC:
https://tests.stockfishchess.org/tests/view/6430b76a028b029b01ac9bfd
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 91232 W: 24686 L: 24547 D: 41999

Bench: 4017320

Example 2: Non-Functional Change

Set the length of GIT_SHA to 8 characters

Previously, the length of git commit hashes could vary depending on the git environment. This change standardizes it.

No functional change

Reference: Testing Methodology

Use these guidelines to choose the correct test parameters on Fishtest. For a visual aid, see the graphical SPRT calculator.

Definitions
  • TC (Time Control): The time allotted for games.
  • STC (Short Time Control): 10s + 0.1s increment.
  • LTC (Long Time Control): 60s + 0.6s increment.
  • SPRT (Sequential Probability Ratio Test): The statistical model used to determine if a change is stronger or weaker. SPRT(elo0, elo1) defines the bounds for passing or failing a test.
  • Simplification: A change that makes the code smaller, faster, or easier to understand without losing strength.
Functional Changes (Standard Workflow)

This is the process for nearly all changes intended to improve playing strength.

  1. Run an STC Test: Use the default "STC" test type.
  2. Run an LTC Test: If the STC test passes, run a second test using the "LTC" test type. If the master branch has not changed, you can use the "Reschedule" button on your passed test page.
  3. Submit PR: If both tests pass, you are ready to create a pull request.
Functional Changes (Simplifications & Bug Fixes)

For changes that simplify code or fix bugs, we don't require a strict Elo gain. The goal is to ensure no Elo is lost.

  • Follow the same STC -> LTC workflow as above, but use non-regression SPRT bounds. Manually set elo0 to -1.75 and elo1 to 0.25. This requires a custom game task.
Non-Functional Changes (e.g., Speedups)

Changes that don't alter search logic but improve performance (Nodes Per Second) should first be tested locally.

  • Local Benchmarking: Use a tool like perf on Linux or specialized tools like FishBench/BuildTester on Windows to measure the NPS gain. 
    # Example perf command for reliable measurement
sudo perf stat -r 5 -a -B -e cycles:u,instructions:u ./stockfish bench > /dev/null
  • Fishtest Verification: If the speedup is significant (around 0.5% or more) and the change is not trivial, it should be tested on Fishtest using the standard STC -> LTC workflow to confirm the Elo impact. A speedup translates to Elo gain, so it is tested like any other functional patch.
  • Cleanups and Refactoring: Trivial changes (typos, comments, dead code) that don't change the bench can often be submitted directly via PR without testing. More complex refactoring should be run through non-regression tests to ensure no crashes or unexpected slowdowns occur.

Advanced Topics & Resources

Advanced Fishtest Options

These options on the test creation page should only be used if you have a specific reason.

  • Auto-purge: Toggles the automatic removal of statistically unusual games. Disable this when testing time management.
  • Time odds: Uses different time controls for the base and test branches.
  • Custom book: Uses a custom opening book for tests.
  • Disable adjudication: Prevents games from ending early based on score.
Optional Git Commit Hook

This Git hook automatically builds Stockfish and adds the Bench: XXXXXXX line to your commit message if it finds the keyword autobench.

  1. Save the code below to .git/hooks/commit-msg in your local Stockfish repository.
  2. Make it executable: chmod +x .git/hooks/commit-msg.
  3. Use the word autobench on its own line in your commit message.
#!/bin/sh
set -e
if [ -z "$1" ]; then exit 1; fi
if ! git diff --exit-code --quiet; then
    echo "Working directory is not clean; cannot generate bench" >&2
    exit 0
fi

KEYWORD="autobench"
if ! grep -q -- "$KEYWORD" "$1"; then exit 0; fi

# Build Stockfish from the root directory
(cd src && make -j build)

# Obtain signature
signature=$(./src/stockfish bench 2>&1 | grep "Nodes searched" | awk '{print $4}')

# Replace keyword with bench message
sed -i.bak "/^${KEYWORD}\$/c\Bench: $signature" "$1" && rm "$1.bak"

Useful Resources

stockfishchess.org

Clone this wiki locally