En passant 0%?

[yongyi781]

It seems like all Maia 2 gives extremely low probability to all en passant moves, even when it's quite clearly a move that wins material and quite obvious to see for a human.

e.g. in the position given by FEN rnbqk2r/ppp1n1bp/4p1p1/3pPp2/3P4/2NB1Q2/PPP2PPP/R1B1K1NR w KQkq f6 0 7
picture:

I can see that Maia 2 gives the en passant move exf6 a probability of 0.000008099306796793826.

I seriously doubt that en passants are that rare by humans in the training data. So I believe something is up with the way Maia 2 is trained...

[mcognetta]

I tried to reproduce this bug, but was unable to (I tested this and several other en passant positions with both rapid and blitz models and all combinations of self and opponent elo). The probabilities I found were all at least 0.3 and were often >0.7.

If I had to guess, it is that you actually put in the FEN rnbqk2r/ppp1n1bp/4p1p1/3pPp2/3P4/2NB1Q2/PPP2PPP/R1B1K1NR w KQkq - 0 7 (note that f6 -> -, denoting there is no en passant available). Then, in combination with the bug in #9, the maia models produce probabilities around the magnitude of 1e-7 (for all elo combinations), which is pretty close to what you found (I was not able to reproduce the numbers from your example exactly though).

Could you check the FEN that you input to Maia?

---

Here is an MWE:

from maia2 import model, dataset, inference
from maia2.utils import *
from maia2.main import *

def my_inference_each(model, prepared, fen, elo_self, elo_oppo):
    
    all_moves_dict, elo_dict, all_moves_dict_reversed = prepared
    
    board_input, elo_self, elo_oppo, legal_moves = inference.preprocessing(fen, elo_self, elo_oppo, elo_dict, all_moves_dict)
    
    device = next(model.parameters()).device
    
    model.eval()
    
    board_input = board_input.unsqueeze(dim=0).to(device)
    elo_self = torch.tensor([elo_self]).to(device)
    elo_oppo = torch.tensor([elo_oppo]).to(device)
    legal_moves = legal_moves.unsqueeze(dim=0).to(device)
    
    logits_maia, _, _ = model(board_input, elo_self, elo_oppo)

    # THIS LINE IS BUGGED, SEE https://github.com/CSSLab/maia2/pull/9
    logits_maia_legal = logits_maia * legal_moves
    probs = logits_maia_legal.softmax(dim=-1).cpu().tolist()

    all_move_probs = {}

    black_flag = fen.split(" ")[1] == "b"

    for idx in all_moves_dict_reversed:
        move = all_moves_dict_reversed[idx]
        if black_flag:
            move = mirror_move(move)
        all_move_probs[move] = probs[0][idx]
    
    legal_move_probs = {}
    legal_move_indices = legal_moves.nonzero().flatten().cpu().numpy().tolist()
    legal_moves_mirrored = []
    for move_idx in legal_move_indices:
        move = all_moves_dict_reversed[move_idx]
        if black_flag:
            move = mirror_move(move)
        legal_moves_mirrored.append(move)
    
    for j in range(len(legal_move_indices)):
        legal_move_probs[legal_moves_mirrored[j]] = round(probs[0][legal_move_indices[j]], 4)
    
    legal_move_probs = dict(sorted(legal_move_probs.items(), key=lambda item: item[1], reverse=True))
    
    return legal_move_probs, all_move_probs



maia2_model = model.from_pretrained(type="rapid", device="cpu")
prepared = inference.prepare()
all_moves_dict, elo_dict, all_moves_dict_reversed = prepared

# FEN = "rnbqk2r/ppp1n1bp/4p1p1/3pPp2/3P4/2NB1Q2/PPP2PPP/R1B1K1NR w KQkq f6 0 7"
FEN = "rnbqk2r/ppp1n1bp/4p1p1/3pPp2/3P4/2NB1Q2/PPP2PPP/R1B1K1NR w KQkq - 0 7"

for elo_self in range(1100, 2001, 100):
    for elo_oppo in range(1100, 2001, 100):
        legal_move_probs, all_move_probs = my_inference_each(maia2_model, prepared, FEN, elo_self, elo_oppo)
        print(elo_self, elo_oppo, all_move_probs["e5f6"])

[yongyi781]

The culprit must be frontend, then. I use JS, and borrowed code from maia-platform-frontend.

The issue reproduces on here https://www.maiachess.com/analysis/custom_cd459be52bae/custom -- exf6 is not one of the top human moves.

[mcognetta]

That is what I think as well. There is another comment about this in the discord ( https://discord.com/channels/1275283861890138122/1296691291517878433/1437028166702202955 is this you?) with a similar result when I tested it manually.

[yongyi781]

I am not in the discord, didn't realize there was one!

I appear to get correct en passant probabilities if I change https://github.com/CSSLab/maia-platform-frontend/blob/46584f61a037d30dd68ef174bea24f2927116299/src/lib/engine/tensor.ts#L86 to

const row = rank // Don't invert rank

[mcognetta]

Ah, nice! This backs up my thought that the actual numerical value that you found is from #9 , since it looks like before your fix the model is looking up an invalid move (which will have really small but non-zero probability due to the softmax bug).

But this part of my comment (#7 (comment)) was wrong.

If I had to guess, it is that you actually put in the FEN rnbqk2r/ppp1n1bp/4p1p1/3pPp2/3P4/2NB1Q2/PPP2PPP/R1B1K1NR w KQkq - 0 7 (note that f6 -> -, denoting there is no en passant available).

[kevinjosethomas]

Ah, I think I was debugging this while working on the frontend once. Feel free to make a PR to the frontend repository if you have a fix, or I'll work on it when I'm available. Would appreciate any FEN positions where the issue is reproducible.

[yongyi781]

Made CSSLab/maia-platform-frontend#222.