New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compare playing strength with winboard engines #88
Comments
Yes, there's a program to allow UCI and Winboard engines play together. |
Forever ago I had an environment set up for this sort of testing. |
Thanks for your answers. I can let stockfish play normal chess against other engines with the xboard GUI via polyglot, but it does not allow stockfish to play other variants. Is there a way to make it think stockfish is able to play variants or to disable this check? |
Well, set up Winboard for a variant. Then set Dan's version of Stockfish's UCI parameter to the variant of your choice. |
I think there are five solutions:
@stockfishdeveloper No, see this closed issue for how to specify what variant the engine plays. |
I should have been more precise about the problem. It is no problem at all to set the UCI_Variant option. The problem is that polyglot sends the available variants to xboard (in response to protover 2), but only includes normal and chess960. Therefore, xboard does not recognize that stockfish is able to play variants and does not allow to start a variant game with stockfish. I already use (an outdated and modified version of) python-chess to test stockfish in self play. However, as far as I know python-chess does not support the xboard/winboard protocol. I might try Arena even though I am not very familiar with compiling the engines on windows. I found this thread about this issue and your try to fix it. |
Ah, I had completely forgotten that my PolyGlot fork implemented the entire atomic chess ruleset, and for crazyhouse & other variants similar rule changes would need to be made. Honestly, I don't know how to compile for Windows either. I've not used PyChess or CuteChess, but they may also be options? |
I had also tried Pychess, but there the exact same problem occurs that it does not let stockfish play variants. As far as I know cutechess does only support atomic and losers. So I probably have to modify one of PyChess, CuteChess, PolyGlot, and python-chess. The easiest one might be to overwrite the check in the PyChess code, so that it allows Stockfish to play variants. I will think about which one I will give a try. Edit: CuteChess seems to also support Crazyhouse, and King of the Hill and Racing Kings seem to be added soon. I will try it. |
I guess it's going to be a while before variants take off. |
I got it to work with PyChess and CuteChess. Both seem to use UCI_Crazyhouse to detect whether Stockfish it is able to play crazyhouse [Edit: more recent versions also support UCI_Variant]. An additional problem with crazyhouse is that Stockfish (so far) uses lowercase letters to represent black piece drops. I changed it and was able to play games in both PyChess and CuteChess. Now it is getting really interesting. Stockfish played a game with black against each Sjeng and Sunsetter and it won both games convincingly in 24 and 30 moves, respectively. It even announced a mate in 10 in the game against Sunsetter. I will do some more testing to see whether it is really that strong. Here the quite interesting game against Sunsetter:
Edit: After 50 games with a time control of 10 seconds/40 moves, Stockfish is leading 47 - 3 against Sunsetter. I am going to redo the test with the most recent Sunsetter version. |
Excellent! The wins over Sjeng are especially surprising given how active I presume its development has been... I'll ask around for crazyhouse test positions or puzzles. |
According to this rating list, Sunsetter is much better than Sjeng. Result of Stockfish vs. Sunsetter 9: 44 - 5 - 1 (W - L - D) It is exciting to beat one of the top crazyhouse engines by such a huge margin. Now I will try TJchess, which supposedly is stronger than Sunsetter. |
Testing with CuteChess resulted in: I am stunned. I expected Stockfish to be good in Crazyhouse, but to crush the top engines with only very few changes regarding the search and evaluation compared to standard chess is unexpected. @niklasf,@ornicar: I think it might now be interesting to use Stockfish instead of Sunsetter for Crazyhouse. |
This result is plausible because Stockfish move ordering is good; review |
This is indeed astonishing. |
Since simply enabling one more variant for Stockfish should be rather easy, we might as well enable it right away. |
@ianfab @niklasf Would it be better or worse to use uppercase? (I assume uppercase is a standard somehow in the same sense that uppercase is used for piece moves.) |
ultimately it's no big deal, but i believe uppercase (e.g. B@e4) is the way to go. |
I agree that we should use uppercase. I just opened a pull request. |
Merged #91 . |
Congratulations guys, this is excellent ! Stockfish nicely shows the power of collaboration :-). Just think about how much work of how many really excellent programmers went into it. Compare that to for example Sunsetter. Sunsetter was written by 2 people for the most part 10 years ago. While borrowing many concepts of chess programming from different sources we seldom looked at other peoples code. My guess is that each and every part (be it the transposition table, the move ordering, the move generation etc ..) of Stockfish is multiple levels above Sunsetter in terms of speed and quality of programming. Crazyhouse chess doesnt need all that much evaluation knowledge, therefore Stockfish is not lacking much (if any) knowledge. I am very much looking forward to seeing how strong Stockfish will become!! If you are interested and havn't done that yet, here is what I would try to make a former chess engine play crazyhouse chess even better:
Thanks to @niklasf for pointing me to this cool development. Sorry about the very lengthy message, I just got very excited :-). Keep up the good work! |
@georgvonzimmermann: Thanks for your suggestions. The crazyhouse piece values have been tuned with Stockfish's SPSA tuner and I think they already are quite good. The search is subject to tuning sessions and tests (currently razoring, futility and move count pruning) and probably has a lot of room for improvements even though Stockfish's values also work quite well in crazyhouse. |
I have done new tests with an opening book and more games. Conditions Results
Games |
stockfish now plays & analyses crazyhouse on lichess.org. Here's a list of games recently played against level 8: |
Many thanks Georg! We appreciate the kind words. Lichess Master Atrophied suggests that "if you can move, move; else drop". I'm unsure of the context but it might be interesting to see results if: a) piece drops generate last, not first Maybe in some circumstances drops are favorable, based on how much material is in hand? |
Aren't quiet moves and drops sorted by their movepick score anyway? |
…t_20200819 Modify convert_bin_from_pgn-extract
Is there an easy way to let Stockfish play chess variants against winboard engines to be able to compare the playing strength? This would be especially interesting for crazyhouse and antichess.
Or are there any other UCI engines that are able to play variants?
The text was updated successfully, but these errors were encountered: