Grade puzzle difficulty

[ddugovic]

Players would like to see some indicator about how difficult a puzzle is.

Use something like https://gist.github.com/niklasf/73c9565719d124af64ff (included with python-chess) and engine(s) with various settings to derive difficulty scores.

[timezombi]

This has been suggested by multiple people already. I think it is an excellent idea! 😄

Perhaps the simplest way to achieve this is to assign each user and puzzle a glicko rating like lichess does for their Puzzles feature. ChessTempo takes the concept a step further by assigning puzzles 2 separate ratings: Blitz and Standard. The Standard rating works exactly as you would expect: 1-0 for correct answer, 0-1 for incorrect answer. However, the Blitz ratings on ChessTempo take time taken to solve a puzzle into account, assigning a bonus or penalty to correct solutions. Additionally, there is a weight applied for to reduce the impact of correct solutions for puzzles that have been previously solved.

The ChessTempo website explains their rating system thoroughly enough to implement:
https://old.chesstempo.com/user-guide/en/tacticRatingSystem.html

Perhaps some of the work done by lichess can prove helpful. Lichess seems to have some sort of Scala wrapper class of this glicko implementation in Java by goochjs/glicko2.

You can draw some additional inspiration by searching for "Puzzles" and "Glicko" in the ornicar/lila repository to find additional inspiration.

The primary disadvantage of doing this over an automated engine evaluation is that getting accurate ratings of puzzles for a puzzle database this large will require a large enough pool of tsumeshogi solvers solving many, many puzzles! Still, having a rating system with an initial amount of high variance will be significantly better than nothing.

Thoughts?

[ddugovic]

Having (as a Lichess developer myself) looked at the sausage factory which is the Lichess puzzles & rating system, I've become a vegan. Ratings are complex (especially in players' minds, they need ratings to resemble "real life" ratings)... it would be far simpler to take a hint from book publishers and assign grades to problems (maybe on a 1-5 scale, I don't know).

That said, I could also use a script (with engines of various difficulties) to assist with the problem grading task.

[Tellmarch]

Problem collections have a difficulty set from 1 star to 5 stars