Repository navigation
Replies: 4 comments 5 replies
|
@TBergh98 -- I think this would be a wonderful addition to scidCommunity! I will review in more detail and get back to you with thoughts on integration. |
|
@TBergh98 -- I've taken a first step to incorporate your .obr reader only (your Item #1) by creating a PR at #467 This is your code with a minor change:
My proposal would be to continue incorporating this by adding "translate" tags and translations for the new Book-window column labels and new UI strings in book.tcl. Does that sound ok with you? Regarding future work, I am interested in the CTG - .obr converter implemented in tcl and C++ (I don't want to have to require users to have Python installed) if we can do it in a license-compatible way with scidCommunity's existing status. Thank you again for sharing your work! |
|
Thanks a lot, the changes look good to me. The status-3 fix is a nice catch. Adding translate tags for the new book.tcl strings makes sense. Do you want me to do it? I can push it to a branch you can pull into #467, or I can leave it to you if you prefer. As for the CTG converter, I’d be glad to work on a Tcl/C++ port. I’ll check the licensing on my side first and then get back to you with a proposal. |
CTG → .obr converter inside Scid (C++ only)Following up on the .obr reader (#467): this branch can now convert a ChessBase CTG book to an OpenBook Rich (.obr) book by itself. No Python and no external tool are needed. Branch: What it does
Where the code comes fromThe converter is The conversion logic is unchanged in Scid. The only changes are:
It also contains the fix for CTG move code Speed on the Hiarcs 15 book (2.3 GB TestsSame output as the Python converter. The check runs in two steps, both with
So the .obr that Scid writes matches the Python reference. Summary that Scid reports for that conversion: Every one of the 32,330,172 CTG entries is reached from the initial position. Every move decodes to a legal move, and every rebuilt board gives back the Polyglot key of its position. Behaviour inside Scid, scripted against the Scid build through
GUI: the menu entry, the file dialogs, the progress bar and opening the converted book in the Book window were also checked by hand. Builds: I built and tested it on Windows only. Licence: compatible, but binaries become GPL-3
The .obr format itself ( Not done yet
|
Uh oh!
There was an error while loading. Please reload this page.
Hi! Here is another branch I made, again for my own use:
obr-book-support. It brings ChessBase-style opening books to Scid. You load a CTG book (converted to an open format) and the Book window shows games, scores, ratings and the author's colours for every move, as ChessBase does. Details below.Summary
This branch adds read-only support for OpenBook Rich (
.obr), an open, documented opening-book format I wrote. It keeps all the data a ChessBase CTG book stores. With a.obrbook loaded, the Book window shows what ChessBase shows for each move:Clicking the Games, Frequency, Score, W/D/B, Elo or Perf header sorts the moves by that column. An arrow marks the sort column, and a second click reverses the order. W/D/B puts the move with the best White win rate first; reversed, the move with the best Black win rate comes first.
A CTG book can be converted to
.obrwithout losing any data using my other project, OpenBook Rich. Polyglot.binbooks work exactly as before.The
.obrbook (left panel) next to the same CTG book converted to Polyglot with jja (right panel).Why I did this
I wanted Scid to give the same opening-book experience as a CTG book in ChessBase. Years ago the lack of this was one of the reasons I stopped using Scid.
In ChessBase, a CTG book shows for each move how often it was played, how it scored, the players' average rating and performance, and the author's marks. Scid's Book window reads Polyglot books, and a Polyglot book can only give a move and a weight in %.
This is the ChessBase view I used as a reference for the layout and the user experience:
Converting CTG to Polyglot does not solve this, because the data does not fit. A Polyglot entry has room for a 16-bit weight and a 32-bit "learn" field. Win/draw/loss counts, ratings, performance and colours are lost.
So I wrote a format that keeps everything, a converter, and a reader for Scid.
(Besides, while converting the book I noticed that the
.binproduced by jja 0.9.1 seemed to be missing some positions, about 12% of those with moves, mostly positions without castling rights where the king of the side to move is on files a–d, and the lines after them. I am not sure yet: I would need more tests to rule out a mistake on my side. I mention it only as something to keep in mind when converting CTG books to Polyglot.)The
.obrformat in shortThe full specification is in
docs/obr_format.mdon the branch (CC0). A reference reader of about 80 lines of Python (MIT) is in the OpenBook Rich repo.Validation on Hiarcs 15 (the only book tested)
I checked six positions by eye against ChessBase: the initial position, 1.e4, the Najdorf, the Queen's Gambit, an English position reached through an alias, and a mirrored Dragon position. Moves, games, percentages, symbols and colours all match.
Disk space and memory
All files below are the same Hiarcs 15 book.
.ctg+.cto+.ctb).obr.binexported from.obr.binmade by jja 0.9.1.obris larger than the CTG it comes from: aliases. Without them,.obrtakes about 56 bytes per book entry against about 71 for CTG (1.81 GB in total, 21% smaller). CTG stores each position only in a normalised form and computes the equivalent positions (colours reversed, mirrored) at lookup time, so they cost no space. OBR version 1 instead lists them explicitly: 85.5 million aliases of 12 bytes, 1.03 GB, i.e. 36% of the file. A future version could normalise at lookup time like CTG (at most 4 binary searches per lookup) and drop the alias table. I kept version 1 simple..bin: each one becomes a Polyglot entry, so the full.binis only about a third smaller than the.obr, and holds none of the statistics.seek/read(a second one in the aliases if needed), plus one small read per book move. Memory use does not depend on the size of the book.What the branch changes
About 965 lines of code (plus the 158-line spec). No new dependencies.
src/polyglot/obrbook.{h,cpp}(new).obr, validates the header (rejects empty or truncated files) and looks up a position, including through aliases.src/polyglot/main.cpp,src/polyglot.h.obr, detected by its magic number.sc_book movesreturns the same format as for Polyglot books, so every existing caller works with.obrtoo.src/tkscid.cppsc_book obrmoves slotreturns the statistics.sc_book updateandmovesupdaterefuse.obrbooks, which are read-only.tcl/windows/book.tcl.obrfiles appear in the book list. For a.obrbook the panel shows Move, Games, Frequency, Score, a W/D/B bar (modelled on the game list statistics bar of Scid on SourceForge,::windows::gamelist::updateStats_), Elo and Perf, with move colours and NAGs. Clicking a column header sorts by that column (Games by default; W/D/B sorts by White or, reversed, Black win rate; an arrow shows the sort column and direction). Hovering a move shows its details and clicking plays it..binbooks look and behave as before.gtest/test_obrbook.cppMakefile.vc,gtest/CMakeLists.txtobrbook.cpp.Move colours follow the rule I checked against ChessBase on Hiarcs 15:
?,??,!?,?!or "worst move"Disclaimers
Co-Authored-Byline.0xA4, never occurs in it, so its colour is unverified. The meaning of the CTG move flags (bits 5–7) is not publicly documented, so they are stored verbatim and not interpreted..obrspec is CC0 and the Scid reader on this branch contains no CTG code. In OpenBook Rich the.obrreader and the.binexporter are MIT. The CTG converter is GPL-3.0-or-later because its move table comes from jja. Scid is GPL-2.0-or-later, so the two licences are compatible. A converter shipped inside Scid could keep that table (the combined program would then be distributed under GPL-3), or the table could be rebuilt from Gunderson's specification.How this could be integrated
IF you think this could be useful in scidCommunity, these are the ways I can see to do it. Just to give an idea; any other approach is fine with me too:
.obrreader only (this branch, ready to review). Scid reads.obr, and users convert their CTG books with OpenBook Rich. This is the smallest change: it adds no CTG code to Scid and is the cleanest option. The downside is that users need Python, more than 80 minutes for the conversion (the tree walk alone takes about 80), and 2.8 GB of extra disk space for a book the size of Hiarcs 15..ctg/.cto/.ctbdirectly: no conversion and no extra disk space. This is not implemented yet. I would port the CTG decoding and index lookup to C++. It brings code for an undocumented proprietary format into Scid..obrreader. The user picks a CTG book from a Tools menu entry and Scid writes the.obrfile. The converter is in Python today. I am available to port it to C++, with the user interface in Tcl/Tk like the rest of Scid.Any feedback on the code, the format or the user interface is welcome!
All reactions