Structure application architecture as a pipeline

[jonprairie]

First off: awesome project! I've been using an ad-hoc process to study chess with anki but I've never been totally happy with it. I think this utility has a lot of potential so thanks for putting it together!

My suggestion is to rework the architecture and move to a pipeline-style application with a controller and three distinct phases:

1. sources - build a pgn file with games
2. transformers - take a pgn file of games and build a pgn of "snippets" (pgn syntax but with a SetUp tag of 1, a FEN starting position and a few moves played out. I put an example at the bottom of this comment)
3. ankifier - take a pgn file of "snippets" and build anki cards

The lichess integration would be a "source"; integrations with chess.com, chessgames or chess24 could be other sources. Maybe interfacing with a Chessbase install could be another etc.

The mistakes/openings/tactics functionality would all be examples of various types of "transformers".

The ankifier would simply take these generated snippets and create anki cards.

Then the controller would be responsible for parsing the command line, handling all the state/config (eg api credentials and last downloaded game timestamp for a source), and gluing it all together so the user only has to type something like:

chessli lichess mistakes

to get their recent games from lichess downloaded/analyzed/ankified, instead of running each component in the pipeline on its own.
Or maybe

chessli chessgames:tid=54397 openings

to get cards on the openings used in the Fischer/Spassky match ankified from https://www.chessgames.com/perl/chess.pl?tid=54397

My reasoning for all this is that separating the full program into distinct pipeline phases with no shared state would allow easier contributions and extensions in the future. For instance, I could imagine writing up another transformer for memorizing games. It could take a pgn file and split it up into overlapping, 4-move snippets (moves 1-4 in the first pgn snippet, 3-6 on the next etc.) and with this architecture I could plug it right in and immediately be able to pull games from any of the included sources and make anki cards with zero additional effort. Likewise, if I want to add a source I don't have to build the entire pipeline, just getting a pgn with games is enough. I would also have the option to skip phases as needed, like if I already have a database of endgame positions as pgn "snippets" I can just plug it directly into the ankifier.

Anyways, this is just a thought, I don't know how far you want to take the utility or what use-cases you want to support, or if you even want to support it long-term lol. I just saw this and got excited for what it could be x). Feel free to delete/ignore this.

An example of a snippet might be: (Note, Lichess uses the [Variant] tag instead of the [SetUp] and [FEN] tags, maybe both styles should be supported)

[Event "F/S Return Match"]
[Site "Belgrade, Serbia JUG"]
[Date "1992.11.04"]
[Round "29"]
[White "Fischer, Robert J."]
[Black "Spassky, Boris V."]
[Result "1/2-1/2"]
[Variant "From Position"]
[FEN "rnbqkbnr/pp2pppp/3p4/2p5/4P3/5N2/PPPP1PPP/RNBQKB1R w KQkq - 0 3"]

3. d4 cxd4 4. Nxd4 Nf6 5. Nc3

[pwenker]

Hey, thank you for your helpful input! I very much like your ideas! :)

I agree that the current architecture isn't optimal. Part of the reason is that the codebase originated from a collection of hacky scripts. I did some refactoring when I open-sourced the project, but only as much as needed to make it work in this first iteration.

I will follow up in the next days on your ideas!
I think I it would be a good idea for a version 0.3. So the second next version update.

[pwenker]

Hey, while I appreciate your ideas, I decided after some deliberation that I won't (at least in the near future) support other backends than lichess.
The reason is that I don't know of any features that lichess doesn't have (especially as it is now really polishing up its tactics features) and I want to limit the feature-set to the minimally required in order to reduce maintenance and development efforts.
I'm open to any other ideas that require less heavy refactoring, however! And feel free to open up this issue again, if you think I shouldn't abandon this issue so soon.

[jonprairie]

Yeah that's completely understandable. I just wanted to clarify real quick though that I didn't mean to suggest that you be the one to create all those sources/transformers, just that there be a strictly-defined data interface between the various stages of the "pipeline". If you do that and provide a reference implementation using Lichess then it would make it easier for other people to come along and contribute their own sources (or transformers) while still being able to leverage everything that's already been built.

ie If the Lichess source is only responsible for returning PGNs then all I have to do to contribute my own chess.com source is write a function that returns PGNs from chess.com. Everything else from that point on will be handled since the later stages of the pipeline would already know how to process those PGNs.

Same thing with transformers. If transformers are only responsible for taking in PGNs and returning PGN snippets then contributing transformers would be a lot easier. I don't have to have special logic for each source, or a special ankify function for my transformer. The sources already return PGNs and the ankifier already processes PGN snippets, so I only have to care about my specific PGN->PGN snippet transformation.

Still understandable if it's not something you want to take on; I just wanted to clarify because I think it would make it easier for outsiders to contribute functionality and easier to maintain long-term for you as well.

[pwenker]

Thanks for the clarifications! That makes sense! I reopen this here and return to my initial answer that I want to tackle this with version 0.3. :D I don't follow a real semantic versioning here, so 0.3 just means for now, the next version where I can spend a considerable amount to do a larger refactoring. I'll notify you when I have a better idea on when this will be.

[pwenker]

Sorry for closing the issue again @jonprairie , but in the last month I realized that I won't be able to find the time for a larger rewrite of this project. Thank you again for the time you took to formulate and share your great ideas and I hope you still can find some value in this project even if I won't be able to remodularize it into your suggested pipeline structure.