[Question] Is the offline version ofFSRS in this repo different?

[Expertium]

I noticed that the difference in performance between FSRS offline and online is quite small. Moreover, FSRS offline performs quite poorly, compare to what we saw in the benchmark.
That made me wonder: is the current version of FSRS, the one that is actually used in paractice, different from FSRS offline in this repo?

[L-M-Sherlock]

Moreover, FSRS offline performs quite poorly, compare to what we saw in the benchmark.

Which metrics are poorer than the benchmark?

[L-M-Sherlock]

I use 20 bins for calibration graph. How many bins did you use in the benchmark?

[Expertium]

https://colab.research.google.com/github/open-spaced-repetition/fsrs-vs-sm15/blob/main/compare.ipynb
https://colab.research.google.com/github/open-spaced-repetition/fsrs-vs-sm15/blob/main/compare_offline.ipynb

RMSE and MAE are quite high. In my benchmark data (I believe I used 20 bins, yes) RMSE was below 7.7% in 90% of cases and MAE was below 5.3% in 90% of cases.

[L-M-Sherlock]

My fault. I used 40 bins. It would increase the RMSE and MAE.

[L-M-Sherlock]

By the way, it is good news that the online version's performance approachs the offline version. Because I plan to implement the online version in Anki. The offline version only shows that ceiling of FSRS's capacity of fitting. It is trained in the evaluation data, so the evaluation result of offline version doesn't measure the model's generalization ability.

[Expertium]

What about a hybrid approach?

1. The user exports their collection to Google Colab/Hugging Face/rusn the optimizer locally (hopefully that will be available once the Rust implementation is completed), the optimal parameters are found using offline optimization.
2. The user pastes the optimal parameters into Anki
3. The optimal parameters are updated incrementall after each repetition

[L-M-Sherlock]

The experience replay has completed the work of offline version:

In the part of experience replay, all reviews are used to optimize the parameters. We can also split the reviews and increase the number of epoch there.

[L-M-Sherlock]

We can also allow user to triggle the experience replay manually.

[Expertium]

We can also allow user to triggle the experience replay manually.

That's not a good idea. Most users will have no clue what it means and how to use it. I think that user should control only their retention and max. interval.

[L-M-Sherlock]

OK. Maybe we can set periods for experience replay. Per day/week/month/season or per hundred/thousand reviews.

[Expertium]

Now that I think about it, maybe you should add the option to enable FSRS both globally and for a specific deck individually (that deck will have it's own parameters).
I mean once FSRS is integrated into Anki.

[L-M-Sherlock]

Here is the design by dae: ankitects/anki#2443 (comment)

[Expertium]

Yeah, but he doesn't specify whether users will be able to only have one set of parameters (global), or whether there will be an option have different sets for different decks.

[L-M-Sherlock]

I guess the parameters are bound to the deck options? Like Daily Limits and Advanced.

[L-M-Sherlock]

You can share your suggestion with dae in that issue.