[Doc] Introduction for FSRS v4

[L-M-Sherlock]

1. Re-writing the wiki. I suggest opening a new issue for that, since there is a lot that needs to be changed. Me and @user1823 will help you. The page about formulas obviously has to be changed. The page about comparing FSRS to Anki should be about RMSE and the universal metric, which leads me to my second point
2. @L-M-Sherlock, please make a special file for benchmarking purposes only. It should contain FSRS v4, v3, LSTM, SM-2 and Memrise algorithms. The first 3 must be optimized, all within the same file, and then the RMSE and universal metrics (against FSRS v4) should be displayed. I'm planning to write 2 reddit posts about FSRS, and I need benchmarks for that. You will need them too, to re-write that page on the wiki that currently just compares interval lengths of FSRS and Anki, which is useless.

Originally posted by @Expertium in #342 (comment)

[L-M-Sherlock]

I have updated the wiki page about the formula of FSRS v4:

https://github.com/open-spaced-repetition/fsrs4anki/wiki/Free-Spaced-Repetition-Scheduler#fsrs-v4

[Expertium]

Some suggestions:

The part about S0 doesn't mention that these parameters are estimated differently than the rest, which is probably worth mentioning.

I don't think "mean reversion" is the right term, but I'm also not sure what to call it. It's not really a moving average either, not a usual one, at least. @user1823, do you have suggestions how to re-write the part about difficulty?

Replace "The retrievability of t days since the last review" with "The retrievability after t days since the last review".

Replace "The new stability after recall" with "The new stability after a successful review (the user pressed "Hard", "Good" or "Easy")".

Replace the 4 points about Sinc:

"Sinc denotes the increase in stability.

1. The larger the value of D, the smaller the Sinc, which means that for difficult material, the increase in memory stability is smaller than for easy material.
2. The larger the value of S, the smaller the Sinc, which means that memory saturates. The more stable your memory is, the harder it is to make it even more stable.
3. The smaller the value of R, the larger the Sinc, which means that the best time to review your material is when you almost forgot it.
4. The value of Sinc is always greater than or equal to 1 if the review was successful."

[Expertium]

And this is for another page. Use this to make a table.

Feature name

How does it work?

When should I use it?

Postpone

Increases the intervals of cards that are due today based on current R, requested R, and interval length in a way that minimizes damage to long-term learning.

When you are dealing with a large number of reviews after taking a break from Anki.

Advance

Decreases the intervals of cards that are due today based on current R, requested R, and interval length in a way that minimizes damage to long-term learning.

When you want to review your material ahead of time, for example, before a test.

Free Days

After the optimal interval is calculated, it is slightly adjusted to change the due date.

If you don't want to study on some days of the week, for example, Sundays.

Disperse Siblings

Siblings are cards generated from the same note. Their intervals are adjusted to spread them further apart from each other.

Always. This feature increases the spacing effect; disabling it will only decrease the efficiency of spaced repetition.

Load Balancing

After the optimal interval is calculated, it is adjusted by a random amount to make the distribution of reviews over time more uniform.

Always. This feature makes your workload (reviews per day) more consistent.

[L-M-Sherlock]

And this is for another page. Use this to make a table.

I have added them in the readme of FSRS4Anki Helper.

https://github.com/open-spaced-repetition/fsrs4anki-helper#overview

[Expertium]

I think you should update the text in the part that is related to FSRS v3, to make them consistent. I think this text should be the same for v4 and v3.

[L-M-Sherlock]

Benchmark: https://colab.research.google.com/github/open-spaced-repetition/fsrs4anki/blob/benchmark/benchmark.ipynb

[Expertium]

Nice. I suggest adding Memrise as well. Check the article that I linked in that comment to ensure that my code is consistent with what the article describes.

[L-M-Sherlock]

Updated.

[Expertium]

Display the total number of parameters of the LSTM. Also, it's hard to tell which calibration graph is related to which model, you should annotate them.

[Expertium]

Also, it crashes due to a lack of RAM again. Decrease n_hidden to 5.

[L-M-Sherlock]

Also, it crashes due to a lack of RAM again.

To set n_hidden=8 or n_hidden=4 would be helpful.

Display the total number of parameters of the LSTM. Also, it's hard to tell which calibration graph is related to which model, you should annotate them.

I updated the code and set titles for calibration graphs.

[Expertium]

Here are the results. Note that

1. I'm using additive smoothing for S0.
2. I've limited S0 to 40.
3. filter_out_suspended_cards = True. I think it should be true by default, since suspended cards are not representative of the kind of material that the user will actually learn once he starts using the algorithm.
4. The number of bins for both RMSE and the universal metric was set to 20. Well, 21 for the universal metric, but whatever.
5. n_epoch and n_splits was set to 5 for v4 and to 3 for v3 for the sake of consistency, since those are the values that were used in the respective versions. n_hidden was set to 8.

I will be using these images for my future reddit posts once v4 is released. You can add them to the wiki, or you can run your own benchmarks.

​​​​
​​​​

[Expertium]

Unrelated, but how come LSTM is being trained faster than FSRS, despite having almost 30 times more parameters?

[L-M-Sherlock]

Unrelated, but how come LSTM is being trained faster than FSRS, despite having almost 30 times more parameters?

Because the official implementation of LSTM is coded by C instead of python.

[L-M-Sherlock]

I will be using these images for my future reddit posts once v4 is released. You can add them to the wiki, or you can run your own benchmarks.

Assuming all goes well, I aim to officially launch v4 this week. Besides, I'm considering on creating an open-source dataset. Do you have any advice?

[Expertium]

I don't really have any advice right now, other than a reminder to run v4 on all collections submitted via Google forms to get good average parameters that will be given to users with <1000 reviews.

[L-M-Sherlock]

Change the titles will also change the links, which have been used by many other people.

[L-M-Sherlock]

• In SInc, Inc should be in subscript.

In https://supermemo.guru/wiki/Stabilization, Woz has used SInc. And $S_{Inc}$ is misleading because SInc is not a type of stability.

[user1823]

Change the titles will also change the links, which have been used by many other people.

I got it. But, link rot is a problem with every website. So, it makes sense to rename the titles if we are making them easier to understand. This would help most of the users who will come to the Wiki in the future.

The only affected ones would be those who come to specific pages of the Wiki using specific links. In fact, even those users would simply be redirected to the home page of the wiki (I just tested this) and then can easily navigate to the specific page they want to visit.

[Expertium]

@L-M-Sherlock would you mind updating the benchmark file so that the code for FSRS in that file matches the code of the latest version of FSRS? And add MAE.

[L-M-Sherlock]

I'm fixing a bug. I will update the benchmark tonight.

[L-M-Sherlock]

• filter_out_suspended_cards = True. I think it should be true by default, since suspended cards are not representative of the kind of material that the user will actually learn once he starts using the algorithm.
• The number of bins for both RMSE and the universal metric was set to 20. Well, 21 for the universal metric, but whatever.
• n_epoch and n_splits was set to 5 for v4 and to 3 for v3 for the sake of consistency, since those are the values that were used in the respective versions. n_hidden was set to 8.

These changes have been integrated into benchmark.ipynb:

https://github.com/open-spaced-repetition/fsrs4anki/blob/benchmark/benchmark.ipynb

[L-M-Sherlock]

• Rename "Auto migrate from SM 2 to FSRS". But I am not sure what to call it. Also, I think that it should be contained within another page. It doesn't deserve a page of its own.

It is only documented for the scheduler. Here is the related code:

fsrs4anki/fsrs4anki_scheduler.js

Lines 228 to 241 in d5e15d1

	function convert_states() {
	const scheduledDays = states.current.normal ? states.current.normal.review.scheduledDays : states.current.filtered.rescheduling.originalState.review.scheduledDays;
	const easeFactor = states.current.normal ? states.current.normal.review.easeFactor : states.current.filtered.rescheduling.originalState.review.easeFactor;
	const old_s = +Math.max(scheduledDays, 0.1).toFixed(2);
	const old_d = constrain_difficulty(11 - (easeFactor - 1) / (Math.exp(w[8]) * Math.pow(old_s, -w[9]) * (Math.exp(0.1 * w[10]) - 1)));
	customData.again.d = old_d;
	customData.again.s = old_s;
	customData.hard.d = old_d;
	customData.hard.s = old_s;
	customData.good.d = old_d;
	customData.good.s = old_s;
	customData.easy.d = old_d;
	customData.easy.s = old_s;
	}

[user1823]

• Rename "Auto migrate from SM 2 to FSRS". But I am not sure what to call it. Also, I think that it should be contained within another page. It doesn't deserve a page of its own.

It is only documented for the scheduler.

I recommend merging that page into this section.

Also, I recommend writing "If the memory states are not available during review…" instead of "If the memory states are lost..."

[L-M-Sherlock]

I drafted up a wiki page for the benchmark: https://github.com/open-spaced-repetition/fsrs4anki/wiki/The-Benchmark

[Expertium]

Also, while this is not the right issue for this, I suggest that in the final Rust version of the optimizer that will be used in Anki, this should be addressed: #351 (comment)

Make the two formulas consistent by adding exp() to the second formula: torch.exp(self.w[11])

[L-M-Sherlock]

I need to run the benchmark for this modification.

[L-M-Sherlock]

@L-M-Sherlock a reminder to do this: #351 (comment)

Could you help me write an explanation?

[Expertium]

Could you help me write an explanation?

"Mature cards are defined as cards with an interval of 21 days or longer."

[user1823]

@L-M-Sherlock a reminder to do this: #351 (comment)

Could you help me write an explanation?

Let me create a PR

[L-M-Sherlock]

Make the two formulas consistent by adding exp() to the second formula: torch.exp(self.w[11])

I find out a problem during the benchmark testing. Sometime w[11] will be negative. For example, if the original w[11] < 1, the new w[11] will < 0.

[Expertium]

Why is it a problem?

[L-M-Sherlock]

In v3, some weights are negative. In v4, we try to keep all weights postive, right?

[Expertium]

Ah, that. Then I suggest testing whether this change affects RMSE and running statistical significance tests. If this change decreases RMSE, let's accept that one parameter can be negative. If it makes no difference, let's keep the current formula.

[L-M-Sherlock]

Done in 30 minutes.

Algorithm	Log Loss	RMSE	RMSE(bins)
FSRS v4 old	0.3830	0.3311	0.0543
FSRS v4 new	0.3830	0.3311	0.0540

The improvement in RMSE(bins) is less than 0.5%

[user1823]

Does this mean that using exp to w[11] and also not allowing negative values resulted in a smaller RMSE (though not significantly small)?

If so, it makes sense to use exp.

[Expertium]

Does this mean that using exp to w[11] and also not allowing negative values resulted in a smaller RMSE (though not significantly small)?

Sherlock should run the statistical significance test to answer this.

[L-M-Sherlock]

Does this mean that using exp to w[11] and also not allowing negative values resulted in a smaller RMSE (though not significantly small)?

I didn't clip w[11] in the test. And p = 0.96. It's not statistically significant.

[Expertium]

Well, forget about it then. Let's keep the current formula.

[Expertium]

@L-M-Sherlock I believe this is misleading

First of all, you can set the interval modifier above 1.00, but not the requested retention. This alone makes that statement dubious at best. Second of all, it can be interpreted by a user as "I can set the interval modifier in Anki instead of setting the requested retention in FSRS and they will do exactly the same thing", which is not true.
@user1823 do you have an idea how to rewrite it?

[user1823]

@user1823 do you have an idea how to rewrite it?

I wrote this a long time back: #191 (comment)

[Expertium]

@L-M-Sherlock I think it's best to just remove that line (see above) about the interval modifier entirely.

[Expertium]

I wrote a simple Python code to check new posts on r/Anki for mentions of FSRS (or to check posts on any subreddit for any string of text). It will check the subreddit every 5 minutes and make a sound alert and display a pop-up notification if the title or the text of the posts contain the string FSRS. And it will also display the title of the new post and the link to it.

Scraper.zip

@L-M-Sherlock, you might be interested. This can help with responding to questions about FSRS quickly; I made it so that I won't have to manually check r/Anki for FSRS-related posts. The only downside is that if you turn your PC off, go to sleep, and then turn it back on, you will have to manually check posts that have been posted while you were asleep. Also, it's inconvenient that you can't copy the link from the notification, so I added a print() statement too.

[Expertium]

The link on wiki still leads to the old benchmark: https://github.com/open-spaced-repetition/fsrs4anki/wiki/The-Benchmark

Perhaps it's time to update it so that it leads to the new benchmark repo?

[L-M-Sherlock]

This issue hasn't been closed:

• Can we remove d99c0ae from the dataset? srs-benchmark#1

If you think the issue has been solved, I will update the wiki page.

[Expertium]

I don't really have much to add to that issue. If the metrics are consistent among benchmarks of SM-15 and SM-17, then I think that's it.
Unrelated, but I'd like you to read this: #461 (comment)

[user1823]

Huge interval for new cards seems to be a common question.

Recent post: https://www.reddit.com/r/Anki/comments/16kufs4/huge_interval_for_new_cards_fsrs/

So, we should probably add this to the FAQs.

In the answer, we can mention:

For many users, SM-2 tends to show new cards at unnecessarily short intervals. So, when users switch to FSRS, they tend to feel that the intervals for new cards are too large. But these larger intervals match the target retention (configured in the scheduler code) better. By using these larger intervals, FSRS can prevent many of the unnecessary reviews that happen when using SM-2.

If you still want to decrease the intervals, you can increase the requestRetention in the scheduler. But note that this will decrease all the intervals, not just the first intervals.

@Expertium, when you come across a question that is already answered on our FAQ page, I recommend you to guide the user to our FAQ page instead of retyping the answer.

Reason: We want more and more people to know about our FAQ page.

Secondly, if you think that the info on the FAQ page is lacking, you can suggest improvements. This way, the FAQ page will eventually contain answers to most of the queries users can have, reducing the number of support requests.

[Expertium]

This way, the FAQ page will eventually contain answers to most of the queries users can have, reducing the number of support requests.

Not exactly. I noticed that a lot of people don't read the FAQ in the first place, so while improving FAQ and adding more questions is good, I don't think that it will reduce the number of support requests; rather, it will make it more helpful to those few users who do read it.

[Expertium]

Btw Sherlock, the script that I shared above (last week) works well, I've been using it for a few days to respond to reddit posts on r/Anki.

[L-M-Sherlock]

Thanks for your tireless replies. By the way, is there anything we need to add to our documentation? If not, I will close this issue.

[Expertium]

I don't think we need to add anything right now, but I wouldn't close this issue just yet, because we will likely have to redo the guide once FSRS is integrated into Anki. Btw, can you give me any estimate on when a new Anki version with FSRS will be released?
Actually, here are some minor suggestions for the helper add-on (it's mostly fixing grammar):
Change "Auto reschedule cards reviewed in other device after sync" to "Auto reschedule cards reviewed on other devices after sync".
Change "Auto reschedule the card after each review" to "Auto reschedule cards after each review".
Change "Auto disperse siblings after sync and review" to "Auto disperse siblings".

[L-M-Sherlock]

I recommend opening a new issue for the document of FSRS Anki integration, because it would require a huge modification and discussion.

For the timing of release, I have not idea. I have completed most of work in the FSRS part. Dae has some other tasks of Anki before the next version.

For the wording of the helper add-on, you can open a new issue in the add-on repo. It's hard for me to track development tasks scattered around hundreds of comments.