Feat/filter outlier in trainset

[L-M-Sherlock]

No description provided.

[user1823]

Can you state in plain English what this outlier filter does?
I am unable to understand it from the code (because I am not a dev) and the added comment also doesn't seem to answer it.

What does rating = rating and delta_t = delta_t actually mean?

[L-M-Sherlock]

Can you state in plain English what this outlier filter does?

I rewrite the comments in the last commit.

[user1823]

keep the items in trainset if they are not removed from filtered_items

Does this mean that if a card is removed from the pretrain dataset, then all its reviews will be removed from the trainset?

[L-M-Sherlock]

Does this mean that if a card is removed from the pretrain dataset, then all its reviews will be removed from the trainset?

Yes. It's consistent with the Python optimizer.

[user1823]

Does this mean that if a card is removed from the pretrain dataset, then all its reviews will be removed from the trainset?

Yes. It's consistent with the Python optimizer.

As I mentioned in #88 (comment), this outlier filter is too aggressive (it removes too many reviews). This is fine for the pretrain function (because the pretrain method is fragile). However, the train function should have access to more reviews.

Let me explain with an example:

Let's say that for first_rating = 3 and delta_t = 9 days, there are 8 cards and for delta_t = 10 days, there are 100 cards.
Since the cards with delta_t = 9 is too low, the average R will be unreliable. So, it is desirable to filter this case in pre-train.
However, the further review history of these cards is not expected to be very different from those with initial delta_t = 10 days. So, it is not desirable to filter the further review history of these cards.

I agree that some sort of outlier filter is required for the trainset also. But, I think that the same outlier filter should not be used for both.

[L-M-Sherlock]

However, the further review history of these cards is not expected to be very different from those with initial delta_t = 10 days.

I don't think so. If a card is reviewed too early or too late, the first response will be unreliable. If we use the same model the calculate the initial memory state with the unreliable response, the subsequent training will be polluted.

[user1823]

If a card is reviewed too early or too late, the first response will be unreliable.

I agree, but 9 days can't be called "too early" when compared to 10 days.

This is the reason I am suggesting to use a less aggressive outlier filter for the trainset.

[user1823]

Taking the example of my collection (see first review data at the bottom of the comment),

• For first_rating = 1, I think that it is reasonable to keep delta_t = 1 and delta_t = 2 (in my collection) in the trainset.
• For first_rating = 2, the data in my collection is insufficient. So, I will not discuss about it.
• For first_rating = 3, the initial stability is calculated as about 14. So, it doesn't make sense to ONLY keep the data with delta_t < 5 in the trainset.
• For first_rating = 4, there is no major problem in how the outlier filter is working with my data.

If a card is reviewed too early or too late, the first response will be unreliable.

So, what about using an outlier filter based on the ratio between delta_t and stability for the trainset?

First review data: stability_for_pretrain.zip

[L-M-Sherlock]

The current outlier filter only removes 5% cards. Is it aggressive?

[user1823]

Yes (for trainset) / No (for pretrain)

See my above comment; it includes specific examples.

[L-M-Sherlock]

It's really hard to design a perfect solution for that. And I think 11 0.97 33 could be also included in pretrainset.

[user1823]

It's really hard to design a perfect solution for that.

Yes, but we can try to design a solution better than the current one.

As I said in #119 (comment), what if create another condition based on the ratio (or something similar) of the delta_t and stability and then remove the reviews of cards that fulfill BOTH the conditions.
(By both, I mean the ratio condition and the pretrain outlier condition.)

This suggestion is for trainset. I recommend keeping the pretrain filter unchanged.

And I think 11 0.97 33 could be also included in pretrainset.

Based on these values, this group had only 1 lapse. Any group with only 1 lapse is too small to be used for calculating the stability.

[L-M-Sherlock]

As I said in #119 (comment), what if create another condition based on the ratio (or something similar) of the delta_t and stability and then remove the reviews of cards that fulfill BOTH the conditions.
(By both, I mean the ratio condition and the pretrain outlier condition.)

I recommend opening a new issue to discuss about it.