Book retention rules are still a bit incorrect

[benoit74]

See https://cms.openzim.org/title/tomlehrersongs.com_en_all

The oldest book (2025-06-01) should be marked for deletion and then deleted. Not sure why it is not happening.

[elfkuzco]

I thought the retention rules meant to keep all books less than 30 days and then for those older than 30 days, we want to keep last two versions from the last two distinct months. Did I misinterpret it?

[elfkuzco]

https://wiki.openzim.org/wiki/ZIM_Updates

[benoit74]

Nope, this is not what is written: we keep all books less than 30 days and the two latest books from two latest distinct months. The two rules are not related (and will typically select same book for the latest month).

Did we say something different somewhere else? Sorry I did not realized the problem in reviews, I still spent a fair amount of time reading the test cases ... looks like they are not that readable after all ...

[benoit74]

Import to note that the "last 30 days" rules might select no book at all (typical when books are updated once per quarter and last update was two months ago). This is intentional. The other rule still apply and ensures we have at least 2 books (unless title has less obviously).

[elfkuzco]

What happens if there are more than two books less than 30 days?

[benoit74]

Note that if this is a "general" bug, then I don't get why 2025-08-31 book has been deleted on https://cms.openzim.org/title/mdwiki_en_all-app (it is correct behavior)

[benoit74]

What happens if there are more than two books less than 30 days?

We keep all books which are less than 30 days old. Would it be 100 books, we keep all of them. And delete them one by one as they cut the 30 days old threshold (which makes me realize it was a good decision to have a daily task to mark books for deletion, didn't understood it so far ...).

[benoit74]

We need to keep all books for a significant amount of days because there is a significant chance "someone" is downloading it somewhere. Deleting it immediately would cause this download to fail, which is not convenient for end-users, especially when they are downloading a huge ZIM on a slow internet connection. We decide to choose 30 days value a bit arbitrarily, but this is the time we expect users to have completed their download. And we assume new users will in general download the most recent version.

[elfkuzco]

Note that if this is a "general" bug, then I don't get why 2025-08-31 book has been deleted on https://cms.openzim.org/title/mdwiki_en_all-app (it is correct behavior)

This is due to the fact that it is the third oldest book past the 30 days threshold in the maxi flavour of the book. The retention rules are applied per flavour. The first one wasn't selected because it's less than 30 days.

[benoit74]

This is due to the fact that it is the third oldest book past the 30 days threshold in the maxi flavour of the book.

Indeed, makes sense

[elfkuzco]

Will apply the revision as follows:

• fetch books less than 30 days. If there are more than 2 books less than 30 days, then, every old book past 30 days is going to be deleted automatically
• only apply the distinct rule book when there are less than 2 books less than 30 days. And we apply it to make the count to 2. So, if there is one new book, the rule will select only one more book to keep the number at 2.

Am I correct?