Counteract Recency Bias on Lemmy Sorting Algorithm

[8ullyMaguire]

Requirements

• Is this a feature request? For questions or discussions use https://lemmy.ml/c/lemmy_support
• Did you check to see if this issue already exists?
• Is this only a feature request? Do not put multiple feature requests in one issue.
• Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Is your proposal related to a problem?

Yes, the current issue is that on Lemmy there is a recency bias in the sorting algorithm. As communities grow over time, newer posts tend to receive more votes simply because there are more active users than in the past. This skews the "top of all time" sorting, making it more likely for recent posts to appear at the top, regardless of their relative merit compared to older posts.

Describe the solution you'd like.

I propose implementing a new sorting algorithm that adjusts for the number of monthly active users at the time each post was made. This would allow for a fairer comparison of posts' scores over time, ensuring that a post's visibility in the "top of all time" sorting is more reflective of its actual popularity and impact within the community at the time it was posted.

Describe alternatives you've considered.

An alternative could be to implement a weighted scoring system that diminishes the value of a vote as the total number of active users increases. However, this might be less effective than adjusting scores based on active users at the time of posting, as it does not account for the historical context of each post.

Additional context

Here is an implementation in Rust of a sorting algorithm for Lemmy that takes into account the number of monthly active users at the time a post was made:

use std::collections::HashMap;

struct Post {
    id: u32,
    score: u32,
    date: chrono::DateTime,
}

struct Lemmy {
    posts: Vec,
    mau_history: HashMap, u32>, 
}

impl Lemmy {
    fn score_post(&self, post: &Post) -> u32 {
        let mau = self.mau_history.get(&post.date).unwrap_or(&0);
        post.score / mau
    }
    
    fn sort_posts(&self) -> Vec {
        let mut posts = self.posts.clone();
        posts.sort_by(|a, b| self.score_post(b).cmp(&self.score_post(a)));
        posts
    }
}

This keeps track of the monthly active users (MAU) at different points in time in a hashmap. When sorting posts, it calculates a normalized score by dividing the post's score by the MAU at the time it was posted. This means that older posts with lower MAU will have their scores boosted compared to newer posts with higher MAU.

[Nutomic]

Sorting is implemented in SQL not Rust, you can see an example here: #3907

[dessalines]

Yes, the current issue is that on Lemmy there is a recency bias in the sorting algorithm. As communities grow over time, newer posts tend to receive more votes simply because there are more active users than in the past. This skews the "top of all time" sorting, making it more likely for recent posts to appear at the top, regardless of their relative merit compared to older posts.

Storing historical monthly active users just for this seems way overkill. If you want to work on this I'll leave it open, but otherwise its not something we have time for.