Adding a scaled sort, to boost smaller communities.

[dessalines]

• Previously referred to as best .
• Fixes Rework "Hot" sorting to show posts from more varied communities #3622
• Fixes The rank of a post in the aggregated feed should be inversely proportional to the size of the community #1026

Still needs lots of testing with prod data, as well as verifying the scheduled_tasks changes are working correctly.

An example of some ranks:

select 
  pa.post_id, 
  now() - pa.published as time_diff, 
  ca.users_active_month, pa.hot_rank, 
  pa.scaled_rank 
from post_aggregates pa 
  inner join community_aggregates ca 
  on pa.community_id = ca.community_id 
order by pa.scaled_rank desc, pa.published desc 
limit 20;

[dessalines]

I can probably move this query down into the update statement, to avoid the round-trip cost.

edit: tried and failed at this.

[phiresky]

I think this value should be a float? Hot_rank for example really quickly goes to zero (after just ~ 2 days mostly) and all information of values between 0 and 1 and 1 and 2 is lost.

For example, right now select scaled_rank(80, '2023-08-20', 10000); and select scaled_rank(1000, '2023-08-20', 10000); both have the exact same rank even though one has > 10 times the votes!

Really hot rank should also be a float imo but that's maybe out of scope here.

[dessalines]

That's true, these all should probably be made into floats (not sure why I didn't want to use floats originally), could you open an issue for that? Lets do that as a separate issue.

Also being a "news" type sort, the time decay is supposed to go to zero after ~ 2 days, after which published descending takes over.

• https://join-lemmy.org/docs/contributors/07-ranking-algo.html
• https://medium.com/hacking-and-gonzo/how-hacker-news-ranking-algorithm-works-1d9b0cf2c08d

[phiresky]

Mh, that's interesting. Then maybe we can just keep hot_rank as is but make the new scaled_rank a float? I think it would make sense to make the new scaled_rank function and column float from the start instead of migration later.

There's no code change required for making this a float but keeping hot_rank an int

[dessalines]

Maybe I should just change hot_rank to a float then as a part of this, since scaled_rank depends on it.

[phiresky]

Sounds good. Note it will reduce the performance of the scheduled task a fair bit since afterwards almost no post will have a hot rank of 0 and be filtered out (ideas @sunaurus? )

[dessalines]

That's def a concern... We could either use the published timestamp as a filter, or minimum hot_rank threshold.

[sunaurus]

I haven't had a chance to look at the implementation yet, so sorry if this is a dumb idea, but can we not just update the rank functions to set the rank 0 after it decays to some threshold, to guarantee that the majority of posts will still have a 0 value for rank and thus get filtered out during updates?

[sunaurus]

I mean something like if the rank is below 1.0 (or perhaps even higher!) then just set it to 0

[phiresky]

Yes, that should work. But it would need careful consideration what the threshold should be both for hot_rank and scaled_rank. According to dess above, the published sort is supposed to take over after a few days. So maybe it would be better to make the function return zero if published is more than 7 days in the past? The hot_rank function gets published as a parameter anyways.

So that change would just be to replace IF (hours_diff > 0) THEN with IF (hours_diff > 0 AND hours_diff < 24 * 7) THEN

[dessalines]

This 0.1 scale factor is the "I made it up on the spot" part of this PR, and I don't know if it makes any sense.

I tried factors of 1, 0.1, 0.01, using a graphing calculator and some regular hot ranks, and compared them with various community sizes. Using smaller numbers lessens the initial sharp decline sensitivity.

[phiresky]

I can do the int-> float convert for hot_rank and scaled_rank if you don't have time @dessalines

[dessalines]

@phiresky No that's okay, I'll work on that today.

[dessalines]

Okay that's updated now. I also removed the 10k scaling factor from the hot_rank function, as its pointless now that they're floats.

[Die4Ever]

Will this mean that bot communities like Lemmit will dominate my feed? I like Lemmit but I wouldn't want it to be all of the top posts.

[dessalines]

This has nothing to do with bots, but communities with few active users.

You can already block bot accounts universally in your user profile settings, as well as block their users or communities if you wish.

[Die4Ever]

This has nothing to do with bots, but communities with few subscribers.

You can already block bot accounts universally in your user profile settings, as well as block their users or communities if you wish.

Yes but I enjoy the Lemmit bot, I just don't want the top 100 posts on my feed to be entirely Lemmit.

I'm glad to know it's based on subscribers and not active users, but I still feel like posts from bot-only communities might be strongly favored by this sorting. Maybe I'll just be toggling the show bots option on and off each day.

[dessalines]

Sry I misspoke above, this is absolutely based on active monthly users, not subscribers, as subscribers is really a pointless metric, considering how communities can have a ton of subscribers but little to no activity.

Bot communities have nothing to do with this, this will boost any community with few active users.

This won't do a spread either (IE picking a single post from a lot of communities, as that's too slow to do performance-wise).

[Die4Ever]

Sry I misspoke above, this is absolutely based on active users, not subscribers, as subscribers is really a pointless metric, considering how communities can have a ton of subscribers but little to no activity.

Bot communities have nothing to do with this, this will boost any community with few active users.

This won't do a spread either (IE picking a single post from a lot of communities, as that's too slow to do performance-wise).

Right it'll be something to get used to. Lemmit communities are mostly just a single active user (the bot itself) since they don't allow posts from other users.

[phiresky]

My suggestion to both improve the performance and to go in the direction of "published sort should take over after a few days" would be to replace IF (hours_diff > 0) THEN with IF (hours_diff > 0 AND hours_diff < 24 * 7) THEN in the hot_rank function. That way the function will predictably go to 0 after 7 days and published sort will take over. It will also mean the scheduled tasks only have to scan 7 days of content.

[phiresky]

for performance, either the index on idx_post_aggregates_nonzero_hotrank should be replaced and conditioned on this condition, or the OR pa.scaled_rank != 0 be removed

removing the scaled_rank != 0 check should not impact the output I think because scaled_rank with it being floats scaled_rank will only be 0 if hot_rank is also 0

[dessalines]

That sounds right, and you're correct that it does seem pointless to change that index.

"idx_post_aggregates_nonzero_hotrank" btree (published DESC) WHERE hot_rank::double precision <> 0::double precision OR hot_rank_active::double precision <> 0::double precision

[dessalines]

IF (hours_diff > 0 AND hours_diff < 24 * 7)

Good call, I'll do that now.

[Nutomic]

Instead of adding another sort option I would rather use this scaling logic for the existing Active and Hot sorts. That way users will actually notice an improvement immediately, instead having to tell everyone to change sorts manually. If this works well it will be a clear improvement and there should be no need for the current Active/Hot sorts. If it doesnt work, we can finetune during rc process.

[dessalines]

I don't feel too comfortable with replacing them, because they are completely different sorts. One will show posts from popular communities, the other will show sorts from unpopular communities.

I'm open to making it the new default tho, if it ends up looking good with production data.

EDIT: I'm converting this to a draft, so I can test with some local production data, to make sure things look okay.

[dessalines]

Okay its ready for review. The only part I'm unsure about, is a scale factor added to .../(log(2 + X*users_active_month)) . I currently have X=1, and added a comment about this.

I tested this with production data, and the scaled_rank values seem fine. But its something that we'll really have to see in practice. We can always alter that function later on to suit gentler log effects.

[phiresky]

Just as a note: with this change it might make sense to do the post ordering / batch selection with ORDER BY (community_id, published) so that the join on community_aggregates is less expensive. But it's probably not a worth the extra pagination complexity if we assume that the whole communities table+indexes will be in memory in any case.

[phiresky]

LGTM. This will require a new index as well if #3872 is merged.