Adding some recurring lemmy tasks. #1386

dessalines · 2021-01-28T03:13:10Z

Add active users by day, week, month, and half year to site and
community. Fixes Create an active users in the last week query. #1195
Periodically re-index the aggregates tables that use hot_rank.
Fixes Indexes based on hot rank degrade over time due to postgres function mutability. #1384
Clear out old activities (> 6 months). Fixes Periodically clear out activity table. #1133

- Add active users by day, week, month, and half year to site and community. Fixes #1195 - Periodically re-index the aggregates tables that use hot_rank. Fixes #1384 - Clear out old activities (> 6 months). Fixes #1133

dessalines · 2021-01-28T03:13:52Z

crates/db_queries/src/source/activity.rs

+
+  fn delete_olds(conn: &PgConnection) -> Result<usize, Error> {
+    use lemmy_db_schema::schema::activity::dsl::*;
+    diesel::delete(activity.filter(published.lt(now - 6.months()))).execute(conn)


I went with deleting activities older than 6 months.

dessalines · 2021-01-28T03:14:41Z

migrations/2021-01-27-202728_active_users_monthly/up.sql

+    select c.creator_id from comment c
+    inner join user_ u on c.creator_id = u.id
+    where c.published > ('now'::timestamp - i::interval) 
+    and u.local = true


For the site counts, it makes sense to use user.local = true, because these are the values on the-federation.info ... and they wanna know counts on your specific instance.

dessalines · 2021-01-28T03:15:52Z

migrations/2021-01-27-202728_active_users_monthly/up.sql

+  select count(*), community_id
+  from (
+    select c.creator_id, p.community_id from comment c
+    inner join post p on c.post_id = p.id


For community counts tho, I don't use a community.local or user.local filter, because federated users might be doing lots of things on your community. Or you might be viewing a non-local community on your instance, and you still want to know the active users.

For remote communities you are not gonna see the correct user count, only the number of users from your instance which are subscribed to that community. Only if you are viewing the community on the instance where it is hosted will you see the correct follower count this way. It would be better to use the follower count from /c/{community}/followers (although that can be manipulated).

For remote communities you are not gonna see the correct user count, only the number of users from your instance which are subscribed to that community.

As soon as someone subscribes to that remote community, your instance should fetch users as they comment and post over there. So the counts shouldn't be too far off. I do wanna keep these as local DB queries rather than remote fetches, even if the numbers are a little off.

It will fetch the users, but I dont think they get inserted into the Followers table. That happens only on the instance where the community is, and where the user is. You can check a production DB to make sure.

The queries actually don't care about the followers table, they only care whether any users, subscribed or not, have made a post or comment.

Ah I see, then it should be mostly fine. Only problem is when no local user is subscribed to the community, or when local users only have been subscribed for a short time (eg, first local user subscribed a week ago, then the one month and six month statistics would be wrong). You might hide some info in those cases, or simply put a disclaimer that data might be inaccurate, and link to the original instance.

dessalines · 2021-01-28T03:16:46Z

src/main.rs

+  let pool2 = pool.clone();
+  thread::spawn(move || {
+    scheduled_tasks::setup(pool2);
+  });
+


I messed with things for a while, trying to do it via actix threads, but this seemed to be the only one that creates a non-blocking / background thread.

dessalines · 2021-01-28T03:18:15Z

src/scheduled_tasks.rs

+
+  // Reindex the aggregates tables every one hour
+  // This is necessary because hot_rank is actually a mutable function:
+  // https://dba.stackexchange.com/questions/284052/how-to-create-an-index-based-on-a-time-based-function-in-postgres?noredirect=1#comment555727_284052


Read this if you want to see some DBA's hate on me :) . I can't find another way around this issue tho. The performance gain from adding an index (even one that degrades over time), is gigantic: (2 seconds vs 20 ms).

dessalines · 2021-01-28T03:25:54Z

src/scheduled_tasks.rs

+  active_counts(&conn);
+  reindex_aggregates_tables(&conn);
+  scheduler.every(1.hour()).run(move || {
+    active_counts(&conn);
+    reindex_aggregates_tables(&conn);
+  });


Running actives and re-indexing every hour, and right at the beginning.

dessalines · 2021-01-28T03:26:22Z

src/scheduled_tasks.rs

+  let conn = pool.get().unwrap();
+  clear_old_activities(&conn);
+  scheduler.every(1.weeks()).run(move || {
+    clear_old_activities(&conn);
+  });


Running clearing out old activities every week (it deletes them older than 6 months)

Nutomic · 2021-01-29T13:59:52Z

I think you should also write some documentation with details about how these stats are generated.

dessalines · 2021-01-29T14:56:57Z

K done here: LemmyNet/lemmy-docs#14

Adding some recurring lemmy tasks.

a6d7809

- Add active users by day, week, month, and half year to site and community. Fixes #1195 - Periodically re-index the aggregates tables that use hot_rank. Fixes #1384 - Clear out old activities (> 6 months). Fixes #1133

dessalines commented Jan 28, 2021

View reviewed changes

Some cleanup, recalculating actives every hour.

98163e9

dessalines commented Jan 28, 2021

View reviewed changes

dessalines marked this pull request as ready for review January 28, 2021 14:10

dessalines requested a review from Nutomic January 28, 2021 14:10

Merge branch 'main' into recurring_tasks

a8d5ade

dessalines merged commit 0fd0279 into main Jan 29, 2021

dessalines deleted the recurring_tasks branch April 7, 2021 20:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding some recurring lemmy tasks. #1386

Adding some recurring lemmy tasks. #1386

dessalines commented Jan 28, 2021

dessalines Jan 28, 2021

dessalines Jan 28, 2021

dessalines Jan 28, 2021

Nutomic Jan 29, 2021

dessalines Jan 29, 2021

Nutomic Jan 29, 2021

dessalines Jan 29, 2021

Nutomic Jan 29, 2021

dessalines Jan 28, 2021

dessalines Jan 28, 2021

dessalines Jan 28, 2021

dessalines Jan 28, 2021

Nutomic commented Jan 29, 2021

dessalines commented Jan 29, 2021

Adding some recurring lemmy tasks. #1386

Adding some recurring lemmy tasks. #1386

Conversation

dessalines commented Jan 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nutomic commented Jan 29, 2021

dessalines commented Jan 29, 2021