reduce frequency of lastoriginupdate db writes

[e-five256]

potentially limit the amount of times we update magazine entries to only update if it has been an hour since the last update, as before it was updating every single event

[ghost]

Output of https://wiki.postgresql.org/wiki/Lock_Monitoring#.D0.A1ombination_of_blocked_and_blocking_activity on kbin.run before applying this patch:

[BentiGorlich]

What do you think about instead of updating with every announce (which includes like and is probably the biggest problem) to only update with announced create activities?

[e-five256]

Even if this was moved to comments vs likes I think we still don't need every single update always as the check is >24h, it seems like we shouldn't landrush the db just like we don't for apfetchedat

[BentiGorlich]

apfetchedat means something totally different though. That is the date we last fetched an update from the remote server. This date means when did the remote server last send us something. And nothing is attached to the date. I would like that to be as accurate as poosible and likes do not really count, so I am willing to remove them, but not creates...

[e-five256]

When your queues fall behind and you try to catch up, every single event locks the magazine row to update this. It most likely causes deadlocks as debounced has shown (though the root cause might be unrelated, it still showed just how often it's required). You just don't need this value to be accurate to the nanosecond of every event. I'm willing to bring it down to a minute if that makes you feel better, but I don't think there should be no limit whatsoever, causing this performance hit on the DB

Like, imagine you're 5k activities behind on /m/196. You don't need 5k DB writes, you need to catch up, this update should be throttled somehow

[ghost]

Agreed that we should not be hitting the db everytime considering what this "feature" is doing. 1 minute resolution is plenty (and probably still overkill).

[e-five256]

@BentiGorlich want to make sure this seems acceptable to you after the change

[BentiGorlich]

I've been thinking about it and came to the conclusion that a ~3h timespan would be a good value. Atm this galue is only used to display magazine stalenes and only if no one is subscribed to the magazine... However I think we should still limit it to create activities, because even if it is just a redis hit when getting all the likes, it still involves latency. If the redis server is located on a differenf machine (who knows) that latency is super noticable... Let me know what you think.

I'd accept it the way it is now as well as it is obviously an improvement

[ghost]

I've been thinking about it and came to the conclusion that a ~3h timespan would be a good value. Atm this galue is only used to display magazine stalenes and only if no one is subscribed to the magazine... However I think we should still limit it to create activities, because even if it is just a redis hit when getting all the likes, it still involves latency. If the redis server is located on a differenf machine (who knows) that latency is super noticable... Let me know what you think.

I'd accept it the way it is now as well as it is obviously an improvement

i can reapprove if this is better, that is ~3h timespan @e-five256

[e-five256]

I'll work on making the changes suggested

changes in process

[e-five256]

definitely super look over me here as this is outside my wheelhouse, kind of guessing based on grabbing payloads from past logs shared in matrix, as obviously this is difficult to test locally. If you meant announces of creates and this should go inside dewrapping announces, let me know.

[e-five256]

I think I've handled the situations as described now where it will update the value when a create or an announced create after 3 hours

[e-five256]

I don't think I really have the expertise to make this change, as I'm basically taking a shot in the dark each time since I'm not familiar with AP and don't have a cluster to test what real events look like.

Not sure if this could be tackled by someone else or just closed. I think assuming @nobodyatroot isn't running it anymore since that one time since he's been running the tag PR since, maybe this isn't a problem that needs to be handled and the problem at the time was the foreign key constraint violation.

I'm just very, super overly optimized where I come from (even two DB calls in the same request are against my best designs, I'm like, forcing people to use refcursors if they need multiple data sets from a DB), whereas Mbin seems to be more like "haha, postgresql connections go brrrrrrrr"

[BentiGorlich]

I am happy to take this on. I am more on the side of "optimize where it is doable with reasonable effort". I agree that this should be optimized and I think I have the best understanding of the AP side in our team. And as my vacation "season" is over for now I should have a lot more time

[BentiGorlich]

Ok, I made it so it updates the lastOriginUpdate field only if the last update was at least 3 hours ago and the incoming activity is an Announce(Create) with a magazine as the actor

[BentiGorlich]

I had to approve, since I requested changes last time. I am obviously going to wait for an approve from somebody else :)