Events: Periodically switch out of the event parsing loop #686
Conversation
squeaky-pl
changed the title
inbox/events/google.py
Outdated
| @@ -92,7 +93,7 @@ def sync_events( | |||
| updates = [] | |||
| items = self._get_raw_events(calendar_uid, sync_from_time) | |||
There was a problem hiding this comment.
So, if sync_from_time is way back in the past, then _get_raw_events may return millions of events here? This isn't just a yield-to-the-loop-every-now-and-then kind of a problem – it's also a memory problem, right?
Also, nit: items is too vague a name for something that we have a clear, precise name for. Maybe rename to raw_events?
There was a problem hiding this comment.
Yes, this is true. The original Google implementaiton "hoards" the events into a single list, while it could actually perfectly yield them one by one. Microsoft one does not have this problem as it uses generators all the way down the stack. I will port Google implementation to generators as well on a separate PR.
There was a problem hiding this comment.
I will change the name as well here.
Normally these loops parse a very limited number of events, but at the initial sync the implementation will fetch all the events into a list and then start parsing them all. This might block the loop for a little too long and result in the calendar syncing greenlet being interrupted and killed.