Fix sprint overviews for large projects #154
Conversation
Doing this would probably reduce memory consumption by a factor of 2 but also break snapshots when switching between status and workboard mode.
In order to have working sprint snapshots while also avoiding memory issues this will filter out all transactions except the ones that are relevant for status and workboard changes when creating snapshots. Additionally a settings aware filter was added to filter out even more transactions for live sprint overviews according to the workboardMode settings to reduce memory usage.
jakobw
changed the title
|
Would you mind splitting this into several small pull requests or at least re-submit #155 against master. That would make reviewing this stuff a lot easier. ;-) |
Lazyload
Doing this would probably reduce memory consumption by a factor of 2 but also break snapshots when switching between status and workboard mode.
In order to have working sprint snapshots while also avoiding memory issues this will filter out all transactions except the ones that are relevant for status and workboard changes when creating snapshots. Additionally a settings aware filter was added to filter out even more transactions for live sprint overviews according to the workboardMode settings to reduce memory usage.
…into large_projects_fix
|
Rebased to get the changes of #160 in here. |
| <?php | ||
| namespace Phragile; | ||
|
|
||
| /*** |
There was a problem hiding this comment.
I think we use only two starts in this project? Or is this inconsistent anyways?
|
This still does two things in a row that could possibly be split but I think if the mentioned comments get resolved/answered we can merge this :-) |
Does that mean that with this patch it is still impossible to view a sprint? Or to view a snapshot? Do we need to disable switching the mode for snapshots? |
|
@Benestar It's possible to view sprints and snapshots with this patch :) |
|
@Benestar It would be awesome if you could have another look at this :) |
Fix sprint overviews for large projects
|
I just noticed that http://phragile-test.wmflabs.org/sprints/782 looks very strange. But at least it loads something! |
Task on Phab: https://phabricator.wikimedia.org/T110330
There were two different problems with large projects.
It is not possible to load all task transactions of large projects at once through Conduit (Phabricator's API). This was solved by fetching transactions in batches of 200. More details: https://phabricator.wikimedia.org/T109036
Loading all transactions is solved by loading them in batches, however, processing all the transactions consumes a lot of memory. This can be improved by filtering out irrelevant transactions. For snapshots it'll store all transactions regarding workboard and status changes so that switching between workboard and status mode still works. Unfortunately this still exceeded the memory limit for sprint overviews but
can bethat has been fixed by also removing status transactions in workboard mode and workboard transactions in status mode.I deployed the changes to a test instance and you can check it out here: http://phragile-test.wmflabs.org/sprints/483
Snapshots are still somewhat broken for large projects unfortunately. For some reason they can be created but not read.Snapshots are fixed in #155