You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The number of "collapse" iterations you need to do scales with the square of the number of event cohorts, O(n^2) maybe?, so with a large number of event cohorts the number of iterations can be quite large.
Stopping early is one idea to speed up execution but involves information loss (some collapse steps will not be done).
But this collapse step can be partitiond by person. So if a person's data has not changed from iteration 3 to iteration 4 for example we do not need to include that person in interation 5.
So the idea is this: We just remove patients from the cohort table while doing the collapse algorithm if the number of records they have in the table did not change in the previous iteration. This would mean that for later iterations of the algorithm would run much faster than earlier iterations because it would run on fewer records.
Just an idea...
The text was updated successfully, but these errors were encountered:
ablack3
changed the title
Idea for speeding up the collapse algorithm
Idea for speeding up the collapse algorithm when it requires a large number of iterations
Jan 29, 2024
@mvankessel-EMC, @schuemie,
The number of "collapse" iterations you need to do scales with the square of the number of event cohorts,
O(n^2)
maybe?, so with a large number of event cohorts the number of iterations can be quite large.Stopping early is one idea to speed up execution but involves information loss (some collapse steps will not be done).
But this collapse step can be partitiond by person. So if a person's data has not changed from iteration 3 to iteration 4 for example we do not need to include that person in interation 5.
So the idea is this: We just remove patients from the cohort table while doing the collapse algorithm if the number of records they have in the table did not change in the previous iteration. This would mean that for later iterations of the algorithm would run much faster than earlier iterations because it would run on fewer records.
Just an idea...
The text was updated successfully, but these errors were encountered: