Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Idea for speeding up the collapse algorithm when it requires a large number of iterations #212

Open
ablack3 opened this issue Jan 29, 2024 · 0 comments

Comments

@ablack3
Copy link
Collaborator

ablack3 commented Jan 29, 2024

@mvankessel-EMC, @schuemie,

The number of "collapse" iterations you need to do scales with the square of the number of event cohorts, O(n^2) maybe?, so with a large number of event cohorts the number of iterations can be quite large.
Stopping early is one idea to speed up execution but involves information loss (some collapse steps will not be done).

But this collapse step can be partitiond by person. So if a person's data has not changed from iteration 3 to iteration 4 for example we do not need to include that person in interation 5.

So the idea is this: We just remove patients from the cohort table while doing the collapse algorithm if the number of records they have in the table did not change in the previous iteration. This would mean that for later iterations of the algorithm would run much faster than earlier iterations because it would run on fewer records.

Just an idea...

@ablack3 ablack3 changed the title Idea for speeding up the collapse algorithm Idea for speeding up the collapse algorithm when it requires a large number of iterations Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant