Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft: Build tree manually to discard low importance paths prematurely #69

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ogxd
Copy link
Collaborator

@ogxd ogxd commented May 13, 2023

With sequences

14:55:07 [INFO] Garbage collection done in 3.48 ms
14:55:07 [INFO] Graph built in 17.638 ms
14:55:07 [INFO] Tree built in 5.24 ms
14:55:07 [INFO] Tree sorted in 15.876 ms
14:55:07 [INFO] Report written in 30.628 ms

Building the graph of linear sequences takes quite some time to compute, even though not all data will be needed afterward. Discarding early would speed up the whole process significantly. The problem is that at this stage we don't know yet the number of occurrences of a given retention path (it's in fact what this stage is supposed to do), so we can't discard in advance based on the number of occurrences. What can be done however is, instead of retrieving the full retention paths and then looking at several occurrences, drill through paths one level at a time, and discard at each level.

image

Without sequences (manual tree build + early discard)

wip

@ogxd ogxd added the performance Performance related topics label May 13, 2023
@ogxd ogxd changed the title Build tree manually to discard low importance paths prematurely Draft: Build tree manually to discard low importance paths prematurely May 13, 2023
@ogxd ogxd force-pushed the master branch 2 times, most recently from ab0f5b8 to 4a8a696 Compare May 27, 2023 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Performance related topics
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant