Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataflow construction seems slow-ish #308

Open
frankmcsherry opened this issue Nov 18, 2019 · 1 comment

Comments

@frankmcsherry
Copy link
Member

@frankmcsherry frankmcsherry commented Nov 18, 2019

I'm not certain this is a real issue, but the event_driven example can be made to create 1000 dataflows each a sequence of 1000 nodes, and doing so takes about 10s on my modern desktop. That seems like a long time to put together what is meant to be a linear amount of information. It does seem to be linear (scaling up either number by 10 changes the time by 10), but it would still be good to get a sense for what is going on. Initial investigation suggests allocations in reachability, which indeed does have a bunch of small per-operator allocations. Perhaps those could be consolidated somehow?

@frankmcsherry

This comment has been minimized.

Copy link
Member Author

@frankmcsherry frankmcsherry commented Nov 19, 2019

#312 improves construction slightly by reducing the (linear) number of allocations in is_acyclic() down to just one HashMap allocation. The penalty is that we revisit the source of edges when a count hits zero, instead of loading them to the hash map in the first place. Seems like a plain improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.