Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
[performance] feed_to_graph_path is slow on larger feeds #12
Both are executing Pandas functions so, beneath them, are just Pandas
For example, since these are all wrapped in a single route iteration, the whole operation is embarrassingly parallelizable.
This was referenced
Mar 29, 2018
referenced this issue
Apr 11, 2018
On smaller feeds (or even mid-sized feeds, like AC Transit), MP is slower. I need to figure out how to intelligently navigate away from using MP in these situations.
Sigh, this whole performance issue is not good.
Above run once with MP as False and one time as True.
LA Metro (without digging around for the exact numbers) used to take 12-15 minutes.
It now takes:
So, no observable improvement. Of course, it's running in a Docker environment that only has access to 2 CPUs on my '16 Macbook Pro. A better test would be to use a virtual machine on AWS / GCloud or wherever and see what gains are achieved there.
That said, we can observe that there are pretty limited (essentially no observable) gains to be had by MP for the typical user/use case (local machine, in a Notebook like environment). This is something that should be addressed long term.