Improve large profile handling

[jiffyclub]

This issue is a replacement for #2 and eventually even this issue may need to be broken out into some more specific issues.

At the moment "large" profiles cause a number of problems for websnakeviz:

• They can overwhelm the stats_to_tree_dict function by creating so many dictionaries that memory usage becomes unsustainable.
• If stats_to_tree_dict does return the dictionary the tree may be so large that it swamps the json.dumps function.
• And then even if that returns the call tree might just have more nodes/complexity than can be elegantly displayed by the current sunburst visualization while staying responsive.

This issue should be a good place to discuss how to tackle the various aspects of large profile handling and any specific plans can spawn new issues.

[embray]

I made a few more comments on #2 about this. It still wouldn't solve all the problems but it's a start.

[jiffyclub]

If you start working on it you could make a new issue for those changes specific to stats_to_tree_dict since they can be made on their own.

[pllim]

Good work on snakeviz! The installation was a breeze.

However when I try to visualize my cProfile stats, I got "Error Loading Profile -- Sorry, we were not able to load this profile! You can try profiling a smaller portion of your code. The statistics table below might help you narrow down which portion of your code to focus on." Didn't have such problem with RunSnakeRun. Seems like this issue is already known here.

My stats file is 292K in size and the program took about 2 minutes to run. If you want, I can email you the stats file, just let me know.

[jiffyclub]

Please do send us a copy, it's always good to have more test data. That does sound like the kind of profile SnakeViz has a hard time with. The number of nodes in the call tree is the real limiting factor.

[davidszotten]

a possible short-term workaround might be an option to stop traversing once a node reaches a certain size. In my current example, the full profile failed to render in the allotted 10 seconds, but setting a cut-off at d['size'] <= 0.5 it rendered ok.

[jiffyclub]

That's a good idea, and if we continue using SVG as the visualization we'll have to keep that as a core part of SnakeViz, I think.