Improve the profile data to JSON conversion

[jiffyclub]

The function stats_to_tree_dict in upload.py currently fails for some profiles for unknown reasons. The current known profile which causes it to fail is large, deep, and contains complex recursion.

[jiffyclub]

Referred to the wrong ticket in a commit and accidentally closed this.

[jiffyclub]

dba503e improves the stats_to_tree_dict function but large profiles still break json.dumps and would probably not work well with the d3 visualization anyway. To truly scale WSV we'll want to explore options like proactive pruning of the result JSON and allowing the user to select a sub-node of the program tree to be the root of the visualization (or pick a sensible sub-node automatically).

[jiffyclub]

I had to undo the changes from dba503e because under that code a node's size was calculated only once the first time the node was visited, when really the node size needs to be calculated separately each time the node is visited from a different parent, meaning basically that it is not possible to re-use dictionaries to represent nodes at different places in the call tree.

We could cut down on memory usage by tracking fewer properties of every node, by somehow reusing all properties but the size, or using something more memory efficient than dictionaries, but this may only get us a little way (and cause problems with the JSON module). We were finding that the JSON module couldn't handle the larger call trees anyway, so for the time being it may be most practical to just focus on providing elegant failures when it's not possible to process a profile. That is described in issues #6 and #11.

[jiffyclub]

The problem we are having with this particularly large profile is that the memory necessary to represent it as a dictionary tree and/or JSON string is extremely large. There also seem to be practical limits on the size/complexity of a sunburst that can be elegantly displayed and still have a responsive visualization.

I am going to close this issue and open a new one more properly focused on large profile handling. See new issue #12 for continued conversation.

[embray]

I'm not so sure about this. Why not use the dictionaries like you were before, but use a separate data structure just for tracking the node sizes?

[embray]

Or for each visited node you could do a structure like:

{'size': ..., 'invariants': {...}}

Where invariants is the part of the node dict that's the same each time it's visited, and can be copied.