[WIP] Compact Variable Explorer with customizable tooltip

[d1manson]

Copied and edited from original post: #2353

I've noticed that most of the time my variable explorer is not visible on screen (it's "behind" some other widget tab). The main reason for this is that its "area-to-usefulness" ratio is not that good...each variable takes up a lot of space vertically and you need to set the width of the widget to be pretty wide in order be able to see the value properly.

It occurs to me that there are 3 usage cases for the widget:

• find a variable that you can't remember the name of or aren't sure if you've currently got
• check the value/type/size/min/max of a variable...or get a plot of it.
• edit a variable

For the first usage, you normally only need to be able to see the names of the variables, so a nice compact list is probably sufficient. However I do note that in some cases you may actually need to do a visual "search" by some heuristic combination of name +value/size/type etc. This heuristic visual search is not really that easy using the current editor, not to mention that it hides many kinds of objects (or else shows you hundreds you don't care about). [A nice, but complicated UI for dealing with this less common task would be a fancy search system a bit like gmail where you say something like "size>100, dtype:int name:_old", ideally with auto-complete helping you to fill out your query, and instant results....but that's not what I am suggesting here...instead....] to address this heuristic visual search task, it would need to be possible to toggle between the full and compact modes.

For the second case you are generally interested in a single variable at a given time (if you wanted to compare multiple values it's currently not really that easy unless the variables happen to be adjacent in the list), so you could have a tooltip showing you the info that you care about. Indeed since this tooltip is generated on request for a single variable you could perform a computation that takes up to about 100-200ms without the user noticing. You also get quite a bit of space to display the data because it's the data for a single variable at a given time. I suggest that the script for generating the tooltip data is exposed to the user so they can fully customize it to render data in a way that is convenient for them (i.e. in the same way users can customize their startup scripts. However, that's not to say that you can't write something fairly simple which guesses what kind of plotting and summary data would be of interest to a "generic" user, .e.g. matshow large 2d arrays, lineplot long 1d arrays, describe data frames etc. And you can show more than one plot/table if you want.

For the third usage case, the editing, you either double click the variable name and launch a separate widget (as you currently do), or if the variable is simple enough you can edit it inline in a similar manner to the current tool, except that the edittext would replace the name cell rather than the value cell (because the value column is not showing).

So...with the above in mind I have forged ahead and had a go at implementing something.

It is still a bit rough around the edges but it does do most of what I describe above. I have submitted this as a PR so that it can more easily be commented on and examined.

(As discussed in #2353, there is a plan to make all toolbars toggle-able)

For reference, in Matlab you can (under certain circumstances) get a tooltip when you mouse over a var in the editor itself:

It also offers the following choice of columns in the variable editor: Value, Size, Bytes, Class, Min, Max, Range, Mean, Median, Mode, Var, Std. But there's no obvious way to arrange your screen in such a way as to actually make space for all those columns,

Update: features implemented so far...

Easily toggle between compact and full mode.

Edit simple values inline (and launch edit dialog for more complex values).

Provide a custom function for generating meta data and html.

In addition to the matshow example above, default function shows things like:

DataFrame.describe.to_html()

simple lineplot of 1d data

sphinx doc for objects that dont fall into other categories and have docstrings

TODOs:

• It would be good to make the tooltip reszing a bit better, possibly providing the option of pinning it with Alt or something and then having scrollbars to navigate longer content like the docstrings.
• Ideally it would be good to implement canceling and timeout for the metadata generation. This would help users with their own custom version of the make_meta_dict function, but also in cases where the default version of the function is causing the program to hang because the object in question is large.
• It would be nice to hook up a custom filter function in addition to the custom make_meta_dict function. Otherwise the user would be forced to turn off the filters in order to find the custom objects for which they have implemented representation code. (Note that this filtering has to be separate from the actual generation of the representation because we assume the representation is a costly process and should only be done in full when required, i.e. not just as a test of feasibility).

[ccordoba12]

This is very interesting!! I'll come back to you after I release beta1 (hopefully by the end of the week ;-)

[d1manson]

@ccordoba12. I'm tempted to try re-factoring all the relevant code as it's currently pretty convoluted, so don't look too closely at the details of the implementation yet...for now I'm more interested in what people think about the two main concepts- compactness and extensive/customizable tooltips...you should be able to play with it as it currently is and see what you like/don't like and what other features you'd want.

[Nodd]

Using the tooltip seems fragile to me, because you have to not move the mouse at all to see the information. Why not display the information in the Object inspector instead ? Of course it means that some place would be reserved for the widget.

Also the make_meta_dist parameter included in the settings seems out of place. Maybe these functions could be added as plugins ? This would enable to have multiple versions to match different types.

The visualizations are quite cool I think, it won't ne adapted to all data sets but when it works it works well ! I remember someone asking for a display for large sparse matrix, this could be a first step.

[goanpeca]

I agree with @Nodd , the tooltip is not meant to be used this way, meaning, holding so much information.

Putting this info in the object inspector would then jump tabs if you have them in the same dockposition...

Why not just a new modal widget that disappears after esc or after some time... or if the a variable is selected and the variable inspector has focus

[goanpeca]

Plus using a dedicated widget would allow for a more fluid xp, the user can see the fast to calculate info, and a loading image can appear while the image (if there is a picture representation) is drawn...

[d1manson]

The way it's setup here it's not a real tooltip, it's a qdialog made to look like a tooltip, the idea being that you have enough control to do the kind of things you are suggesting.

Regarding the get_meta_dict function..although you could create one or more miniplugins for each type, I think in the end the user would probably prefer you to just give them full control over what is generated - there's the logic of guessing exactly what should be displayed (not only for a given type/subclassed type but also for different flavours of data within that type), and then the actual code that generates the displayed data.

Although you can have a pretty good attempt at writing a default script which does a reasonable job of both the guessing and the rendering, I'm sure that people would very quickly feel the need to customize it to do things exactly the way they want them, and I don't see what's wrong with that.

[goanpeca]

Ah ok.. good to know, although i this case I do not think it should actually look like a tooltip

[goanpeca]

In general I like the idea, worth discussing in more depth 👍

[d1manson]

Just to let you guys know... I have indeed attempted a fairly major rewrite of the whole variable explorer/namespace/dict editor stuff. I've still got some way to go, but it's coming on ok I think.

https://github.com/d1manson/spyder/tree/rewrite-var-exp

The colors down the right hand side show the base classes (it hashes each of the string name into a color. so user defined stuff should work just fine, although I'm not entirely convinced it's useful yet!).

The panel at the bottom is a list of filters, which are applied in order: you can type in the box to edit the list, there is autcomplete on filter names. the idea is to make a simple modal dialog to allow the user to add their own filters to their personal filter library ..these are simple things like regex matches on names, or lists of exact names, or lists of types.

This doesn't really need any comments right now, but I just wanted to let interested parties know!

[Nodd]

This is really interesting but it's also really advanced features! It would fit better as an external optional plugin (yes, I'm deep into plugins these days). It should be quite easy to do the transformation since almost everything in spyder follows the same structure.

[d1manson]

yeah i think that might be a good plan

[ccordoba12]

+1 from me too to developing this as an external plugin :-)

[Nodd]

@d1manson The plugins API is currently undergoing changes, so continue to develop on spyder's branch for now, I'll help you do the conversion once it's more stable.

[goanpeca]

@d1manson are you implementing some sort of caching?

Adding a flag to each variable in the variable explorer to track if it has been modified, and cached the data and metadata stats so that the next time (if variable has not changed) the already calculated stuff can be used?

Also, you could implement these calculations in a worker thread as each variable is created so that the display of information appears to be seamless

[d1manson]

I'm not going to have a lot of time to work on this for the next few days...but I will get back to it.

I agree that performance is critical here, and I am a bit of a perf fanatic. But for now I'm trying to keep things simple and implement as many features as possible...perf will come later, even if it means making major changes.

Regarding caching specifically, I think it's a bit tricky to detect changes in an arbitrary object. For example (admittedly a stupid example, but it makes the point)...a user defined (or monkey patched) object could have a __len__ method that returns random values each time you call it...or it could perform a twitter search and tell you how many results there are in the last 5seconds or check the volume of trades on a stock or something....or there could be some kind of fancy "view" class which presents a concatenated view of other lists/arrays, so that even when the instance of the view doesn't change, it may still want to update it's len and other properties.

But in general, the idea is to keep the calculation time down to about 100ms, as you wont really notice that. The important thing is it needs to be cancel-able - if the user moves the cursor somewhere else or it's just taking too long it then becomes important to be able to stop the computation and work on a different variable's meta data or just return a timeout message.

In terms of threading, I will look in to it, but I imagine it's a bit tricky to make it work safely, without implementing some complicated recursive pickle check and copy operation, or else forcing a GIL lock in CPython (or some catastrophic non-locking behaviour in other Pythons?)...I know numpy releases the GIL for many operations, which would be helpful if you are doing things like taking sums and max's etc, (also for pandas I guess), but those kinds of operations tend to be fast enough already unless you are talking about >300MB or a really slow computer...there's a lot of possibilities as to how to cope with slow stuff, but for now I will aim to keep it simple!

The next thing on my list is traversing down into a namespace, keeping the UX fairly tidy.

[goanpeca]

@d1manson do you want to implement this as a plugin :-) ? preliminary plugin support will come with Spyder 3.0 so it is a good time to test and have a plugin for this.

[d1manson]

At some point, yes, I suppose I could implement it as a plugin, but I'm not expecting to work on this anytime soon though, which is why I decided to close this PR. [Thanks for taking over on my other PR!]

The code here was getting pretty unwieldy and a long way from acceptable perf-wise, so there could be quite a lot of work to get this thing to where it needs to be as a plugin or otherwise. Also, I'm not sure what the plugin API looks like at the moment (are there up-to-date docs somewhere?), but to make this work, there would need to be serious integration opportunities with the Spyder core.

If anyone else wants to work on the ideas I've played around with here I'd be very happy for them to do so.

(Right now I'm supposed to be working hard finishing off my PhD, I've also got a fairly major side project on the go in my spare time.)