[ENH] OWFeatureStatistics

[pavlin-policar]

Show basic statistics for every feature.

Features of note:

• Widget is usable with large datasets (means, vars and missing are computed on init, hisograms are computed and drawn as needed and cached)
• Histograms keep their aspect ratio as the widget resizes (however, this ratio is ignored if the height is too small).
• Dropdown to enable coloring by any target variable, or disable coloring entirely.
• Filter textbox for variable names
• Sparse support

Todo:

• Sorting (currently uses slow QSortFilterProxyModel), special sorting by variable type, special sorting by first and second moment
• Implement report
• Attribute selection sends dataset with these attributes to output.

Current state (housing and GDS1615, respectively):

[kernc]

See Orange.data.util.scale.

[pavlin-policar]

Awesome! This is exactly what I needed.

[kernc]

getattr itself raises AttributeError if not hasattr and no default provided.

[pavlin-policar]

Yes, that's true, but then in case the error does actually get raised, the AttributeError message will refer to the self.attribute, which can be very confusing. This way, it is clear where the error occurs.

[kernc]

Could show n_defined and n_unique for string variable?

[kernc]

What if the variable were in Y or metas?
Prefer:

col = self._data.get_column_view(self.attribute)[0]

[nikicc]

This widget crashes with sparse data since there are many calls to numpy's function that cannot handle sparse data sets (e.g. np.is_nan). Could you make it work on sparse data as well? Please, take a look at the Orange/statistics/util.py module which has some functions that are equivalents of the numpy's just that they also handle sparse data sets.

If adopting to sparse data doens't seem feasible, can you at least add a warning, that the widget doensn't yet handle sparse data sets, so it doens't crashes? Look for example in OWBaseLearner class (Orange/widgets/utils/owlearnerwidget.py)

[nikicc]

This widget crashes with sparse data since there are many calls to numpy's function that cannot handle sparse data sets (e.g. np.is_nan). Could you make it work on sparse data as well? Please, take a look at the Orange/statistics/util.py module which has some functions that are equivalents of the numpy's just that they also handle sparse data sets.

If adopting to sparse data doens't seem feasible, can you at least add a warning, that the widget doensn't yet handle sparse data sets, so it doens't crashes? Look for example in OWBaseLearner class (Orange/widgets/utils/owlearnerwidget.py)

[nikicc]

Please use our own implementation of mean (Orange/statistics/util.py) which can also handle sparse data sets.

[nikicc]

Please use our own implementation of countnans (Orange/statistics/util.py) which can also handle sparse data sets.

[nikicc]

Please use our own implementation of bincount (Orange/statistics/util.py) which can also handle sparse data sets.

[kernc]

Early exit saves an indent level and is generally easier to follow.

[kernc]

No commit comments, except when preceded by meta-comments.

[kernc]

This will take up all the space?

self.mainArea.layout().addWidget(self.view)

[pavlin-policar]

This PR depends on #2458, for the implementation of digitize.

[nikicc]

@pavlin-policar the widget crashes if I input some data set with textual features (i.e. StringVariables). We should probably just skip textual features.

[pavlin-policar]

Yes, I'm also quite sure TimeVariable would crash the widget as well, but I'm not entirely sure how to handle them. On the other hand, I'm not sure hiding them would be the best idea either, since one would expect all the features to have some sort of summary.

For a StringVariable it may make sense to display the most common string (if it exists) and the number of occurences. Drawing a distribution would make no sense, since the number of possible values will probably be way to high and the bars would be next to invisible.

As for TimeVariable, I have basically no idea what we could display for that... Perhaps the date range, but that seems like a reach...

I guess I'll just not display them until we can come up with something better.

[nikicc]

For a StringVariable it may make sense to display the most common string (if it exists) and the number of occurences. Drawing a distribution would make no sense, since the number of possible values will probably be way to high and the bars would be next to invisible.

I'm afraid that in many cases this would yield stopwords like a, the etc. Perhaps, should we plot a histogram of number of words inside documents?

[kernc]

As for TimeVariable, I have basically no idea what we could display for that...

Obviously was designed by preschool children, but may otherwise work:

tvar = data.domain['time']
time_col = data.get_column_view(tvar)[0]
tvar.rerp_val(nanmin(time_col))
tvar.rerp_val(nanmax(time_col))
tvar.rerp_val(nanmean(time_col))
...

[ajdapretnar]

As for TimeVariable, I think we should display a line chart of occurrences in time. I know we can't even handle that in a proper plot, but perhaps this is a good opportunity to think about it. Could we create a heuristic that figures out what is the proper granularity of time instances (minute, hour, day, year?) and plot it in a line chart as sum of instances? Normally I'd like a quick glance at when was a particular event the most frequent.

[pavlin-policar]

This PR depends on #2558, for the proper implementation of countnans.

[pavlin-policar]

For the time variable, I agree with @ajdapretnar; a line chart would be far more appropriate than a histogram. As she said in a histogram, we have no way of determining a good granularity for the bins.

However, how would we color the line charts to show the values of the target variable? For a discrete target, we could simply plot multiple lines, each in a different color, but what about for a continuous target? Coloring the area under the curve might be a good solution... If this turns out to work well, it may be worth testing for continuous variables as well.

[ajdapretnar]

With the recent master on core and this PR I get:
AttributeError: 'OWFeatureStatistics' object has no attribute 'proxy_model'

[kernc]

Looks like a remnant since 2e44023. AbstractTableModel unfortunately does not do filtering. So you should probably adapt/remove this.

But I think filtering is nice to have. @pavlin-policar What are your thoughts on it? Would you be willing to amend AbstractSortTableModel with filtering capabilities, or wrap FeatureStatisticsTableModel in a QSortFilterProxyModel just for its filtering?

[pavlin-policar]

My bad, I must have missed it when converting over to the new AbstractTableModel.

I think filtering is a very important feature, since the GEO datasets can have thousands of features and looking for features by scrolling can get tedious, very fast. I will look into it.

[pavlin-policar]

@kernc, @nikicc This should now be stable enough for another review. I've also squashed all the commits so it's not a complete mess anymore.

[kernc]

Histograms are too high? I'd expect them to be the height of a normal table view line, about 2em at most, likened to sparklines.

[pavlin-policar]

That's actually a feature 😄 The histogram resizes to take up the entire area that the text doesn't need (this could be useful if you find an interesting histogram and want to take a closer look, you resize the window and make it wider, and the histogram grows). If you open and resize the widget, you'll see it work, the default sizes should look all right.

Since it does look quite extreme on your screenshot, perhaps it may be a good idea to set some kind of a limit to how big it can grow?

[kernc]

Resizing in width is fine, but height? The histogram doesn't shrink smaller when the window is shrunk vertically.

[pavlin-policar]

That's true, it is prevented from shrinking into nothing. I'll tinker with fixing the height or limiting it. The thing is that you probably want a compact overview of histograms, so you can see many at a time, and want them to be fairly small, but then one may catch your eye and you'd want to inspect it further and make it bigger. Having a fixed height doesn't allow that.

[kernc]

Mouse hover could pop over a larger copy of the histogram for closer inspection? Or better yet, since the binning is as crude, one could instead select the attribute and pass it out into a Distributions widget for more precise inspection.

[kernc]

@pavlin-policar Could you possibly fix this row height by the end of the weekend so it gets featured in the next asap release? I would but don't know how.

[pavlin-policar]

Sure thing! The latest commit limits the max height the histogram can grow to. This way the histograms still maintain a reasonable aspect ratio at a couple window sizes. If it turns out that the growth is pointless, it's easy to throw out.

[kernc]

I made a change allowing to color by any variable. Is it ok?

[pavlin-policar]

I like this change a lot, however, now when I open the widget, there is no option selected for colouring (see screenshot). I think having the colouring default first target variable would make the most sense.

[kernc]

Fixed.

[pavlin-policar]

Looks good to me 👍