BUG: SparseSeries.value_counts ignores fill_value

[sinhrks]

• closes SparseSeries.value_counts doesn't include fill_value counts #6749
• tests added / passed
• passes git diff upstream/master | flake8 --diff
• whatsnew entry

Also fixed an issue which categorical value_counts resets name.

[jreback]

actually with only a small rewriting I think you could do:

if com.is_extension_dtype(values) or com.is_datetimelike(values):
    result = values.values_counts(dropna)
    result.name = name
else:
   .....

if we define DateTimeIndex.value_counts that properly handles localization before/after (instead of in here)
will be much cleaner.

Further it makes sense to define .value_counts() in tseries/base.py (and move the inference from core/base.py/value_counts.py there. To handle the dispatching of period/DatetimeIndex.

a bit of refactoring but moves functionaility where it should be.

[sinhrks]

Found current master doesn't work for datetimetz. Let me check.

values = [pd.Timestamp('2011-01-01 09:00', tz='US/Eastern'),
                  pd.Timestamp('2011-01-01 10:00', tz='US/Eastern'),
                  pd.Timestamp('2011-01-01 11:00', tz='US/Eastern'),
                  pd.Timestamp('2011-01-01 09:00', tz='US/Eastern'),
                  pd.Timestamp('2011-01-01 09:00', tz='US/Eastern'),
                  pd.Timestamp('2011-01-01 11:00', tz='US/Eastern')]
s = pd.Series(values, name='xxx')

# NG, index must have tz
s.value_counts()
# 2011-01-01 14:00:00    3
# 2011-01-01 16:00:00    2
# 2011-01-01 15:00:00    1
# Name: xxx, dtype: int64

[sinhrks]

Additionally fixing the other Categorical issue.

pd.CategoricalIndex([1, 2, 3, 1, 2, 3]).value_counts(normalize=True)
# UnboundLocalError: local variable 'counts' referenced before assignment

[jreback]

@sinhrks pls rebase when you can

[codecov-io]

Current coverage is 84.04%

Merging #12835 into master will increase coverage by +0.07%

@@             master     #12835   diff @@
==========================================
  Files           136        136          
  Lines         49908      49931    +23   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          41876      41962    +86   
+ Misses         8032       7969    -63   
  Partials          0          0          

1. File ...ndas/core/groupby.py (not in diff) was modified. more
   • Misses -63
   • Partials 0
   • Hits +63

Powered by Codecov. Last updated by 38a7531

[jreback]

this won't happen here as is_extension_type will do this

[sinhrks]

I removed @slow as groupby.value_coutns has separate logic depending on numpy version.

• https://github.com/pydata/pandas/blob/master/pandas/core/groupby.py#L2892

[jreback]

ok, though I still think we should handle in tseries/base, e.g. define DatetimeIndexOpsMixIN and just defer to it like we are doing for sparse/categorical.

[sinhrks]

sparse/categorical value_counts only supports dropna kwds. It may better to be an internal method.

If we do the same thing using DatetimeIndexOpsMixin, the impl should be:

• DatetimeIndexOpsMixin 's public value_counts calls algorithms.py (as it needs to handle kwds like normalize)
• algorithms.py convert datetime-likes to datetime-like Index to handle Series
• Calls DatetimeIndexOpsMixin 's internal _value_counts

Suppose to be little complex, or do you have anything in your mind?

[jreback]

yeah ok, it can ge tricky. ok, we'll keep in mind for later. ideally to put specific logic. merging.