ENH: correlation function accepts method being a callable #22684

shadiakiki1986 · 2018-09-13T08:44:25Z

other than the listed strings for the method argument, accept a callable for generic correlation calculations
closes #xxxx (no issue opened for PR)
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry (not sure if I should do this or the person merging this PR)

pep8speaks · 2018-09-13T08:44:33Z

Hello @shadiakiki1986! Thanks for updating the PR.

There are no PEP8 issues in the file pandas/core/frame.py !
There are no PEP8 issues in the file pandas/core/nanops.py !
There are no PEP8 issues in the file pandas/core/series.py !
There are no PEP8 issues in the file pandas/tests/series/test_analytics.py !

Comment last updated on September 26, 2018 at 02:15 Hours UTC

TomAugspurger

I like the idea. Does this close an issue?

We'll need a release note in 0.24.0.txt

TomAugspurger · 2018-09-13T11:45:46Z

ci/requirements-optional-pip.txt

@@ -14,7 +14,7 @@ lxml
 matplotlib
 nbsphinx
 numexpr
-openpyxl=2.5.5
+openpyxl==2.5.5


This file is autogenerated. #22689

ok should I roll this edit back?

Yes, you should.

TomAugspurger · 2018-09-13T11:46:09Z

doc/source/computation.rst

+.. ipython:: python
+
+   # histogram intersection
+   histogram_intersection = lambda a, b: np.minimum( np.true_divide(a, a.sum()), np.true_divide(b, b.sum())).sum()


Make this pep8 compliant

fixed in new commit

TomAugspurger · 2018-09-13T11:46:37Z

pandas/core/frame.py

@@ -6652,10 +6652,12 @@ def corr(self, method='pearson', min_periods=1):

        Parameters
        ----------
-        method : {'pearson', 'kendall', 'spearman'}
+        method : {'pearson', 'kendall', 'spearman', callable}


I think it'll be `}, or callable on the outside of the options

moved it to outside of the options

TomAugspurger · 2018-09-13T11:47:15Z

pandas/core/frame.py

            * pearson : standard correlation coefficient
            * kendall : Kendall Tau correlation coefficient
            * spearman : Spearman rank correlation
+            * callable: callable with input two numpy 1d-array


"callable expecting two 1d ndarrays and returning a float" (does the callable get ndarrays or series?)

TomAugspurger · 2018-09-13T11:51:38Z

pandas/tests/series/test_analytics.py

@@ -789,6 +789,41 @@ def test_corr_invalid_method(self):
        with tm.assert_raises_regex(ValueError, msg):
            s1.corr(s2, method="____")

+    def test_corr_callable_method(self):


Hmm I care less about testing this exact way of computing the correlation, and more about ensure that the method is dispatched to.

Would it be possible to define a very simple "correlation" function that just returns something like the index of the columns? So the correlation of the nth or and mth column would be like (n + m). Not sure if that's possible.

ok will re-write the test tomorrow

The test now includes a simpler correlation function. It is not possible to identify the nth/mth column as in your example because the correlation function itself does not know about the dataframe as a whole but only as each series on its own. The correlation function I chose is a simple 1 if exact equality else 0

shadiakiki1986 · 2018-09-13T15:47:30Z

This doesn't fix any issue, but I added a note in the 0.24.0 release notes

gfyoung · 2018-09-14T04:48:50Z

doc/source/whatsnew/v0.24.0.txt

@@ -17,6 +17,10 @@ New features

 - ``ExcelWriter`` now accepts ``mode`` as a keyword argument, enabling append to existing workbooks when using the ``openpyxl`` engine (:issue:`3441`)

+
+- :meth:`DataFrame.corr` and :meth:`Series.corr` now accept a callable for generic calculation methods of correlation, e.g. histogram intersection


Use your PR as the issue number. Also, no new line above this sentence.

TomAugspurger

cc @jreback if you have a chance.

This provides a nice, generic alternative to #21925.

jreback

small comments.

jreback · 2018-09-18T14:29:18Z

pandas/core/frame.py

            * pearson : standard correlation coefficient
            * kendall : Kendall Tau correlation coefficient
            * spearman : Spearman rank correlation
+            * callable: callable with input two 1d ndarrays


can you add an example in Examples

jreback · 2018-09-18T14:29:24Z

pandas/core/nanops.py

@@ -764,6 +764,9 @@ def nancorr(a, b, method='pearson', min_periods=None):


 def get_corr_func(method):
+    if callable(method):
+        return method
+
    if method in ['kendall', 'spearman']:


jreback · 2018-09-18T14:29:43Z

pandas/core/series.py

            * pearson : standard correlation coefficient
            * kendall : Kendall Tau correlation coefficient
            * spearman : Spearman rank correlation
+            * callable: callable with input two 1d ndarray


I am not sure how to doc-string this signature here

Should I just leave it as is?

Probably fine as is. The type would be Callable[[ndarray, ndarray], float], but I'm not sure how familiar people are with typing yet.

jreback · 2018-09-18T14:30:12Z

@TomAugspurger yep agreed

codecov · 2018-09-20T11:05:58Z

Codecov Report

Merging #22684 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #22684      +/-   ##
==========================================
+ Coverage   92.18%   92.19%   +<.01%     
==========================================
  Files         169      169              
  Lines       50819    50821       +2     
==========================================
+ Hits        46850    46852       +2     
  Misses       3969     3969

Flag	Coverage Δ
#multiple	`90.6% <100%> (ø)`	⬆️
#single	`42.37% <0%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/nanops.py	`95.14% <100%> (+0.01%)`	⬆️
pandas/core/series.py	`93.74% <100%> (ø)`	⬆️
pandas/core/frame.py	`97.2% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a936399...dbfd95f. Read the comment docs.

TomAugspurger · 2018-09-20T13:56:24Z

pandas/core/frame.py

+
+        >>> df = pd.DataFrame([(1, 2), (0, 3), (2, 0), (1, 1)],
+        ...                   columns=['dogs', 'cats'])
+        >>> df.corr(method = histogram_intersection)


This is failing our doctests. Is there an issue with the output?

Also, pep8: no spaces around the =

TomAugspurger · 2018-09-20T13:57:16Z

pandas/core/series.py

+        >>> s1 = pd.Series([1, 0, 2, 1])
+        >>> s2 = pd.Series([2, 3, 0, 1])
+        >>> s1.corr(s2, method = histogram_intersection)
+        0.416667


Will need to round this, or write it out at full precision.

TomAugspurger · 2018-09-20T13:59:13Z

pandas/core/series.py

            * pearson : standard correlation coefficient
            * kendall : Kendall Tau correlation coefficient
            * spearman : Spearman rank correlation
+            * callable: callable with input two 1d ndarray


Probably fine as is. The type would be Callable[[ndarray, ndarray], float], but I'm not sure how familiar people are with typing yet.

jreback · 2018-09-23T13:15:25Z

pandas/tests/series/test_analytics.py

+        # simple correlation example
+        # returns 1 if exact equality, 0 otherwise
+        my_corr = lambda a, b: 1. if (a == b).all() else 0.
+


can you use result= and expected= here, rather than expected_1 and such. its much easier to follow

- other than the listed strings for the `method` argument, accept a callable for generic correlation calculations

jreback · 2018-09-26T10:20:26Z

thanks @shadiakiki1986 nice change!

…#22684)

TomAugspurger reviewed Sep 13, 2018

View reviewed changes

gfyoung added Enhancement Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff labels Sep 14, 2018

gfyoung reviewed Sep 14, 2018

View reviewed changes

shadiakiki1986 force-pushed the corr-method-callable branch 2 times, most recently from 10dcb68 to 2e22403 Compare September 18, 2018 05:43

TomAugspurger mentioned this pull request Sep 18, 2018

ENH: implement pd.Series.corr(method="distance") #22402

Closed

TomAugspurger approved these changes Sep 18, 2018

View reviewed changes

jreback requested changes Sep 18, 2018

View reviewed changes

jreback added this to the 0.24.0 milestone Sep 18, 2018

jreback mentioned this pull request Sep 18, 2018

ENH: Enable DataFrame.corrwith to compute rank correlations #22375

Merged

dsaxton mentioned this pull request Sep 19, 2018

DOC: Add cookbook entry using callable method for DataFrame.corr #22761

Merged

shadiakiki1986 force-pushed the corr-method-callable branch from 2e22403 to c75bc10 Compare September 20, 2018 11:05

shadiakiki1986 force-pushed the corr-method-callable branch from c75bc10 to 3ca092a Compare September 20, 2018 11:10

TomAugspurger reviewed Sep 20, 2018

View reviewed changes

shadiakiki1986 force-pushed the corr-method-callable branch 2 times, most recently from df06de9 to dc87331 Compare September 22, 2018 07:29

jreback requested changes Sep 23, 2018

View reviewed changes

shadiakiki1986 force-pushed the corr-method-callable branch 2 times, most recently from 60b3498 to 4a69a70 Compare September 26, 2018 02:15

ENH: correlation function accepts method being a callable

dbfd95f

- other than the listed strings for the `method` argument, accept a callable for generic correlation calculations

shadiakiki1986 force-pushed the corr-method-callable branch from 4a69a70 to dbfd95f Compare September 26, 2018 02:23

jreback approved these changes Sep 26, 2018

View reviewed changes

jreback merged commit a393675 into pandas-dev:master Sep 26, 2018

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

ENH: correlation function accepts method being a callable (pandas-dev…

9143599

…#22684)

		@@ -17,6 +17,10 @@ New features

		- ``ExcelWriter`` now accepts ``mode`` as a keyword argument, enabling append to existing workbooks when using the ``openpyxl`` engine (:issue:`3441`)


		- :meth:`DataFrame.corr` and :meth:`Series.corr` now accept a callable for generic calculation methods of correlation, e.g. histogram intersection

ENH: correlation function accepts method being a callable #22684

ENH: correlation function accepts method being a callable #22684

Conversation

shadiakiki1986 commented Sep 13, 2018 • edited

pep8speaks commented Sep 13, 2018 • edited

Comment last updated on September 26, 2018 at 02:15 Hours UTC

TomAugspurger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shadiakiki1986 Sep 17, 2018 • edited

Choose a reason for hiding this comment

shadiakiki1986 commented Sep 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomAugspurger left a comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Sep 18, 2018

codecov bot commented Sep 20, 2018 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Sep 26, 2018

shadiakiki1986 commented Sep 13, 2018 •

edited

pep8speaks commented Sep 13, 2018 •

edited

shadiakiki1986 Sep 17, 2018 •

edited

codecov bot commented Sep 20, 2018 •

edited