[SPARK-36470][PYTHON] Implement `CategoricalIndex.map` and `DatetimeIndex.map` #33756

xinrong-meng · 2021-08-17T01:25:42Z

What changes were proposed in this pull request?

Implement CategoricalIndex.map and DatetimeIndex.map

MultiIndex.map cannot be implemented in the same way as the map of other indexes. It should be taken care of separately if necessary.

Why are the changes needed?

Mapping values using input correspondence is a common operation that is supported in pandas. We shall support that as well.

Does this PR introduce any user-facing change?

Yes. CategoricalIndex.map and DatetimeIndex.map can be used now.

CategoricalIndex.map

>>> idx = ps.CategoricalIndex(['a', 'b', 'c'])
>>> idx  
CategoricalIndex(['a', 'b', 'c'], categories=['a', 'b', 'c'], ordered=False, dtype='category')

>>> idx.map(lambda x: x.upper())  
CategoricalIndex(['A', 'B', 'C'],  categories=['A', 'B', 'C'], ordered=False, dtype='category')

>>> pser = pd.Series([1, 2, 3], index=pd.CategoricalIndex(['a', 'b', 'c'], ordered=True))
>>> idx.map(pser) 
CategoricalIndex([1, 2, 3], categories=[1, 2, 3], ordered=True, dtype='category')

>>> idx.map({'a': 'first', 'b': 'second', 'c': 'third'}) 
CategoricalIndex(['first', 'second', 'third'], categories=['first', 'second', 'third'], ordered=False, dtype='category')

DatetimeIndex.map

>>> pidx = pd.date_range(start="2020-08-08", end="2020-08-10")
>>> psidx = ps.from_pandas(pidx)

>>> mapper_dict = {
...   datetime.datetime(2020, 8, 8): datetime.datetime(2021, 8, 8),
...   datetime.datetime(2020, 8, 9): datetime.datetime(2021, 8, 9),
... }
>>> psidx.map(mapper_dict)
DatetimeIndex(['2021-08-08', '2021-08-09', 'NaT'], dtype='datetime64[ns]', freq=None)

>>> mapper_pser = pd.Series([1, 2, 3], index=pidx)
>>> psidx.map(mapper_pser)
Int64Index([1, 2, 3], dtype='int64')
>>> psidx
DatetimeIndex(['2020-08-08', '2020-08-09', '2020-08-10'], dtype='datetime64[ns]', freq=None)

>>> psidx.map(lambda x: x.strftime("%B %d, %Y, %r"))
Index(['August 08, 2020, 12:00:00 AM', 'August 09, 2020, 12:00:00 AM',
       'August 10, 2020, 12:00:00 AM'],
      dtype='object')

How was this patch tested?

Unit tests.

SparkQA · 2021-08-17T01:30:00Z

Test build #142521 has finished for PR 33756 at commit 29aeaec.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-17T01:47:40Z

Test build #142523 has finished for PR 33756 at commit 6a1578a.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-17T02:13:29Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47022/

SparkQA · 2021-08-17T02:27:11Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47023/

SparkQA · 2021-08-17T03:05:17Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47023/

SparkQA · 2021-08-17T03:08:16Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47022/

SparkQA · 2021-08-17T21:21:33Z

Test build #142568 has finished for PR 33756 at commit 08aa1f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-17T21:25:07Z

Test build #142569 has finished for PR 33756 at commit c598f53.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-17T21:37:56Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47070/

SparkQA · 2021-08-17T22:01:43Z

Test build #142570 has finished for PR 33756 at commit f47ccfa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-17T22:02:20Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47071/

SparkQA · 2021-08-17T22:15:00Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47070/

SparkQA · 2021-08-17T22:38:45Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47071/

SparkQA · 2021-08-18T01:10:57Z

Test build #142572 has finished for PR 33756 at commit b831748.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-18T01:32:08Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47073/

SparkQA · 2021-08-18T02:10:00Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47073/

ueshin

Otherwise, LGTM.

python/docs/source/reference/pyspark.pandas/indexing.rst

python/pyspark/pandas/missing/indexes.py

SparkQA · 2021-08-19T20:37:06Z

Test build #142653 has finished for PR 33756 at commit ad0ab47.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

python/pyspark/pandas/indexes/category.py

ueshin · 2021-08-19T20:36:30Z

python/pyspark/pandas/tests/indexes/test_base.py

@@ -516,19 +515,6 @@ def test_missing(self):
                getattr(psdf.set_index("c").index, name)()

        # CategoricalIndex functions


We should remove from this line to line 525 as well.

Good catch! Removed.

SparkQA · 2021-08-19T20:42:38Z

Test build #142654 has finished for PR 33756 at commit 9938b4b.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-19T21:31:52Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47154/

SparkQA · 2021-08-19T21:32:37Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47155/

SparkQA · 2021-08-19T22:28:10Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47154/

ueshin

LGTM, pending tests.

SparkQA · 2021-08-20T18:04:54Z

Test build #142674 has finished for PR 33756 at commit 080b746.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-20T18:21:53Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47176/

HyukjinKwon · 2021-08-23T01:08:14Z

Merged to master.

HyukjinKwon · 2021-08-23T01:11:09Z

Merging to branch-3.2 too since RC1 failed.

…ndex.map` Implement `CategoricalIndex.map` and `DatetimeIndex.map` `MultiIndex.map` cannot be implemented in the same way as the `map` of other indexes. It should be taken care of separately if necessary. Mapping values using input correspondence is a common operation that is supported in pandas. We shall support that as well. Yes. `CategoricalIndex.map` and `DatetimeIndex.map` can be used now. - CategoricalIndex.map ```py >>> idx = ps.CategoricalIndex(['a', 'b', 'c']) >>> idx CategoricalIndex(['a', 'b', 'c'], categories=['a', 'b', 'c'], ordered=False, dtype='category') >>> idx.map(lambda x: x.upper()) CategoricalIndex(['A', 'B', 'C'], categories=['A', 'B', 'C'], ordered=False, dtype='category') >>> pser = pd.Series([1, 2, 3], index=pd.CategoricalIndex(['a', 'b', 'c'], ordered=True)) >>> idx.map(pser) CategoricalIndex([1, 2, 3], categories=[1, 2, 3], ordered=True, dtype='category') >>> idx.map({'a': 'first', 'b': 'second', 'c': 'third'}) CategoricalIndex(['first', 'second', 'third'], categories=['first', 'second', 'third'], ordered=False, dtype='category') ``` - DatetimeIndex.map ```py >>> pidx = pd.date_range(start="2020-08-08", end="2020-08-10") >>> psidx = ps.from_pandas(pidx) >>> mapper_dict = { ... datetime.datetime(2020, 8, 8): datetime.datetime(2021, 8, 8), ... datetime.datetime(2020, 8, 9): datetime.datetime(2021, 8, 9), ... } >>> psidx.map(mapper_dict) DatetimeIndex(['2021-08-08', '2021-08-09', 'NaT'], dtype='datetime64[ns]', freq=None) >>> mapper_pser = pd.Series([1, 2, 3], index=pidx) >>> psidx.map(mapper_pser) Int64Index([1, 2, 3], dtype='int64') >>> psidx DatetimeIndex(['2020-08-08', '2020-08-09', '2020-08-10'], dtype='datetime64[ns]', freq=None) >>> psidx.map(lambda x: x.strftime("%B %d, %Y, %r")) Index(['August 08, 2020, 12:00:00 AM', 'August 09, 2020, 12:00:00 AM', 'August 10, 2020, 12:00:00 AM'], dtype='object') ``` Unit tests. Closes #33756 from xinrong-databricks/other_indexes_map. Authored-by: Xinrong Meng <xinrong.meng@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 0b6af46) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

xinrong-meng added 2 commits August 16, 2021 18:21

CategoricalIndex.map

7a81d3f

DatetimeIndex.map

29aeaec

github-actions bot added CORE PYTHON labels Aug 17, 2021

missing

6a1578a

xinrong-meng added 4 commits August 17, 2021 13:42

typing

08aa1f3

doctest

c598f53

doctest

abff4cc

test

f47ccfa

xinrong-meng changed the title ~~[SPARK-36470][PYTHON] Implement CategoricalIndex.map and DatetimeIndx.map~~ [SPARK-36470][PYTHON] Implement CategoricalIndex.map and DatetimeIndex.map Aug 17, 2021

xinrong-meng marked this pull request as ready for review August 17, 2021 22:17

doctest

b831748

ueshin approved these changes Aug 19, 2021

View reviewed changes

python/docs/source/reference/pyspark.pandas/indexing.rst Outdated Show resolved Hide resolved

python/pyspark/pandas/missing/indexes.py Outdated Show resolved Hide resolved

del

ad0ab47

def __getattr__

9938b4b

ueshin reviewed Aug 19, 2021

View reviewed changes

deprecated_functions in missing_functions

080b746

ueshin approved these changes Aug 20, 2021

View reviewed changes

HyukjinKwon approved these changes Aug 23, 2021

View reviewed changes

HyukjinKwon closed this in 0b6af46 Aug 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-36470][PYTHON] Implement `CategoricalIndex.map` and `DatetimeIndex.map` #33756

[SPARK-36470][PYTHON] Implement `CategoricalIndex.map` and `DatetimeIndex.map` #33756

xinrong-meng commented Aug 17, 2021 •

edited

Loading

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 18, 2021

SparkQA commented Aug 18, 2021

SparkQA commented Aug 18, 2021

ueshin left a comment

SparkQA commented Aug 19, 2021

ueshin Aug 19, 2021

xinrong-meng Aug 20, 2021

SparkQA commented Aug 19, 2021

SparkQA commented Aug 19, 2021

SparkQA commented Aug 19, 2021

SparkQA commented Aug 19, 2021

ueshin left a comment

SparkQA commented Aug 20, 2021

SparkQA commented Aug 20, 2021

HyukjinKwon commented Aug 23, 2021

HyukjinKwon commented Aug 23, 2021

		@@ -516,19 +515,6 @@ def test_missing(self):
		getattr(psdf.set_index("c").index, name)()

		# CategoricalIndex functions

[SPARK-36470][PYTHON] Implement CategoricalIndex.map and DatetimeIndex.map #33756

[SPARK-36470][PYTHON] Implement CategoricalIndex.map and DatetimeIndex.map #33756

Conversation

xinrong-meng commented Aug 17, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 17, 2021

SparkQA commented Aug 18, 2021

SparkQA commented Aug 18, 2021

SparkQA commented Aug 18, 2021

ueshin left a comment

Choose a reason for hiding this comment

SparkQA commented Aug 19, 2021

ueshin Aug 19, 2021

Choose a reason for hiding this comment

xinrong-meng Aug 20, 2021

Choose a reason for hiding this comment

SparkQA commented Aug 19, 2021

SparkQA commented Aug 19, 2021

SparkQA commented Aug 19, 2021

SparkQA commented Aug 19, 2021

ueshin left a comment

Choose a reason for hiding this comment

SparkQA commented Aug 20, 2021

SparkQA commented Aug 20, 2021

HyukjinKwon commented Aug 23, 2021

HyukjinKwon commented Aug 23, 2021

[SPARK-36470][PYTHON] Implement `CategoricalIndex.map` and `DatetimeIndex.map` #33756

[SPARK-36470][PYTHON] Implement `CategoricalIndex.map` and `DatetimeIndex.map` #33756

xinrong-meng commented Aug 17, 2021 •

edited

Loading