Skip to content

Fix transform to support multi-index columns.#800

Merged
ueshin merged 1 commit into
databricks:masterfrom
ueshin:transform
Sep 20, 2019
Merged

Fix transform to support multi-index columns.#800
ueshin merged 1 commit into
databricks:masterfrom
ueshin:transform

Conversation

@ueshin
Copy link
Copy Markdown
Contributor

@ueshin ueshin commented Sep 19, 2019

No description provided.

@softagram-bot
Copy link
Copy Markdown

Softagram Impact Report for pull/800 (head commit: a90966b)

⭐ Change Overview

Showing the changed files, dependency changes and the impact - click for full size
(Open in Softagram Desktop for full details)

📄 Full report

Impact Report explained. Give feedback on this report to support@softagram.com

@codecov-io
Copy link
Copy Markdown

Codecov Report

Merging #800 into master will decrease coverage by 2.26%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #800      +/-   ##
==========================================
- Coverage    94.3%   92.03%   -2.27%     
==========================================
  Files          32       32              
  Lines        5828     5827       -1     
==========================================
- Hits         5496     5363     -133     
- Misses        332      464     +132
Impacted Files Coverage Δ
databricks/koalas/frame.py 95.1% <100%> (-1.68%) ⬇️
databricks/koalas/usage_logging/__init__.py 23.14% <0%> (-74.08%) ⬇️
databricks/koalas/usage_logging/usage_logger.py 50% <0%> (-50%) ⬇️
databricks/koalas/__init__.py 77.5% <0%> (-7.5%) ⬇️
databricks/conftest.py 90.69% <0%> (-6.98%) ⬇️
databricks/koalas/namespace.py 84.81% <0%> (-1.33%) ⬇️
databricks/koalas/plot.py 93.34% <0%> (-0.96%) ⬇️
databricks/koalas/utils.py 97.24% <0%> (-0.69%) ⬇️
databricks/koalas/testing/utils.py 77.35% <0%> (-0.63%) ⬇️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb147d1...a90966b. Read the comment docs.

from pandas.core.dtypes.inference import is_sequence
from pyspark import sql as spark
from pyspark.sql import functions as F, Column
from pyspark.sql.functions import pandas_udf
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like we don't need this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it is used at line 446.

@pandas_udf(returnType=as_spark_type(pser.dtype.type))
def calculate_columns_axis(*cols):
return getattr(pd.concat(cols, axis=1), name)(axis=axis, numeric_only=numeric_only)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use F.pandas_udf there, but let me merge this for now.
Please feel free to submit the PR if needed.

@ueshin
Copy link
Copy Markdown
Contributor Author

ueshin commented Sep 20, 2019

Thanks! merging.

@ueshin ueshin merged commit 8ad9b4f into databricks:master Sep 20, 2019
@ueshin ueshin deleted the transform branch September 20, 2019 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants