Skip to content

[SPARK-37352][PYTHON] Silence the index_col advice in to_spark() for internal usage.#34626

Closed
itholic wants to merge 3 commits intoapache:masterfrom
itholic:SPARK-37352
Closed

[SPARK-37352][PYTHON] Silence the index_col advice in to_spark() for internal usage.#34626
itholic wants to merge 3 commits intoapache:masterfrom
itholic:SPARK-37352

Conversation

@itholic
Copy link
Contributor

@itholic itholic commented Nov 17, 2021

What changes were proposed in this pull request?

This PR proposes adding internal method _to_spark() to silence the advice log when it's used as an internal purpose.

  • Also fix the (DataFrame|Series|Index).to_pandas() to reuse the (DataFrame|Series|Index)._to_pandas().

Why are the changes needed?

to_spark() without specifying index_col parameter is used in many places when implementing API, but users might not want to see the every warnings for internal usage.

Showing the advice log when only users explicitly use the to_spark() without specifying index_col parameter is enough.

Does this PR introduce any user-facing change?

Reduce the advice log for _to_spark() without specifying index_col parameter.

How was this patch tested?

Manually check the result

@itholic itholic changed the title [SPARK-37352] Silence the index_col advice in to_spark() for internal usage. [SPARK-37352][PYTHON] Silence the index_col advice in to_spark() for internal usage. Nov 17, 2021
@SparkQA
Copy link

SparkQA commented Nov 17, 2021

Test build #145319 has finished for PR 34626 at commit 41d5b54.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 17, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49790/

@SparkQA
Copy link

SparkQA commented Nov 17, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49790/

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants