Skip to content

[SPARK-33014][SQL] Support multiple bucket columns in DataSourceV2 table#29890

Closed
LantaoJin wants to merge 4 commits intoapache:masterfrom
LantaoJin:SPARK-33014
Closed

[SPARK-33014][SQL] Support multiple bucket columns in DataSourceV2 table#29890
LantaoJin wants to merge 4 commits intoapache:masterfrom
LantaoJin:SPARK-33014

Conversation

@LantaoJin
Copy link
Contributor

@LantaoJin LantaoJin commented Sep 28, 2020

What changes were proposed in this pull request?

In V2SessionCatalog.convertTransforms(), partitions only match one bucket column by
case BucketTransform(numBuckets, FieldReference(Seq(col))) =>
It should be a bug since there is no comment to explain the reason.
This PR accepts more than one bucket columns for V2 table.

Why are the changes needed?

Create a data source V2 table only accept one bucket column, two or more bucket columns in V2 table will show exception SessionCatalog does not support partition transform ....

Does this PR introduce any user-facing change?

No

How was this patch tested?

Add UT.

@SparkQA
Copy link

SparkQA commented Sep 28, 2020

Test build #129176 has finished for PR 29890 at commit b5d4e59.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 28, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33793/

@SparkQA
Copy link

SparkQA commented Sep 28, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33793/

@SparkQA
Copy link

SparkQA commented Sep 28, 2020

Test build #129178 has finished for PR 29890 at commit 4cd6f22.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 29, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33833/

@SparkQA
Copy link

SparkQA commented Sep 29, 2020

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33833/

@SparkQA
Copy link

SparkQA commented Sep 29, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33842/

@SparkQA
Copy link

SparkQA commented Sep 29, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33842/

@SparkQA
Copy link

SparkQA commented Sep 29, 2020

Test build #129218 has finished for PR 29890 at commit eeb19e7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 29, 2020

Test build #129226 has finished for PR 29890 at commit 5d85106.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@LantaoJin
Copy link
Contributor Author

ping @cloud-fan

@LantaoJin
Copy link
Contributor Author

gentle ping @cloud-fan @HyukjinKwon @dongjoon-hyun

@LantaoJin
Copy link
Contributor Author

Gentle ping @cloud-fan @HyukjinKwon @dongjoon-hyun again

@@ -115,11 +115,10 @@ private[sql] final case class BucketTransform(
}

private[sql] object BucketTransform {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you should update DataFrameWriter.scala, functions.scala, functions.py, etc related codes. cc @rdblue, @brkyvz and @zero323 FYI

@github-actions
Copy link

github-actions bot commented Mar 4, 2021

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

@github-actions github-actions bot added the Stale label Mar 4, 2021
@github-actions github-actions bot closed this Mar 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants