Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-37330][SQL] Migrate ReplaceTableStatement to v2 command #34764

Closed
wants to merge 5 commits into from

Conversation

dchvn
Copy link
Contributor

@dchvn dchvn commented Dec 1, 2021

What changes were proposed in this pull request?

This PR migrates ReplaceTableStatement to the v2 command

Why are the changes needed?

Migrate to the standard V2 framework

Does this PR introduce any user-facing change?

No

How was this patch tested?

existing tests

@github-actions github-actions bot added the SQL label Dec 1, 2021
@SparkQA
Copy link

SparkQA commented Dec 1, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50270/

@SparkQA
Copy link

SparkQA commented Dec 1, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50270/

@SparkQA
Copy link

SparkQA commented Dec 1, 2021

Test build #145795 has finished for PR 34764 at commit 9947a2e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50311/

@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50311/

@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Test build #145836 has finished for PR 34764 at commit 66a2aaf.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50323/

@dchvn
Copy link
Contributor Author

dchvn commented Dec 2, 2021

CC @cloud-fan @Peng-Lei @huaxingao , Thanks!

@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50323/

@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Test build #145848 has finished for PR 34764 at commit ec966bf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@Peng-Lei
Copy link
Contributor

Peng-Lei commented Dec 2, 2021

LGTM, But I just think @huaxingao maybe is planning to do this issue. because it is very similar with CRETE TABLE #34060.

@@ -3563,9 +3563,12 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
case _ =>
// Note: table schema includes both the table columns list and the partition columns
// with data type.
val tableSpec = TableSpec(bucketSpec, properties, provider, options, location, comment,
serdeInfo, external)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to move this out of the pattern matching block so it can be used by both ReplaceTableAsSelect and ReplaceTable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated, thanks!

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50346/

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50346/

options, location, comment, serdeInfo, orCreate = orCreate)
ReplaceTable(
UnresolvedDBObjectName(table, isNamespace = false),
schema, partitioning, tableSpec.copy(external = external), orCreate = orCreate)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think external is outdated. Today Spark have no key word EXTERNAL for CREATE/REPLACE TABLE. We use LOCATION to diff the external table and managed table to unify the syntax for create data souce table. So maybe #Line need to be removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you all,
I think remove external checking code should be in other PR, right?

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50357/

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50357/

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Test build #145871 has finished for PR 34764 at commit 7b358d9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Test build #145882 has finished for PR 34764 at commit f3cca49.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

sorry there are code conflicts now

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Test build #145904 has finished for PR 34764 at commit 05d5a41.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50379/

}.getOrElse(props)
val propsWithOwner = CatalogV2Util.withDefaultOwnership(newProps)
case ReplaceTable(ResolvedDBObjectName(catalog, ident), schema, parts, tableSpec, orCreate) =>
val qualifiedLocation = tableSpec.location.map(makeQualifiedDBObjectPath(_))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not related to this PR: seems path is not qualified in CreateTableAsSelect? cc @imback82

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it doesn't seem to be qualified. I will create a PR for that.

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50383/

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50383/

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Test build #145908 has finished for PR 34764 at commit 31cfd01.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@huaxingao
Copy link
Contributor

@dchvn Could you rebase? I know the Git Action failures are not related to this PR, but with so many failures, I don't feel comfortable to merge the PR.

@dchvn
Copy link
Contributor Author

dchvn commented Dec 4, 2021

Ping @huaxingao , GitHub action is good now, thanks

@huaxingao huaxingao closed this in c411d26 Dec 4, 2021
@huaxingao
Copy link
Contributor

Merged to master. Thanks!

@dchvn
Copy link
Contributor Author

dchvn commented Dec 4, 2021

many thanks! @huaxingao

@dchvn dchvn deleted the migrate-replacetable branch December 4, 2021 02:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
6 participants