[SPARK-32838][SQL]Check DataSource insert command path with actual path #30057

AngersZhuuuu · 2020-10-15T15:28:18Z

What changes were proposed in this pull request?

Currently, we verify path in DataSourceAnalysis

// For dynamic partition overwrite, we do not delete partition directories ahead.
// We write to staging directories and move to final partition directories after writing
// job is done. So it is ok to have outputPath try to overwrite inputpath.
 if (overwrite && !insertCommand.dynamicPartitionOverwrite) {
      DDLUtils.verifyNotReadPath(actualQuery, outputPath)
}

/**
   * Throws exception if outputPath tries to overwrite inputpath.
   */
  def verifyNotReadPath(query: LogicalPlan, outputPath: Path) : Unit = {
    val inputPaths = query.collect {
      case LogicalRelation(r: HadoopFsRelation, _, _, _) =>
        r.location.rootPaths
    }.flatten

    if (inputPaths.contains(outputPath)) {
      throw new AnalysisException(
        "Cannot overwrite a path that is also being read from.")
    }
  }

For partition table, we can know that both outputPath and collected inputPaths are table's , not partition's path.
Actually, when we overwrite table A's partition a1 select from partition a2, then path won't conflict. But it compare both use table's path, then throw AnalysisException.

So we need to check this after Optimizer(got partition pushed down), If we don't care about dynamicPartitionFilter, we can just add a check rule about this issue on SparkPlan using FileSourceScanExec.selectedPartitions.

Why are the changes needed?

Fix bug

Does this PR introduce any user-facing change?

No

How was this patch tested?

UT

AngersZhuuuu · 2020-10-15T15:31:16Z

FYI @dongjoon-hyun @cloud-fan

SparkQA · 2020-10-15T15:42:07Z

Test build #129842 has finished for PR 30057 at commit 9eb5073.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-15T16:10:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34447/

SparkQA · 2020-10-15T16:23:13Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34448/

SparkQA · 2020-10-15T16:34:20Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34447/

SparkQA · 2020-10-15T16:39:53Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34448/

SparkQA · 2020-10-16T04:17:44Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34478/

SparkQA · 2020-10-16T04:43:39Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34478/

SparkQA · 2020-10-16T04:43:53Z

Test build #129873 has finished for PR 30057 at commit 620956d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-16T05:49:43Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34480/

SparkQA · 2020-10-16T06:08:06Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34480/

SparkQA · 2020-10-16T06:43:32Z

Test build #129875 has finished for PR 30057 at commit 199aa8f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-17T05:02:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34540/

SparkQA · 2020-10-17T05:22:36Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34540/

SparkQA · 2020-10-17T07:05:01Z

Test build #129935 has finished for PR 30057 at commit 9d9ab88.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-10-17T04:12:53Z

...ain/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala

+          throw new AnalysisException(
+            s"Cannot overwrite a path that is also being read from.")
+        }
+      }


If we don't care about dynamicPartitionFilter, we can just add a check rule about this issue on SparkPlan using selectedpartition.

AngersZhuuuu · 2020-10-17T10:43:28Z

retest this please

SparkQA · 2020-10-17T11:37:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34549/

SparkQA · 2020-10-17T12:00:15Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34549/

SparkQA · 2020-10-17T15:06:37Z

Test build #129943 has finished for PR 30057 at commit 9d9ab88.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-10-19T02:00:17Z

ping @dongjoon-hyun @cloud-fan

dongjoon-hyun · 2020-10-19T02:11:28Z

cc @viirya

viirya · 2020-10-19T02:26:42Z

Why are the changes needed?
Fix bug

Can you describe clearly in the PR description what bug this fixes? The PR description is important for reviewers to understand the issue quickly. Please describe the issue the PR tries to fix in details. Thanks.

SparkQA · 2021-03-29T06:44:57Z

Test build #136640 has finished for PR 30057 at commit 806aab1.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-03-29T10:03:11Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41222/

yaooqinn · 2021-03-30T10:20:53Z

retest this please

SparkQA · 2021-03-30T10:47:19Z

Test build #136710 has finished for PR 30057 at commit 806aab1.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-03-30T10:53:36Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41292/

SparkQA · 2021-03-31T06:42:54Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41333/

SparkQA · 2021-03-31T07:17:05Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41333/

SparkQA · 2021-03-31T09:08:58Z

Test build #136751 has finished for PR 30057 at commit 81b1bd8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-04-02T07:41:02Z

retest this please

SparkQA · 2021-04-02T09:07:07Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41427/

SparkQA · 2021-04-02T13:05:20Z

Test build #136849 has finished for PR 30057 at commit 81b1bd8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

github-actions · 2021-04-15T10:55:49Z

Test build #751743803 for PR 30057 at commit 979e314.

SparkQA · 2021-04-15T11:56:49Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41991/

SparkQA · 2021-04-15T11:56:51Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41991/

SparkQA · 2021-04-15T15:04:40Z

Test build #137415 has finished for PR 30057 at commit 979e314.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-05-10T13:50:47Z

Gentle ping

github-actions · 2021-08-19T00:12:12Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

AngersZhuuuu · 2021-08-19T02:16:25Z

gentle ping @cloud-fan @viirya

AngersZhuuuu · 2021-08-20T07:05:19Z

retest this please

SparkQA · 2021-08-20T08:30:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47167/

SparkQA · 2021-08-20T09:09:20Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47167/

SparkQA · 2021-08-20T12:20:51Z

Test build #142666 has finished for PR 30057 at commit 979e314.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

CHENXCHEN · 2022-01-17T02:52:13Z

@AngersZhuuuu any update about this, thanks!

shrprasa · 2023-04-06T05:27:29Z

@AngersZhuuuu @cloud-fan We are facing this same issue with Spark 3.2. can we get this PR merged if there are no concerns?

AngersZhuuuu added 3 commits October 15, 2020 23:25

[SPARK-32838][SQL]Check DataSource insert command path with actual path

2f0ff97

Update InsertIntoHadoopFsRelationCommand.scala

c613d2a

add UT

9eb5073

dongjoon-hyun added the SQL label Oct 15, 2020

Update InsertIntoHadoopFsRelationCommand.scala

620956d

Update InsertIntoHadoopFsRelationCommand.scala

199aa8f

check with subDir

9d9ab88

AngersZhuuuu commented Oct 17, 2020

View reviewed changes

Merge branch 'master' into SPARK-32838

806aab1

Update InsertIntoHadoopFsRelationCommand.scala

81b1bd8

AngersZhuuuu mentioned this pull request Apr 1, 2021

[SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null #32018

Closed

Merge branch 'master' into SPARK-32838

979e314

github-actions bot added the Stale label Aug 19, 2021

github-actions bot closed this Aug 21, 2021

[SPARK-32838][SQL]Check DataSource insert command path with actual path #30057

[SPARK-32838][SQL]Check DataSource insert command path with actual path #30057

Conversation

AngersZhuuuu commented Oct 15, 2020 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

AngersZhuuuu commented Oct 15, 2020

SparkQA commented Oct 15, 2020

SparkQA commented Oct 15, 2020

SparkQA commented Oct 15, 2020

SparkQA commented Oct 15, 2020

SparkQA commented Oct 15, 2020

SparkQA commented Oct 16, 2020

SparkQA commented Oct 16, 2020

SparkQA commented Oct 16, 2020

SparkQA commented Oct 16, 2020

SparkQA commented Oct 16, 2020

SparkQA commented Oct 16, 2020

SparkQA commented Oct 17, 2020

SparkQA commented Oct 17, 2020

SparkQA commented Oct 17, 2020

AngersZhuuuu Oct 17, 2020

Choose a reason for hiding this comment

AngersZhuuuu commented Oct 17, 2020

SparkQA commented Oct 17, 2020

SparkQA commented Oct 17, 2020

SparkQA commented Oct 17, 2020

AngersZhuuuu commented Oct 19, 2020

dongjoon-hyun commented Oct 19, 2020

viirya commented Oct 19, 2020 • edited

SparkQA commented Mar 29, 2021

SparkQA commented Mar 29, 2021

yaooqinn commented Mar 30, 2021

SparkQA commented Mar 30, 2021

SparkQA commented Mar 30, 2021

SparkQA commented Mar 31, 2021

SparkQA commented Mar 31, 2021

SparkQA commented Mar 31, 2021

AngersZhuuuu commented Apr 2, 2021

SparkQA commented Apr 2, 2021

SparkQA commented Apr 2, 2021

github-actions bot commented Apr 15, 2021

SparkQA commented Apr 15, 2021

SparkQA commented Apr 15, 2021

SparkQA commented Apr 15, 2021

AngersZhuuuu commented May 10, 2021

github-actions bot commented Aug 19, 2021

AngersZhuuuu commented Aug 19, 2021

AngersZhuuuu commented Aug 20, 2021

SparkQA commented Aug 20, 2021

SparkQA commented Aug 20, 2021

SparkQA commented Aug 20, 2021

CHENXCHEN commented Jan 17, 2022

shrprasa commented Apr 6, 2023

AngersZhuuuu commented Oct 15, 2020 •

edited

viirya commented Oct 19, 2020 •

edited