[SPARK-41448] Make consistent MR job IDs in FileBatchWriter and FileFormatWriter #38980

boneanxs · 2022-12-08T10:30:30Z

What changes were proposed in this pull request?

Make consistent MR job IDs in FileBatchWriter and FileFormatWriter

Why are the changes needed?

SPARK-26873 fix the consistent issue for FileFormatWriter, but SPARK-33402 break this requirement by introducing a random long, we need to address this to expects identical task IDs across attempts for correctness.

Also FileBatchWriter doesn't follow this requirement, need to fix it as well.

Does this PR introduce any user-facing change?

no

How was this patch tested?

boneanxs · 2022-12-09T06:20:50Z

@cloud-fan @rdblue @dongjoon-hyun @steveloughran could you please take a look?

steveloughran

looks ok from the perspective of someone who has written some of the committers but isn't confident they understand all the nuances of spark commit protocols (i never knew that FileBatchWriter added a third way to to try and commit non-stream jobs...)

steveloughran · 2022-12-09T10:45:47Z

core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriterUtils.scala

+  }
+
+  /**
+   * Create a job ID.


how about extending the comment here by noting that the job id needs to be unique across all jobs (linking to SPARK-33402) and consistently across places used (SPARK-26873 + SPARK-41448). That way, whoever next goes near the code knows what is needed

steveloughran · 2022-12-09T10:50:25Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala

@@ -201,14 +201,14 @@ object FileFormatWriter extends Logging {
        rdd
      }

-      val jobIdInstant = new Date().getTime
+      val jobTrackerID = SparkHadoopWriterUtils.createJobTrackerID(new Date())


i must have missed this -but also we've not had any reports of the timstamp clash surfacing.

that's probably because the s3a and abfs/gcs committers all pick up the uuid in "spark.sql.sources.writeJobUUID" in preference to anything else, and they are being generated uniquely

steveloughran · 2022-12-09T11:03:58Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala

@@ -29,6 +29,9 @@ import org.apache.spark.sql.execution.datasources.{DynamicPartitionDataSingleWri
 case class FileWriterFactory (
    description: WriteJobDescription,
    committer: FileCommitProtocol) extends DataWriterFactory {
+
+  private val jobId = SparkHadoopWriterUtils.createJobID(new Date, 0)
+


I wonder if a uuid should be created here which is then passed down in that "spark.sql.sources.writeJobUUID" option in createTaskAttemptContext(). It would be consistent with the rest and the mapreduce manifest committer would pick it up. (so would the s3a one, but as you can't do dynamic partitioning there it's less relevant)

AmplabJenkins · 2022-12-10T15:39:01Z

Can one of the admins verify this patch?

cloud-fan

good catch!

…ormatWriter ### What changes were proposed in this pull request? Make consistent MR job IDs in FileBatchWriter and FileFormatWriter ### Why are the changes needed? [SPARK-26873](https://issues.apache.org/jira/browse/SPARK-26873) fix the consistent issue for FileFormatWriter, but [SPARK-33402](https://issues.apache.org/jira/browse/SPARK-33402) break this requirement by introducing a random long, we need to address this to expects identical task IDs across attempts for correctness. Also FileBatchWriter doesn't follow this requirement, need to fix it as well. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Closes #38980 from boneanxs/SPARK-41448. Authored-by: Hui An <hui.an@shopee.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 7801666) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

cloud-fan · 2022-12-12T10:21:19Z

thanks, merging to master/3.3./3.2

…ormatWriter ### What changes were proposed in this pull request? Make consistent MR job IDs in FileBatchWriter and FileFormatWriter ### Why are the changes needed? [SPARK-26873](https://issues.apache.org/jira/browse/SPARK-26873) fix the consistent issue for FileFormatWriter, but [SPARK-33402](https://issues.apache.org/jira/browse/SPARK-33402) break this requirement by introducing a random long, we need to address this to expects identical task IDs across attempts for correctness. Also FileBatchWriter doesn't follow this requirement, need to fix it as well. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Closes apache#38980 from boneanxs/SPARK-41448. Authored-by: Hui An <hui.an@shopee.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…riter.executeTask ### What changes were proposed in this pull request? This PR is a followup of #38939 that fixes a logical conflict during merging PRs, see #38980 and #38939. ### Why are the changes needed? To recover the broken build. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually tested: ``` ./build/sbt -Phive clean package ``` Closes #39194 from HyukjinKwon/SPARK-41407. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

yikf · 2023-02-17T05:45:45Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala

@@ -29,6 +29,9 @@ import org.apache.spark.sql.execution.datasources.{DynamicPartitionDataSingleWri
 case class FileWriterFactory (
    description: WriteJobDescription,
    committer: FileCommitProtocol) extends DataWriterFactory {
+
+  private val jobId = SparkHadoopWriterUtils.createJobID(new Date, 0)


@boneanxs It looks like tha jobId is not serializable?

it does implement WritableComparable; you could use WritableConverter to do the marshalling if you don't just want to do via a string

…ormatWriter ### What changes were proposed in this pull request? Make consistent MR job IDs in FileBatchWriter and FileFormatWriter ### Why are the changes needed? [SPARK-26873](https://issues.apache.org/jira/browse/SPARK-26873) fix the consistent issue for FileFormatWriter, but [SPARK-33402](https://issues.apache.org/jira/browse/SPARK-33402) break this requirement by introducing a random long, we need to address this to expects identical task IDs across attempts for correctness. Also FileBatchWriter doesn't follow this requirement, need to fix it as well. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Closes apache#38980 from boneanxs/SPARK-41448. Authored-by: Hui An <hui.an@shopee.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 7801666) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…inator should abort stage when committed file not consistent with task status ### What changes were proposed in this pull request? Revert #36564 According to discuss #36564 (comment) When spark commit task will commit to committedTaskPath `${outputpath}/_temporary//${appAttempId}/${taskId}` So in #36564 's case, since before #38980, each task's job id's date is not the same, when the task writes data success but fails to send back TaskSuccess RPC, the task rerun will commit to a different committedTaskPath then causing data duplicated. After #38980, for the same task's different attempts, the TaskId is the same now, when re-run task commit, will commit to the same committedTaskPath, and hadoop CommitProtocol will handle such case then data won't be duplicated. Note: The taskAttemptPath is not same since in the path contains the taskAttemptId. ### Why are the changes needed? No need anymore ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existed UT ### Was this patch authored or co-authored using generative AI tooling? No Closes #46696 from AngersZhuuuu/SPARK-48292. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…inator should abort stage when committed file not consistent with task status ### What changes were proposed in this pull request? Revert apache#36564 According to discuss apache#36564 (comment) When spark commit task will commit to committedTaskPath `${outputpath}/_temporary//${appAttempId}/${taskId}` So in apache#36564 's case, since before apache#38980, each task's job id's date is not the same, when the task writes data success but fails to send back TaskSuccess RPC, the task rerun will commit to a different committedTaskPath then causing data duplicated. After apache#38980, for the same task's different attempts, the TaskId is the same now, when re-run task commit, will commit to the same committedTaskPath, and hadoop CommitProtocol will handle such case then data won't be duplicated. Note: The taskAttemptPath is not same since in the path contains the taskAttemptId. ### Why are the changes needed? No need anymore ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existed UT ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#46696 from AngersZhuuuu/SPARK-48292. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

Make consistent MR job IDs in DSV2 and FileFormatWriter

659c938

github-actions bot added CORE SQL labels Dec 8, 2022

steveloughran reviewed Dec 9, 2022

View reviewed changes

cloud-fan approved these changes Dec 12, 2022

View reviewed changes

cloud-fan closed this in 7801666 Dec 12, 2022

HyukjinKwon mentioned this pull request Dec 23, 2022

[SPARK-41407][SQL][FOLLOW-UP] Use string jobTrackerID for FileFormatWriter.executeTask #39194

Closed

yikf reviewed Feb 17, 2023

View reviewed changes

yikf mentioned this pull request Feb 17, 2023

[KYUUBI #4338] Bump Spark from 3.3.1 to 3.3.2 apache/kyuubi#4339

Closed

3 tasks

dongjoon-hyun mentioned this pull request Mar 3, 2023

[SPARK-42478] Make a serializable jobTrackerId instead of a non-serializable JobID in FileWriterFactory #40064

Closed

boneanxs deleted the SPARK-41448 branch June 3, 2024 06:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-41448] Make consistent MR job IDs in FileBatchWriter and FileFormatWriter #38980

[SPARK-41448] Make consistent MR job IDs in FileBatchWriter and FileFormatWriter #38980

boneanxs commented Dec 8, 2022 •

edited

boneanxs commented Dec 9, 2022

steveloughran left a comment

steveloughran Dec 9, 2022

steveloughran Dec 9, 2022

steveloughran Dec 9, 2022

AmplabJenkins commented Dec 10, 2022

cloud-fan left a comment

cloud-fan commented Dec 12, 2022

yikf Feb 17, 2023

steveloughran Feb 17, 2023

[SPARK-41448] Make consistent MR job IDs in FileBatchWriter and FileFormatWriter #38980

[SPARK-41448] Make consistent MR job IDs in FileBatchWriter and FileFormatWriter #38980

Conversation

boneanxs commented Dec 8, 2022 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

boneanxs commented Dec 9, 2022

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran Dec 9, 2022

Choose a reason for hiding this comment

steveloughran Dec 9, 2022

Choose a reason for hiding this comment

steveloughran Dec 9, 2022

Choose a reason for hiding this comment

AmplabJenkins commented Dec 10, 2022

cloud-fan left a comment

Choose a reason for hiding this comment

cloud-fan commented Dec 12, 2022

yikf Feb 17, 2023

Choose a reason for hiding this comment

steveloughran Feb 17, 2023

Choose a reason for hiding this comment

boneanxs commented Dec 8, 2022 •

edited