Please sign in to comment.
[SPARK-10878][CORE] Fix race condition when multiple clients resolves…
… artifacts at the same time ## What changes were proposed in this pull request? When multiple clients attempt to resolve artifacts via the `--packages` parameter, they could run into race condition when they each attempt to modify the dummy `org.apache.spark-spark-submit-parent-default.xml` file created in the default ivy cache dir. This PR changes the behavior to encode UUID in the dummy module descriptor so each client will operate on a different resolution file in the ivy cache dir. In addition, this patch changes the behavior of when and which resolution files are cleaned to prevent accumulation of resolution files in the default ivy cache dir. Since this PR is a successor of #18801, close #18801. Many codes were ported from #18801. **Many efforts were put here. I think this PR should credit to Victsm .** ## How was this patch tested? added UT into `SparkSubmitUtilsSuite` Author: Kazuaki Ishizaki <email@example.com> Closes #21251 from kiszk/SPARK-10878. (cherry picked from commit d3c426a) Signed-off-by: Marcelo Vanzin <firstname.lastname@example.org>
- Loading branch information...
Showing with 47 additions and 10 deletions.