[SPARK-27563][FOLLOWUP] Fix to download new release from dist.apache.org
#24544
Conversation
|
Test build #105209 has finished for PR 24544 at commit
|
|
I want to know how the previous version of spark was downloaded to |
| import scala.io.Source | ||
| try { | ||
| Source.fromURL("https://dist.apache.org/repos/dist/release/spark/").mkString | ||
| Source.fromURL(s"${releaseMirror}/spark").mkString |
There was a problem hiding this comment.
Maybe $releaseMirror is better.
There was a problem hiding this comment.
If we're picking nits and you change it again, it's not really a mirror. It's the primary dist server. asfDist or something.
|
Test build #105210 has finished for PR 24544 at commit
|
|
Wouldn't it be uploaded soon? |
|
Test build #105211 has finished for PR 24544 at commit
|
Maybe @cloud-fan knows, but we can avoid this error in the next release. |
|
After uploading, it takes a while for the resources to be available in the Mirror Network. This means the PR builder will be broken for a while. This fix LGTM |
dongjoon-hyun
changed the title
dist.apache.org
| import scala.io.Source | ||
| try { | ||
| Source.fromURL("https://dist.apache.org/repos/dist/release/spark/").mkString | ||
| Source.fromURL(s"${releaseMirror}/spark").mkString |
There was a problem hiding this comment.
If we're picking nits and you change it again, it's not really a mirror. It's the primary dist server. asfDist or something.
dongjoon-hyun
changed the title
dist.apache.orgdist.apache.org
|
I removed |
|
Merged to master |
7 participants
What changes were proposed in this pull request?
https://archive.apache.org/dist/spark/does not have latest Spark 2.4.3:This pr add
https://dist.apache.org/repos/dist/release/spark/to mirrors list to download latest Spark.https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105202/testReport/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
How was this patch tested?
manual tests: