[SPARK-27525][SQL] Upgrade commons-httpclient to 3.1 when interacting with different versions of the HiveMetastoreClient

[wangyum]

What changes were proposed in this pull request?

Our jenkins always failed download commons-httpclient#commons-httpclient;3.0.1 when testing with hadoop-3.2:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104768/testReport/org.apache.spark.sql.hive.execution/Hive_2_1_DDLSuite/SPARK_21617__ALTER_TABLE_for_non_compatible_DataSource_tables

The root cause is it evicte commons-httpclient#commons-httpclient;3.0.1 when testing with hadoop-2.7, but hadoop-3.2 didn't(Hadoop removes commons-httpclient since HADOOP-13382(Hadoop 2.8.0)):

2019-04-23 02:13:57.781 - stderr> 	:: evicted modules:
2019-04-23 02:13:57.782 - stderr> 	log4j#log4j;1.2.16 by [log4j#log4j;1.2.17] in [default]
2019-04-23 02:13:57.782 - stderr> 	org.slf4j#slf4j-api;1.7.5 by [org.slf4j#slf4j-api;1.7.10] in [default]
2019-04-23 02:13:57.782 - stderr> 	org.slf4j#slf4j-log4j12;1.7.5 by [org.slf4j#slf4j-log4j12;1.7.10] in [default]
2019-04-23 02:13:57.782 - stderr> 	org.apache.hadoop#hadoop-annotations;2.6.0 by [org.apache.hadoop#hadoop-annotations;2.7.4] in [default]
2019-04-23 02:13:57.782 - stderr> 	org.codehaus.jackson#jackson-core-asl;1.9.2 by [org.codehaus.jackson#jackson-core-asl;1.9.13] in [default]
2019-04-23 02:13:57.782 - stderr> 	org.codehaus.jackson#jackson-mapper-asl;1.9.2 by [org.codehaus.jackson#jackson-mapper-asl;1.9.13] in [default]
2019-04-23 02:13:57.782 - stderr> 	org.codehaus.jackson#jackson-jaxrs;1.9.2 by [org.codehaus.jackson#jackson-jaxrs;1.9.13] in [default]
2019-04-23 02:13:57.782 - stderr> 	org.codehaus.jackson#jackson-xc;1.9.2 by [org.codehaus.jackson#jackson-xc;1.9.13] in [default]
2019-04-23 02:13:57.782 - stderr> 	org.apache.hadoop#hadoop-yarn-common;2.6.0 by [org.apache.hadoop#hadoop-yarn-common;2.7.4] in [default]
2019-04-23 02:13:57.782 - stderr> 	org.apache.hadoop#hadoop-yarn-api;2.6.0 by [org.apache.hadoop#hadoop-yarn-api;2.7.4] in [default]
2019-04-23 02:13:57.782 - stderr> 	org.apache.hadoop#hadoop-yarn-server-common;2.6.0 by [org.apache.hadoop#hadoop-yarn-server-common;2.7.4] in [default]
2019-04-23 02:13:57.782 - stderr> 	commons-httpclient#commons-httpclient;3.0.1 by [commons-httpclient#commons-httpclient;3.1] in [default]

This pr upgrade commons-httpclient#commons-httpclient;3.0.1 to commons-httpclient#commons-httpclient;3.1 when interacting with different versions of the HiveMetastoreClient to fix this issue.

How was this patch tested?

Exist unit tests and manual tests

[SparkQA]

Test build #104773 has finished for PR 24423 at commit 8e8546e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Hm, you're saying this makes something not work or is that a typo?
Why does it fail to download? that seems like the issue. Can we exclude this if it's used?

[wangyum]

@srowen This issue is very similar to #24388

[SparkQA]

Test build #104831 has finished for PR 24423 at commit 8af647f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Can you help me understand this more... shouldn't we have an entry in pom.xml to manage this version?
3.0.1 does exist: https://search.maven.org/artifact/commons-httpclient/commons-httpclient/3.0.1/jar

[wangyum]

Yes. 3.0.1 exist and I can't reproduce this issue on my local machine.

May be it's our jenkins issue. I have tested it a few times and only once ran successfully at amp-jenkins-worker-03.

[srowen]

Hm, @shaneknapp is it possible the Ivy / Maven cache is corrupted on some workers? that's the only reason that comes to mind that would explain why it thinks it can't get the dependency. Is it possible to delete them? ~/.ivy2/cache and ~/.m2/repository?

[wangyum]

Please check these workers: amp-jenkins-worker-06, amp-jenkins-worker-04.

[shaneknapp]

Please check these workers: amp-jenkins-worker-06, amp-jenkins-worker-04.

you caught me w/a little bit of downtime @ the spark summit. i'll do this now.

[shaneknapp]

marked these workers as offline and WOW! the .m2/respository dirs are massive and taking ~minutes to delete. :)

[shaneknapp]

45 minutes later, all the repos have been wiped and the workers are back up.

[shaneknapp]

test this please

[shaneknapp]

since the build system is absolutely DESERTED right now, i marked all of the other workers offline and will launch a build of this PR on either -04 or -06.

[shaneknapp]

test this please

[shaneknapp]

Test build #104912 has started for PR 24423 at commit 8af647f.

phew. i was starting to panic and wondering why this wasn't being triggered. :)

anyways, this is building on -06. now we wait!

[shaneknapp]

Test build #104912 has started for PR 24423 at commit 8af647f.

phew. i was starting to panic and wondering why this wasn't being triggered. :)

anyways, this is building on -06. now we wait!

argh... i may have jumped the gun on this. the ivy cache dirs are based on the executor that the build is running on for each particular worker, not in ~jenkins/.ivy/....

if this fails again, i'll try and get to wiping these caches later tonite, but since i'm working at the summit, most likely tomorrow.

[wangyum]

Sorry @shaneknapp please verify this issue on this PR: #24044. It only happens on hadoop-3.2.
All test passes or HiveExternalCatalogSuite test failures are what we expect.

[SparkQA]

Test build #104912 has finished for PR 24423 at commit 8af647f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[wangyum]

@shaneknapp
amp-jenkins-worker-03, amp-jenkins-worker-04 and amp-jenkins-worker-06 works fine.
amp-jenkins-worker-02 and amp-jenkins-worker-05 also has this issue.
I didn't verify amp-jenkins-worker-01.

[shaneknapp]

ill do my best today to get the workers fixed... the main jenkins process is misbehaving (again), and i'm also dealing w/a major sewage problem at my house right now.

[shaneknapp]

wiping these dirs on -01, -02 and -05 now.

[shaneknapp]

wiping these dirs on -01, -02 and -05 now.

...and done.

[wangyum]

@shaneknapp Thank you