[SPARK-33288][YARN][FOLLOW-UP][test-hadoop2.7] Fix type mismatch error

[wangyum]

What changes were proposed in this pull request?

This pr fix type mismatch error:

[error] /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-2.3/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:320:52: type mismatch;
[error]  found   : Long
[error]  required: Int
[error]         Resource.newInstance(resourcesWithDefaults.totalMemMiB, resourcesWithDefaults.cores)
[error]                                                    ^
[error] one error found

Why are the changes needed?

Fix compile issue.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing test.

[wangyum]

Hadoop 2.7 only support Int type:
https://github.com/apache/hadoop/blob/release-2.7.4-RC0/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java#L56

[SparkQA]

Test build #131090 has finished for PR 30375 at commit 6002504.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[wangyum]

cc @tgravescs

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35693/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35693/

[wangyum]

retest this please.

[SparkQA]

Test build #131093 has finished for PR 30375 at commit 6002504.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35696/

[mridulm]

Given the value is in MB's, perhaps we should maintain it as int and not do .toInt from a long, thoughts @tgravescs ?

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35696/

[dongjoon-hyun]

Thank you, @wangyum .

Ya. Hadoop 2.7 profile is easily outdated in these days. Given that, hopefully, Apache Spark 3.1 can give more benefits and help to migrate to Hadoop 3.2+ and we can remove Hadoop 2.7 maintenance burden eventually.

BTW, @wangyum and @mridulm and @tgravescs . Do you think it's possible for us to start discussion for dropping Hadoop 2.7 at Apache Spark 3.2?

[mridulm]

BTW, @wangyum and @mridulm and @tgravescs . Do you think it's possible for us to start discussion for dropping Hadoop 2.7 at Apache Spark 3.2?

Is the proposal to drop 2.7 and move to 2.10 ? Or to drop 2.x entirely and move to hadoop 3.x ?

Hadoop 2.7.4, which we use, was released 3 years back - while 2.10 was released 1 year back.
Given hadoop 3.x is a major version change, IMO we need to support 2.x branch for longer until larger set of users have migrated.

Assuming we are still supporting 2.10, will the issues we are facing get addressed ?

[dongjoon-hyun]

My initial question was about dropping hadoop-2.x profile completely like we did for hive-1.2, @mridulm . It was just a question for the possibility~

If you think so, I will focus on supporting Hadoop 2.x LTS officially in Apache Spark 3 too.

[dongjoon-hyun]

I made a PR to protect Hadoop 2 profile. Could you review that, @mridulm ?

• [SPARK-33454][INFRA] Add GitHub Action job for Hadoop 2 #30378

[wangyum]

+1 for supporting Hadoop 2.x LTS officially in Apache Spark 3.

[wangyum]

Merged to master.

[dongjoon-hyun]

Thank you, @wangyum .

[tgravescs]

thanks for fixing, I don't think we should drop Hadoop 2.x, I think to many people are still using it.

[mridulm]

@tgravescs any thoughts on this comment ?
This probably got merged before you got to it.

[tgravescs]

sorry missed your comment @mridulm yeah we can just use int since this are all MiB, I'll file a followup and put up a PR.