[#951] build(operator): Add the HADOOP_VERSION for Docker image.

[qijiale76]

What changes were proposed in this pull request?

Add the HADOOP_VERSION for Docker image.

Why are the changes needed?

Fix: #951

Does this PR introduce any user-facing change?

No.

How was this patch tested?

By hand.

[jerqi]

@qijiale76 Could you correct the description of pr?

[jerqi]

@connorlwilkes Could you help review this pr if you have time?

[zuston]

I'm not sure is it necessary to bind hadoop version with image. If user don't use HDFS, this is weird

[jerqi]

I'm not sure is it necessary to bind hadoop version with image. If user don't use HDFS, this is weird

Shuffle Server must use the HDFS jars. Because some configuration use HDFS jars.

[zuston]

I'm not sure is it necessary to bind hadoop version with image. If user don't use HDFS, this is weird

Shuffle Server must use the HDFS jars. Because some configuration use HDFS jars.

Oh, I forgot this. This should be optimized in the later PR.

[zhengchenyu]

I'm not sure is it necessary to bind hadoop version with image. If user don't use HDFS, this is weird

@zuston

Hadoop version is introduced by me in #896
I think we should bind hadoop version, even if we don't use HDFS. We know mapreduce, spark and tez depend on hadoop. Because the API is changed, some version of computing framework may not be compatibility with some verison of hadoop.
In production cluster, use the computing framework which is not compatibility with hadoop may produce many unexpected error.

[zuston]

I'm not sure is it necessary to bind hadoop version with image. If user don't use HDFS, this is weird

@zuston

Hadoop version is introduced by me in #896 I think we should bind hadoop version, even if we don't use HDFS. We know mapreduce, spark and tez depend on hadoop. Because the API is changed, some version of computing framework may not be compatibility with some verison of hadoop. In production cluster, use the computing framework which is not compatibility with hadoop may produce many unexpected error.

You are right. I have checked that all the spark distribution are all bind to hadoop.

[zhengchenyu]

I'm not sure is it necessary to bind hadoop version with image. If user don't use HDFS, this is weird

@zuston
Hadoop version is introduced by me in #896 I think we should bind hadoop version, even if we don't use HDFS. We know mapreduce, spark and tez depend on hadoop. Because the API is changed, some version of computing framework may not be compatibility with some verison of hadoop. In production cluster, use the computing framework which is not compatibility with hadoop may produce many unexpected error.

You are right. I have checked that all the spark distribution are all bind to hadoop.

I just copy spark! ^_^

[connorlwilkes]

LGTM - fixes #951

[jerqi]

LGTM, thanks all.

[codecov-commenter]

Codecov Report

Merging #1027 (d116cfe) into master (d0f0b57) will decrease coverage by 1.02%.
Report is 41 commits behind head on master.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1027      +/-   ##
============================================
- Coverage     54.76%   53.75%   -1.02%     
+ Complexity     2526     2516      -10     
============================================
  Files           362      382      +20     
  Lines         19334    21694    +2360     
  Branches       1799     1799              
============================================
+ Hits          10588    11661    +1073     
- Misses         8116     9329    +1213     
- Partials        630      704      +74     

see 27 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more