[#854][FOLLOWUP] feat(tez): Add Simple Fetched Allocator to allocation memory or disk for shuffle data

[lifeSo]

What changes were proposed in this pull request?

Add Simple Fetched Allocator to allcation memory or disk for shuffle data

Why are the changes needed?

Fix: #854

Does this PR introduce any user-facing change?

No.

How was this patch tested?

unit test case

[codecov-commenter]

Codecov Report

Merging #922 (a23b3fb) into master (6471c5e) will increase coverage by 2.02%.
The diff coverage is 35.08%.

@@             Coverage Diff              @@
##             master     #922      +/-   ##
============================================
+ Coverage     55.53%   57.56%   +2.02%     
- Complexity     2246     2365     +119     
============================================
  Files           341      334       -7     
  Lines         16626    15711     -915     
  Branches       1327     1414      +87     
============================================
- Hits           9234     9044     -190     
+ Misses         6863     6153     -710     
+ Partials        529      514      -15     

Impacted Files	Coverage Δ
...n/shuffle/impl/RssSimpleFetchedInputAllocator.java	35.08% <35.08%> (ø)

... and 41 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[lifeSo]

should we fix this test case ? it can run in my local computer.

Error:  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.625 s <<< FAILURE! - in org.apache.tez.runtime.library.common.shuffle.impl.RssSimpleFetchedInputAllocatorTest
Error:  testAllocate  Time elapsed: 0.622 s  <<< ERROR!
java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosTicket(Ljavax/security/auth/Subject;)Z
	at org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:657)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:848)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:807)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:680)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2978)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2968)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
	at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:360)
	at org.apache.tez.runtime.library.common.shuffle.DiskFetchedInput.<init>(DiskFetchedInput.java:53)
	at org.apache.tez.runtime.library.common.shuffle.impl.RssSimpleFetchedInputAllocator.allocate(RssSimpleFetchedInputAllocator.java:134)
	at org.apache.tez.runtime.library.common.shuffle.impl.RssSimpleFetchedInputAllocatorTest.testAllocate(RssSimpleFetchedInputAllocatorTest.java:72)

[lifeSo]

RssSimpleFetchedInputAllocatorTest

We delete the failed local disk test case to make test pass.
If you think the test case is necessary, we can add later, to not effect the PR

[jerqi]

should we fix this test case ? it can run in my local computer.

Error:  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.625 s <<< FAILURE! - in org.apache.tez.runtime.library.common.shuffle.impl.RssSimpleFetchedInputAllocatorTest
Error:  testAllocate  Time elapsed: 0.622 s  <<< ERROR!
java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosTicket(Ljavax/security/auth/Subject;)Z
	at org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:657)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:848)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:807)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:680)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2978)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2968)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
	at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:360)
	at org.apache.tez.runtime.library.common.shuffle.DiskFetchedInput.<init>(DiskFetchedInput.java:53)
	at org.apache.tez.runtime.library.common.shuffle.impl.RssSimpleFetchedInputAllocator.allocate(RssSimpleFetchedInputAllocator.java:134)
	at org.apache.tez.runtime.library.common.shuffle.impl.RssSimpleFetchedInputAllocatorTest.testAllocate(RssSimpleFetchedInputAllocatorTest.java:72)

You should fix this issue. You can use your own github pipeline to reproduce it.

[jerqi]

RssSimpleFetchedInputAllocatorTest

We delete the failed local disk test case to make test pass. If you think the test case is necessary, we can add later, to not effect the PR

Yes, I think it's necessary and you should set up an issue.

[jerqi]

LGTM, thanks @lifeSo , merged to master.