[fix][offload] use filesystem offloader with class not found error

[ethqunzhong]

Fixes #16955

Motivation

fix the problem about Failing to start standalone pulsar service with tiered storage due to class not found problem
The reason is the lack of dependencies for the org.apache.hadoop.hdfs.DistributedFileSystem class.

Modifications

add DistributedFileSystem class dependency hadoop-client in pom.xml

Verifying this change

• Make sure that the change passes the CI checks.

This change is already covered by existing tests.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: ethqunzhong#5

[ethqunzhong]

@eolivelli PTAL

[tisonkun]

It seems the issued class is located at hadoop-hdfs-client. Could you elaborate a bit why hadoop-client is needed here?

[ethqunzhong]

It seems the issued class is located at hadoop-hdfs-client. Could you elaborate a bit why hadoop-client is needed here?

@tisonkun Sorry for not timely reply. What you said is correct. After testing, simplifying the dependencies still works.

[eolivelli]

+1

it would be great to setup an integration test one day.
I assume that you tested it manually

Is this problem also on Pulsar 3.0 ?
should we cherry-pick this PR to branch-3.0 ?

[codecov-commenter]

Codecov Report

Merging #20365 (3fb0e8f) into master (d85736c) will increase coverage by 0.07%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master   #20365      +/-   ##
============================================
+ Coverage     73.00%   73.07%   +0.07%     
- Complexity    31962    31991      +29     
============================================
  Files          1867     1867              
  Lines        138661   138661              
  Branches      15235    15235              
============================================
+ Hits         101227   101328     +101     
+ Misses        29382    29301      -81     
+ Partials       8052     8032      -20     

Flag	Coverage Δ
inttests	24.13% <ø> (+0.04%)	⬆️
systests	24.97% <ø> (-0.08%)	⬇️
unittests	72.36% <ø> (+0.08%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 65 files with indirect coverage changes

[tisonkun]

Thank you!

[ethqunzhong]

+1

it would be great to setup an integration test one day. I assume that you tested it manually

Is this problem also on Pulsar 3.0 ? should we cherry-pick this PR to branch-3.0 ?

yes. tested it manually and branch-3.0 has the same problem.

[tisonkun]

Merging...

cc @RobertIndie I guess we can include this patch for 3.0.x on 3.0.2 at the earliest?

[Technoboy-]

I see the mentioned issue #16955 with broker-2.10, so we need at least cherry-pick this to 2.10.