[CELEBORN-1356] Split rpc module into rpc_app and rpc_service

[mridulm]

What changes were proposed in this pull request?

Split the rpc transport module into rpc_app and rpc_service to allow for them to be independently configured.

Why are the changes needed?

We need the ability to independently configure communication between application components (driver/executors in spark applications) and those to/from Celeborn service (master/workers) components.

This is particularly relevant for TLS support where applications might be running with TLS disabled for their rpc services or using self-signed certificates (see CELEBORN-1354 for an example), while services would have signed certs.

Does this PR introduce any user-facing change?

Yes, it allows users to independently configure rpc env within the application and those to/from services.
Backward compatibility is maintained - and so existing rpc is the fallback in case rpc_app or rpc_service config is not found.

How was this patch tested?

Unit tests were enhanced, existing tests pass.

[mridulm]

If it makes it easier to review, I can split it into two - updates to CelebornConf, which is mostly config updates; and wiring up the new transport module to RpcEnv and usages.
Please let me know if that helps.

[mridulm]

The test failure seems to be some timeout issue ? this is ready for review.

+CC @otterc , @waitinfuture, @FMX , @pan3793

[FMX]

I think split rpc module looks fine to me, but there is no need to introduce something not used currently.

[pan3793]

code change lgtm, could you pls add an item to the migration guide?

[mridulm]

code change lgtm, could you pls add an item to the migration guide?

I was planning to add that as part of https://issues.apache.org/jira/browse/CELEBORN-1353 .. should I be adding it as part of this PR instead ?

Edit: added a note in CELEBORN-1353 on migration guide

[pan3793]

@mridulm thanks for opening a dedicated ticket for docs, merged to main for 0.5

[mridulm]

Thanks for the reviews @FMX and @pan3793 !