[CELEBORN-1477] Refine the master/worker RESTful APIs

[turboFei]

What changes were proposed in this pull request?

In this pr, I refine the existing master/worker APIs.

Master API

---

• /${version}/conf
  • mapping: /conf
  • method: GET
  • params: none
  • return config key value map
• /${version}/conf/dynamic
  • mapping: /listDynamicConfigs
  • method: GET
  • params: level, tenant, name
  • return
• /${version}/thread_dump
  • mapping: /threadDump
• /${version}/applications
  • mapping: /applications
• /${version}/applications/top_disk_usages
  • mapping: /listTopDiskUsedApps
  • method: GET
• /${version}/applications/hostnames
  • mapping: /hostnames
  • method: GET
  • description: List all running application's LifecycleManager's hostnames of the cluster
• /${version}/shuffles
  • mapping: /shuffle
• /${version}/masters
  • mapping: /masterGroupInfo
  • method: GET
• /${version}/workers
  • mapping: /workerInfo
  • method: GET
  • params:
    • hostname(optional), if no worker hostname, return all
  • return: workers
• /${version}/workers/lost
  • mapping: /lostWorkers
  • method: GET
• /${version}/workers/excluded
  • mapping: /excludedWorkers
  • method: GET
• /${version}/workers/shutdown
  • mapping: /shutdownWorkers
  • method: GET
• /${version}/workers/decommissioned
  • mapping: /decommissionWorkers
  • method: GET
• /${version}/applications/hostnames
  • mapping: /hostnames
  • method: GET
  • description: List all running application's LifecycleManager's hostnames of the cluster
• /${version}/workers/events
  • mapping: /workerEventInfo
  • method: GET
  • description: List all worker event info of the master
• /${version}/workers/exclude
  • mapping: /exclude
  • method: POST
  • description: Excluded workers of the master add or remove the worker manually given worker id. The parameter add or remove specifies the excluded workers to add or remove, which value is separated by commas.
• /${version}/workers/events
  • mapping: /sendWorkerEvent
  • method: POST
  • description: For Master(Leader) can send worker events to manager workers. Legal types are 'None', 'Immediately', 'Decommission', 'DecommissionThenIdle', 'Graceful', 'Recommission', and the parameter workers are separated by commas.

Worker API

---

• /${version}/conf
  • mapping: /conf
  • method: GET
  • params: none
  • return config key value map
• /${version}/conf/dynamic
  • mapping: /listDynamicConfigs
  • method: GET
  • params: level, tenant, name
  • return
• /${version}/thread_dump
  • mapping: /threadDump
• /${version}/shuffles
• /${version}/shuffles/partitions
  • mapping: /listPartitionLocationInfo
  • method: GET
• /${version}/applications
  • mapping: /applications
• /${version}/applications/top_disk_usages
  • mapping: /listTopDiskUsedApps
  • method: GET
• /${version}/workers/unavailable_peers
  • mapping: /unavailablePeers
  • method: GET
• /${version}/workers
  • mapping:
    • /workerInfo
    • /isShutdown
    • /isDecomissioning
    • /isRegistered
• /${version}/exit
  • mapping: /exit

Master:


Worker:

Why are the changes needed?

Before, the api request/response are not normalized.

Does this PR introduce any user-facing change?

No, just introduce new /api/v1 APIs and keep current APIs unchanged.

How was this patch tested?

UT.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 590 lines in your changes missing coverage. Please review.

Project coverage is 38.46%. Comparing base (5dfbfd2) to head (5e5d9f3).
Report is 3 commits behind head on main.

Files	Patch %	Lines
...born/server/common/http/api/v1/dto/WorkerData.java	0.00%	107 Missing ⚠️
.../celeborn/server/common/http/api/v1/ApiUtils.scala	0.00%	43 Missing ⚠️
...eborn/server/common/http/api/v1/ConfResource.scala	0.00%	42 Missing ⚠️
.../common/http/api/v1/dto/PartitionLocationData.java	0.00%	38 Missing ⚠️
...erver/common/http/api/v1/dto/MasterCommitData.java	0.00%	33 Missing ⚠️
...leborn/server/common/http/api/v1/dto/WorkerId.java	0.00%	33 Missing ⚠️
...mmon/http/api/v1/dto/AppDiskUsageSnapshotData.java	0.00%	24 Missing ⚠️
...n/server/common/http/api/v1/dto/DynamicConfig.java	0.00%	24 Missing ⚠️
...server/common/http/api/v1/dto/MasterGroupData.java	0.00%	24 Missing ⚠️
...n/http/api/v1/dto/WorkerPartitionLocationData.java	0.00%	21 Missing ⚠️
... and 17 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2588      +/-   ##
==========================================
- Coverage   40.03%   38.46%   -1.56%     
==========================================
  Files         233      255      +22     
  Lines       14685    15275     +590     
  Branches     1337     1429      +92     
==========================================
- Hits         5877     5874       -3     
- Misses       8476     9069     +593     
  Partials      332      332              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[turboFei]

cc @RexXiong @SteNicholas @pan3793

[SteNicholas]

IMO, you are better to create new CIP for this refine. The REST API refine changes the public interfaces, which should give a CIP to discuss.

[turboFei]

IMO, you are better to create new CIP for this refine. The REST API refine changes the public interfaces, which should give a CIP to discuss.

Ok.