[Ready for review] Initial http jobs server on head node

[jiaodong]

After syncing with Ant this week, they confirmed this folder is not used anymore as they use internal implementation only. So in this PR we can just strip it to the minimum and redo a simple http server.

Changes

• Removed existing job server basically completely .. with only class JobHead(dashboard_utils.DashboardHeadModule): left
• Added pydantic types for each API's request / request as well as job spec
• Extended dashboard_utils.rest_response() to optionally not google style keys (job_id -> jobId)
• Added and verified local + s3 working_dir tests passed with http api calls.
• Changed python job_manager not to call supervisor actor if None is returned (actor exited)

Test plan

Closes #19413

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[edoakes]

Looks pretty good. Do you plan to use pydantic types for validating inputs?

[edoakes]

can we centralize this ray.init logic to one place? Maybe make it a decorator?

Also, we should use some kind of internal namespace ("_ray_internal_jobs"?)

[jiaodong]

decorator sounds great, looks like ray.init() has to happen at api calls (run() didn't work either) but at least only called once.

internal namespace sounds great too !

[jiaodong]

i will refuse to ship job server without pydantic type checks so this has to be included

[edoakes]

Windows failures look unrelated and same tests also have failures on master.