[RayJob] Set the timeout of the HTTP client from 2 mins to 2 seconds

[kevin85421]

Why are these changes needed?

Before #1733, the Ray dashboard required approximately 5 seconds to be ready to serve requests after the Ray head was running and ready. Hence, #1000 increases the timeout of the HTTP client to 2 mins to hotfix this issue. However, if the Ray dashboard crashes, the KubeRay operator will stuck there for 2 mins which is not acceptable.

After #1733, the readiness probe will verify the readiness of the Ray dashboard. Consequently, if the head Pod is ready, it implies that the Ray dashboard is also prepared to serve requests. Hence, we can set the timeout from 2 mins back to 2 seconds.

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

Deploy RayJob with this YAML 25 times.

[kevin85421]

cc @andrewsykim

[kevin85421]

Thanks to @Irvingwangjr for reminding me of this issue!

[Irvingwangjr]

Thanks to @Irvingwangjr for reminding me of this issue!

We test it by using the memory_benmark_utils.py with slight modification, using the http-mode to submit the job after head Pod ReadinessProbe is ready.
We set RayJobConcurrency as 5, and RayClusterConcurrency as 15; submit 1000 RayJob with 5 seconds interval.
Everything looks great