[serve] Rolling updates for user_config changes

[edoakes]

Why are these changes needed?

Config updates may be slow and cause replicas to be unavailable while updating, so we should do rolling updates for these as well as code changes.

This PR uses the existing codepath for rolling updates but adds the hash of the user_config to the version used to check for required updates (this requires the user_config to be hashable, see BackendVersion implementation for details.

When the user-provided version is mismatched, we perform the existing rolling update. When only the user_config is mismatched, we all the reconfigure method and wait for it to return. This transitions the replica back to the STARTING_OR_UPDATING state.

Note that to maintain backwards compatibility this leaves the long poll client for backend configs in place, but once we remove the legacy update_backend_config codepath we can completely remove this from the backend_worker.

Note that #15820 is dependent on this change so that we can properly wait for config changes to go through (i.e., endpoints to be available).

Related issue number

Closes #15136

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[simon-mo]

cc @caitengwei @liuyang-my This PR should help with common reloading models into running backend use case.

[simon-mo]

LGTM overall, just few small comments.