[serve] Support rolling updates for config changes

[edoakes]

Now that we do rolling updates for code changes, we should also consider doing the same for config changes. These may be very slow (e.g., if it loads a new heavyweight model) which could cause downtime if all replicas do it simultaneously.

This should be straightforward to implement: just add another state and relevant transitions for ReplicaState.UPDATING_CONFIG.

[edoakes]

import os
import time

import requests

import ray
from ray import serve

serve.start(detached=True)

@serve.deployment(version="1", num_replicas=2, user_config="test")
class MyDeployment:
    def __init__(self):
        print(f"Replica {serve.get_replica_context().replica_tag} starting up.")

    def reconfigure(self, config):
        print(f"Replica {serve.get_replica_context().replica_tag} reconfiguring.")
        time.sleep(1)

    def __call__(self):
        return os.getpid()

# Initial deployment, replicas come up as they're ready.
MyDeployment.deploy()

# Redeploy new version. Rolling update to tear one replica down at a time.
MyDeployment.options(version="2").deploy()

# Config change. Replicas are long-polling for config changes so they may all
# call reconfigure() with the new user_config at once, which could mean no
# requests are able to be handled if reconfigure is slow.
MyDeployment.options(version="2", user_config="updated").deploy()