Add serve ft nightly #19125
Add serve ft nightly #19125
Conversation
python/ray/serve/storage/kv_store.py
Outdated
| aws_access_key_id=aws_access_key_id, | ||
| aws_secret_access_key=aws_secret_access_key, | ||
| aws_session_token=aws_session_token) | ||
| aws_access_key_id=os.environ.get("AWS_ACCESS_KEY_ID", None), |
There was a problem hiding this comment.
i think previously we're not really getting these tokens in make_kv_store, so adding it at client level with actual s3 integration.
There was a problem hiding this comment.
boto should fall back to environ if we pass in None, so this is not necessary.
| For product testing, we skip the part of actually starting new cluster as | ||
| it's Job Manager's responsibility, and only re-deploy to the same cluster | ||
| with remote checkpoint. |
There was a problem hiding this comment.
Hmm, I think we should still fully test that path where the entire cluster goes down. If we just test the controller being killed there may be some hidden assumptions about state in the GCS or something.
Can you just use cluster_utils and blow away the whole cluster and create a new one?
There was a problem hiding this comment.
I would like to completely start new ray cluster if possible as well, but think cluster_utils is for local tests, is there an equivalent on product to blow away current cluster and start new one ?
There was a problem hiding this comment.
Hmm I'm not sure. cluster_utils would be a good first step, it at least clears all process-level state
There was a problem hiding this comment.
Ask internally on slack if there's a best way to do this? Worst case you just use the SDK to kill the cluster and start a new one.
There was a problem hiding this comment.
cluster_utils will run much faster though and it'll be nice to be able to run it locally for testing. if possible maybe make one test with both "backends" and start with just cluster_utils?
There was a problem hiding this comment.
i asked a few product folks, our SDK has start / terminate cluster APIs but the issues is current releaser/e2e.py is assuming it runs a single script as if on laptop, in single ray cluster. Doing this in our test script might not work well and most feasible way is probably extend existing test script's yaml field to extend a multi-cluster setup.
For now im trying to work with multiple local clusters on product, which should terminate quick but also give us more coverage compare to current revision.
Create a serve FT test for both local testing as well as product nightly.
It uses s3://serve-nightly-tests/fault-tolerant-test-checkpoint as checkpoint path with 7 days TTL, each checkpoint is 260 Bytes in size.
In each test run, it will try to run test in uuid4 namespace with unique storage key, go through deploy() -> kill controller & ray & cluster -> resume in same namespace -> check endpoints availability process, then attempt to clean up checkpoint file and exit.
Works for both local disk and s3 path, since we don't have sufficient infra for multi cluster yet, both local and product tests will be using the same cluster_utils to run local cluster, except using different storage paths.
Checks
scripts/format.shto lint the changes in this PR.