-
Notifications
You must be signed in to change notification settings - Fork 364
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature][Doc] Access S3 bucket from Pods in EKS #958
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It'll be helpful to specify the Ray Serve use-case and have a workaround for that. Other than that, looks good to me!
# botocore==1.8.50 | ||
``` | ||
|
||
## Workaround solutions |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also a use-case where things will break - if you use the ray image < ray:2.3.0
in a RayService
and set the working directory to a zip file in a private S3 bucket, the Ray Serve application will not be able to start.
I have not tested this yet, but a potential solution could be to
- Create a custom docker image on top of
ray:2.3.0
and manually install the latestboto3
version (would defer to the Ray team to make sure this works) - Create a custom docker image on top of
ray:2.3.0
, add your code in the image, and set the working directory (inserveConfig
) to a local directory. This should get serve application running. Any model deployment code trying to access S3 should do that using boto3_example_2.py. (I will be testing this in the next few days, but will defer to the Serve team to make sure this works).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. Runtime environment also downloads working_dir
via boto3
(code). cc @architkulkarni
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it! Thanks for letting me know!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@architkulkarni is working on updating the boto3 version: ray-project/ray#33273.
80ac737
to
531c235
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Looks good to me!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
overall looks good to me.
Access S3 bucket from Pods in EKS
Why are these changes needed?
Users want to access S3 buckets in Ray Pods. See this slack thread for more context.
Related issue number
#956
ray-project/ray#33256
Checks