Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding volume info to batch decorator #441

Closed
mattmcclean opened this issue Feb 20, 2021 · 5 comments
Closed

Adding volume info to batch decorator #441

mattmcclean opened this issue Feb 20, 2021 · 5 comments
Assignees
Labels
enhancement New feature or request

Comments

@mattmcclean
Copy link

Would be useful to add volume information into the batch decorator so that EBS, EFS or FSx volumes can be attached to the jobs to access data quickly for training, cache pre-trained weights, cache data from S3 etc.

Simplest version would be to pass in a mount path parameter into the decorator that will add two entries: volumes and mountPoints to the AWS Batch Job Definition containerProperties object.

https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html#containerProperties

@savingoyal
Copy link
Collaborator

Makes sense. I will add it in the next release.

@savingoyal savingoyal added the enhancement New feature or request label Feb 26, 2021
@savingoyal
Copy link
Collaborator

also, ulimits could be a great addition - https://gitter.im/metaflow_org/community?at=60403e47d2619a4f2e1f6766

@savingoyal savingoyal self-assigned this Mar 4, 2021
@samuelthan
Copy link

samuelthan commented Jun 2, 2021

Yeah, looking for to have this implement too !!

Our use case is different, relates to intergration with AWS CodeArtifact to pull pip libraries, And the only way i could think of, is to perform the AWS Codeartifact login first on the EC2 host of AWS Batch via launch template, then mount the host's volume path to the container.

container_path="/root/.config/pip/pip.conf",
host_path=~/.config/pip/pip.conf",

Hence this seems to be achievable via having the ability to modify the Job definition for the volume.

@savingoyal
Copy link
Collaborator

An update - a partial fix for this issue was part of Metaflow 2.3.5 - https://github.com/Netflix/metaflow/releases/tag/2.3.5#441

cc @samuelthan @mattmcclean

@NerdToMars
Copy link

hi, @savingoyal do you have any plan to fully support this?
I am new to metaflow. What I did is to modify the metaflow-aws terraform code to create efs with the correct subnet settings and mount efs when booting using the user data (https://aws.amazon.com/premiumsupport/knowledge-center/batch-mount-efs/).

Another tots is to modify the metaflow to create full volume json objects in the container definition, but will require user modify their cloud infra. Also clear documentation is required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants