-
Notifications
You must be signed in to change notification settings - Fork 621
-
Notifications
You must be signed in to change notification settings - Fork 621
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow the setting of the bootDiskSize with Google pipelines #1331
Comments
Good, looking for receiving a code contribution from you. |
I'm sorry if that sounded offensive. I just meant if the option is there, that means the developers already thought of/are developing the feature. I would love to contribute however I'm sure this is between GCP and Nextflow. In GCP we can customize disk size, and since Nextflow uses GCP, that functionality is there and just needs to be implemented. Once again, I apologize for wording it the way I did, frustrated since Nextflow is supposed to be the answer to all my problems. |
No problem, I appreciate the enthusiasm. It should be possible but we have other priorities. The code is available so that anybody can use, learn and propose improvements. |
Pipelines API definitely has this functionality.
The size of the boot disk. Defaults to 10 (GB). |
In
|
It should work. |
If I was to make changes "locally", how could I then install that version of Nextflow? Basically, how do you guys package the nextflow? I can try getting it to work, if it does, I can fork, and upload to a feature branch. |
|
|
I just understood that Anyway. I don't think I can get it to work. My groovy skills are insufficient. Until this feature is available, I will look for another solution. If the feature is added, I'll come back to implementing the workflow though Nextflow. |
To be clear, the only change needed here is for the boot disk. The scratch disk is already configurable via the I can take a look at implementing this, however I can't provide a reliable timeframe right now. @pditommaso Any thoughts on which directive to use for this? There appears to be a @daudn In the meantime, can you comment on what takes up most of the space in your image? 7GB seems like alot. |
Actually, I've implemented a workflow in Kubernetes, then using RabbitMQ with manual autoscaling of VM Instances. Eventually had a look at Nextflow which seems like the answer if only I can get around the limitation of the 10 GB disk (which is why im quite persistent on this feature) The docker image is so large because the image has a third party tool installed to do HLA Typing. This tool takes up 6GB of space after being installed (I've tried cleaning up the image), the size of third party tool is not in my hands, I'm afraid. When I was doing this process manually, I had prebuilt (VMI) images on Google Cloud. And whenever a job was to be processed, the instance that booted up already had the previous 'layers' of the docker image and so it was a much quicker process to get the :latest version. The manual workflow I built is def efficient, but it has many moving parts and so a higher chance for it to fail, and would also be difficult to maintain in the long run as compared to Nextflow. Would really appreciate it if you guys could allow customisation of local disk storage. |
@mozack The idea is to deprecate the |
@mozack any update? |
@daudn we are discussing with the google team regarding this, tho no ETA at this time. stay tuned. |
Any updates? Had a look at the release but it doesn't seem to include this. |
It will be included in the next stable release 20.01.0 |
@pditommaso thank you for the update, looking forward to the release! |
The new
You can try it with the latest snapshot:
|
Bug report
In process declaration, the disk directive does not work.
When I run this workflow:
I get an instance with 10 GB (local) and 500GB pipeline-worker persistent disks.
The Docker image I have is 7 GB large and the workflow runs out of memory since it cannot complete the Docker pull.
Is this issue going to be handled any time soon? I increased my persistent disk quota to accommodate 500GB for each instance. But there is no way around using a large docker image (since 10GB local disk)
Please help, or update. If this is not going to change then I need to change the entire design of my pipeline.
The text was updated successfully, but these errors were encountered: