-
Notifications
You must be signed in to change notification settings - Fork 61
ch-run with cachedir - confusing behavior #1770
Comments
Hello Nilkas, Thanks for reaching out! Running images from the storage directory has never been supported, and the reason is that Charliecloud needs full control over everything in the storage directory. That is, That said, there are a couple of workarounds I can think of. The standard way to get a writeable image is something like:
That is, export the image from storage to a directory of your choice, then run that directory. However, with your use case, which sounds something like
you may not need to build an image at all. If
Does that help? |
Hello Reid, I guess I should add that since most the stuff I do is bio-related I typically do not build the image myself but do something like
I am not really sure if I understand your last point. I am indeed doing
which in my case binds the whole filesystem that is in /foo into the container, which is also where all of the files I want to use inside the container live. Of course, when the container was built by biocontainers this directory was not created. As far as I understand that is why I need to add Thanks |
That's correct.
Okay, looks like I misunderstood. Let me try again. You have a directory tree at If that is the case, and the path
Then you can run it with If See also #96, which is a rather embarrassingly old. |
Exactly. I guess for interactive work rebuilding a derived container would be an option, but right now the advantage over making a copy via However, one of my main use-cases for charliecloud is running nextflow pipelines, which usually submit jobs with predefined containers (i.e. pulling from quay.io/biocontainers). In this case the centralization of the containers is an advantage, often I am more interested in actually running an analysis in a broadly reproducible environment.
Did I get that correctly? |
There’s a couple reasons why we consider read-only containers (i.e., no
I'm a bit nervous about sharing storage directory between users in general, because it’s not tested and we have known ways that it breaks. See #1701.
This should work. I’ll also add that after not hearing much about it for a while, five people yesterday had requests that could be addressed by fixing #96. So that will be a higher priority. |
No traffic on this issue in a few weeks and based on the discussion I don’t think we’ll take any action; closing. Please LMK if that’s an error. |
That said, see PR #1793. |
Hi,
the recent change in how ch-run behaves when running with a set
CH_IMAGE_STORAGE
is somewhat confusing.As of version 0.33 (I think) ch-run no longer runs images when given a path that points to $CH_IMAGE_STORAGE/img/name, and complains that images should be run by name. Images that are run by name cannot be combined with
-w
, meaning that it is not possible to bind something to root via-b
. This breaks e.g. my hpc workflows where I bind a storage path to root and I would prefer to not bind it into /mnt/ to avoid having to deal with fixing all paths. I basically run pull with $CH_IMAGE_STORAGE set, then I unset this to run the container from the path to be able to use-w
with-b
and after I am finished with the container I set $CH_IMAGE_STORAGE again. This seems much less straight forward than the previous usage where I could just run from $CH_IMAGE_STORAGE with the path.This also breaks the nextflow integration with charliecloud (see nextflow-io/nextflow/issues/4463).
Is there a way to restore the old usage, or allow running from storage by path, or enable running by name with -w? Maybe there is also a way to start containers with ch-run that I am missing that would work around this issue?
Thanks
Niklas
The text was updated successfully, but these errors were encountered: