find substitute for docker volume mounting

[mwartell]

Armory allows many resources to be mounted from a local filesystem into the docker container at the beginning of an armory run. These include

cwd dataset_dir local_git_dir output_dir saved_model_dir tmp_dir

The mechanism allows, for example, making gigabyte datasets available to the container without rebuilding or adding GB to the image size. While this works nicely for a local filesystem which caches the datasets, it is not exportable to
remote container execution in Kubernetes (or similar).

The armory launcher phase is capable of pulling remote resources to the local disk prior to running a container. That means we already have code capable of demand loading datasets, models, etc. onto the local machine.

In some ways, this issue is calling for delayed binding of the remote resources. That is, rather than pull a dataset to the local disk and mount that into a container, direct access from the container to the resource would free Armory from requiring a pre-loaded disk cache.

We'll need to distinguish remote read-only resources (weights, repos) and write-capable remote resources like output_dir.

[ExistentialRecursionist]

I agree that the current launcher does not implement bind mounts in the most appropriate way, however, kubernetes et. al. do allow for remote bind mounts; be it from a ceph cluster, separate container, etc. Implementing these changes via a compose file(e.g. docker-compose.yml) is more in line with best practices and makes pivoting to a kints cluster a bit simpler; as the yaml for a pod is quite similar(see here for an example armory compose file).