Separated data PVC

[almahmoud]

Separates the volumes that should be mounted in all jobs (cvmfs, scripts, cache, etc...) vs the pvcs that should be mounted per job (input data + job directory).

Friends with galaxyproject/galaxy-helm#314

Example volume mounts per job:

## Unique to job (generated from `k8s_data_volume_mount`)
[{'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/galaxy/server/database/objects/4/1/5/dataset_415a97d5-2b27-4b10-9626-4eb60372e46d.dat', 'subPath': 'objects/4/1/5/dataset_415a97d5-2b27-4b10-9626-4eb60372e46d.dat'},
 {'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/galaxy/server/database/jobs_directory/000/19/outputs', 'subPath': 'jobs_directory/000/19/outputs'},
 {'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/galaxy/server/database/jobs_directory/000/19', 'subPath': 'jobs_directory/000/19'},

## Common among jobs (from `k8s_persistent_volume_claims`)
 {'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/galaxy/server/database/cache', 'subPath': 'cache'},
 {'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/galaxy/server/database/config', 'subPath': 'config'},
 {'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/galaxy/server/database/deps', 'subPath': 'deps'},
 {'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/galaxy/server/database/object_store_cache', 'subPath': 'object_store_cache'},
 {'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/galaxy/server/database/tmp', 'subPath': 'tmp'},
 {'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/galaxy/server/database/tool-data', 'subPath': 'tool-data'},
 {'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/galaxy/server/database/tools', 'subPath': 'tools'},
 {'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/galaxy/server/database/tool_search_index', 'subPath': 'tool_search_index'},
 {'name': 'k8sfiles-gxy-rls-galaxy-cvmfs-gxy-data-pvc', 'mountPath': '/cvmfs/data.galaxyproject.org'},
 {'name': 'k8sfiles-gxy-rls-galaxy-pvc', 'mountPath': '/cvmfs/cloud.galaxyproject.org', 'subPath': 'cvmfsclone'}]

Todo

• Make sure config is backwards compatible (conceptually should be since we can still use k8s_persistent_volume_claims to mount the entire shared FS in all jobs as before, but want to test that it handles empty/null values for k8s_data_volume_claim properly)

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these contributions under Galaxy's current license.
• I agree to allow the Galaxy committers to license these and all my past contributions to the core galaxy codebase under the MIT license. If this condition is an issue, uncheck and just let us know why with an e-mail to galaxy-committers@lists.galaxyproject.org.

[jmchilton]

This won't work with extra files paths and metadata files I assume.

This logic shouldn't be happening at this layer.

[nuwang]

@jmchilton Where do you think it should be happening though? I'm not sure this logic can be moved elsewhere, because mounting PVCs is inherently tied to the k8s runner? The extra files path is a fair point, but the overall way this works is that the extra metadata can be mounted into the job and made available, out-of-band if necessary.

[nuwang]

Suggested change

	self._init_monitor_thread()
	self._init_worker_threads()

[nuwang]

Parsing could be moved to separate function and shared between k8s_persistent_volume_claims and k8s_data_volume_claim and k8s_working_volume_claim

[nuwang]

This is working well, and since it's backward compatible, should pose no issues even if we have to make any changes in future.