conda_auto_install and Collections cause install/remove/install loop

[blankenberg]

If conda_auto_install is set to true, and a dependency isn't installed but is available in conda, and you run a collection in batch mode through a tool, it attempts to install, remove 'failed' installation, and reinstall for each individual job simultaneously(-ish).

Not only is this quite slow and resource intensive (thousands of items in a collection...), but it leads to failures, and may cause instances where an unversioned dependency ends up being installed and used for a particular job (did not closely examine logs, just killed all jobs and datasets and rebooted galaxy, so UV is not confirmed).

[blankenberg]

In a similar vein, it would be helpful if there was some sort of small timed local cache check for conda install. Minutes would be fine in most cases.

For example, when a specific version of a package isn't available, but an unversioned one is, Galaxy will attempt to conda install the versioned one at each execution, so it does the search and says it can't find it and will default to the unversioned one. This check seems to take between 5 seconds and a minute. The response time is probably exacerbated by having it run simultaneously or in quick succession many times and it greatly slows down the scheduling of jobs to a cluster.

One example is the fastq-join tool:
conda create -y --override-channels --channel iuc --channel bioconda --channel conda-forge --channel defaults --name __ea-utils@1.1.2-806 ea-utils=1.1.2-806
which doesn't exist at that version.

[mvdbeek]

Yeah, that's why I have been going round saying don't use conda_auto_install if you can avoid it. What would be great in many ways is if we had a single queue type thread for such things (and maybe another one for TS installs). The queue could even be smart and cache / bundle requests.

The problem I see is that we'll only want a single unique queue, and we need to be able to communicate with it from all web and job handlers. I guess mules could work for this type of thing ?

[dannon]

@mvdbeek We could declare a single queue using the existing queue_workers and have any available worker consume out of it.

[mvdbeek]

Like those that pick up jobs from the database or the communication via kombu? I think the kombu communication doesn't work in all circumstances (uwsgi web handlers communicating with job handlers, but maybe that's better now ?) ?

[dannon]

Right, using that infrastructure. Processes need to be named correctly for the dynamically created specific-to-process queues, but I think that's something we've worked on a lot lately. That said, we're talking about an even simpler queue, here -- a single named queue that many things work from. That should 100% work right now, if we wanted to set one up.