Output extra files no longer collected

[abretaud]

Describe the bug
Since my last update on the 23.0 branch, the extra files from tool output are no longer saved after job completion. Jobs are running in non-pulsar mode, on a standard slurm cluster.

I suspect it comes from #16541 : I tried to downgrade to the commit before this PR, and the problem is gone

Galaxy Version and/or server at which you observed the bug
Galaxy Version: 23.0
Commit: ea2be33

To Reproduce
Steps to reproduce the behavior:

1. Launch a tool like jbrowse or velveth
2. The output extra files are not collected

[mvdbeek]

Do you have a copy of galaxy on your cluster and does it need to be updated ? We've been running for quite some time in production with local and remote runners and it's working fine.

[abretaud]

Nope, no copy of the code, it's on a shared volume. I'm a bit surprised by this bug, I more cleanly reverted #16541 and it really fixes it. But I don't see why we would have a problem and not others...
I'll see if I can make more debugging later

[mvdbeek]

Did you have any time to look into this @abretaud ? What are the settings for outputs_from_working_directory and object_store_store_by ?

[abretaud]

Haven't had the time yet unfortunately, it coul take a few days...
outputs_to_working_directory is set to true, and object_store_store_by is not defined

[mvdbeek]

Can you check if datasets are stored by id or uuid (you can see that for newly created datasets if you look at the file path).

[abretaud]

Oops, sorry for the delay!
They are stored with id currently (galaxy/datasets/006/549/dataset_6549309.dat)

[mvdbeek]

This should be fixed in #17067. I'd encourage you to also consider migrating to an object store that uses the newer store_by="uuid" mechanism, which

• should lead to better job throughput (since we don't need to generate a primary ids up front for every output)
• is much better tested at this point and the default when setting up a new instance
• and would allow you to also use metadata_strategy: extended, where dynamic outputs can be pushed to the object store by the metadata script as part of the job (instead of the job handler doing the data transfer).

You can do this migration by setting up a distributed object store as in https://training.galaxyproject.org/training-material/topics/admin/tutorials/object-store/tutorial.html#distributed-object-store and give the old object_store a weight of 0

[abretaud]

Cool thanks so much Marius!
I'll apply it shortly, and look into the object store change too

[abretaud]

I confirm that the fix works 👍