Fix discarded datasets when importing history from file sources using tasks

[davelopez]

Required for #14839

When importing a history archive from a file source using the tasks system with /api/histories/from_store_async all datasets appear as discarded in the new history regardless of their real state.

This adds test coverage to expose the bug. Currently investigating the fix... 🔍

Update

This PR tries to fix some issues found in this context:

• The tag_handler instance was missing during import so it failed when items had tags.
• The model_store_format was not passed to the import model store and was always the default tgz.
• The default value for discarded_data in the ImportOptions was always FORCE which apparently treated every dataset as discarded independently of their real state.
• We don't have set_external_metadata_tool in the context of tasks so, for now, we try to set the metadata directly withing the task... See 3404db4.

How to test the changes?

• I've included appropriate automated tests.

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[davelopez]

So, in this context (importing histories from file sources using tasks), it looks like we don't have a set_external_metadata_tool defined. Is it ok to skip this regenerate_imported_metadata_if_needed step then? or how should we proceed here? I'm not familiar with the implications.

galaxy/lib/galaxy/model/store/__init__.py

Line 626 in d2163c9

self.app.datatypes_registry.set_external_metadata_tool.regenerate_imported_metadata_if_needed(

[mvdbeek]

Hah, you've reached the limitations of Celery. Setting metadata is another celery task, and you're not supposed to call tasks from tasks. If at all possible I would create a chord, where the first task, import_model_store returns a list of datasets to set metadata on, and the chord just chains them together. Here's a chord like that that chains fetch_data, set_metadata and finish_job: https://github.com/mvdbeek/galaxy/blob/d2850975238d1369c93b9625d9bbfd5ff709689b/lib/galaxy/tools/execute.py#L182-L196

[bgruening]

Thanks a lot for the test and the fixes.