Collection jobs record dataset collection elements instead of HDAs

[wm75]

Edit by @mvdbeek

Nothing is wrong here, we've improved the provenance tracking, but the UI and job rerun routes don't know how to deal with dataset collection elements, these are minor tweaks, not severe bugs.

Describe the bug
When you're running any workflow that is mapping over collections, Galaxy records as input for any collection element not the dataset ID, but a malformed version of the element identifier. With that the possibility to rerun jobs is completely lost!!

Galaxy Version and/or server at which you observed the bug
Galaxy Version: 23.0rc1 on .org and .eu

To Reproduce
Steps to reproduce the behavior:

1. Run any workflow that involves a mapping over a collection
2. Open the output collection and inspect any of the elements

Expected behavior
Each element should have its input dataset from the input collection recorded, but in reality what gets recorded is what's shown below

Screenshots

[mvdbeek]

Nah, we just don't know how to display non-dataset elements in the UI. I don't think that's why you can't rerun, but if you can share a history that'd be helpful.

[wm75]

What do you mean. It is supposed to be a dataset element - that's kind of the point.

[mvdbeek]

SRRblabla is a dataset collection element, this has been that way ever since we've shown input datasets in the job info page. But if you can share a history that'd be great.

[wm75]

Sure, https://usegalaxy.eu/u/sars-cov2-bot/h/gx-surveillance-nt1775844g

[wm75]

and here's what a collection element as input looked before the update:

[mvdbeek]

which element in that history should I look at ?

[wm75]

any from any simple list. They are all broken.

[mvdbeek]

None of it is broken:

the UI just doesn't know how to display DCEs. We'd previously track HDAs, which was not quite correct.

[wm75]

That's better of course, puh. So just an annoying bug then.

[wm75]

Why is this only happening for WFs then, and not for regular job runs on collections?
Why mix things?

[mvdbeek]

It also happens (and has been happening) for normal jobs, just less frequently.

[wm75]

It also happens (and has been happening) for normal jobs, just less frequently.

I don't follow I'm afraid why is this about frequencies? Shouldn't it be either/or?

[mvdbeek]

We used to track the HDA associated associated with a DCE. You've probably noticed that if you re-run a job from a list you'd re-run on the hda, because the element identifier of the element you consume is gone. This change fixes that (among other things).

We've always tracked DCEs when the input parameter was a collection, and the actual input was a nested collection (e.g. a list input on a list:list HDCA). We have now removed the special casing for HDAs and pass through the element instead. So now you see this more often.

[wm75]

I see, but I thought I had tested earlier today that if you manually run a tool over a collection (not an individual job), the UI has no issue with showing the provenance correctly (that's actually why I thought it can't be the UI in the first place). On my phone now so hard to check again, but that's what I think I remember.

[mvdbeek]

I'll check that, but this should be all the same.

[wm75]

Any chance that this can still get fixed before Smorgasbord 3?
The public health track of the training at least will include several tutorials that involve running WFs that produce collections, and in the current state it will be next to impossible for participants to rerun any failed jobs inside these collections.

[mvdbeek]

Yeah, I'll pick this back up.

[mvdbeek]

With that the possibility to rerun jobs is completely lost!!

So I think I fixed the job info page now. Is rerunning these jobs really broken, or did it just seem that way because of the text where you'd expect a dataset ? The import of your example history takes forever, but checking on my instance that all seems to work correctly.

[wm75]

It is really broken on .eu. I can create a minimal example there for you I guess.

[mvdbeek]

That would be helpful, thanks!

[wm75]

Ok,

WF to reproduce the issue:
https://usegalaxy.eu/u/wolfgang-maier/w/minimal-collections-wf

history created with it:
https://usegalaxy.eu/u/wolfgang-maier/h/minimal-collections-wf-1

You should be able to observe the following:

• try to rerun any of the jobs in the 6: Add column or 10: Add column collections.
  These will claim that they've been generated with 14: Add column as input.
• try to hit Run Tool despite the nonsense, and the rerun job will actually use 14 as input (you can confirm it's really happening through the contents of the results).
• try to rerun anything in 14: Add column.
  This should work just fine because that collection was not produced through the WF, but via a manual tool run.

[mvdbeek]

Well that's unfortunate, the API returns the right values but I guess the frontend doesn't know about dces ...

[mvdbeek]

Should be fixed by #15744, I'll probably update .org later today