DM-30767: support component exports (by exporting comparent composites)

[TallJimbo]

Checklist

• ran Jenkins
• added a release note for user-visible changes to doc/changes

[timj]

Looks like a nice improvement. Is this needed because components requested in a pipeline were being skipped before in pipe_base when creating the execution butler but now are included? I'm now worrying whether I need to fix the disassembled composites export problem -- easy to check by defining a pipeline that can use the outputs of the pipelines_check repo and create an execution butler from the output collection.

[TallJimbo]

Is this needed because components requested in a pipeline were being skipped before in pipe_base when creating the execution butler but now are included?

I believe that's the case. But component-dataset inputs are rare in our concrete pipelines, and it's even more rare to start processing with one, so it probably would have been along time before we'd have noticed.

[timj]

It's not about specifying components as input, it's about composite disassembly in general since we are creating the execution butler by exporting then importing datasets rather than using butler.datastore.transfer_from(). The former can't handle disassembled composites regardless whereas the latter can because it deals with the datastore records directly rather than going through that exporter class that doesn't know about disassembly. I think the demonstration is fairly simple: run pipelines_check and then run a new pipeline using those outputs -- the execution butler will refuse to be built I think.

[TallJimbo]

I think the demonstration is fairly simple: run pipelines_check and then run a new pipeline using those outputs -- the execution butler will refuse to be built I think.

Understood. But since it sounds like execution butler + disassembly is already broken, I'm not going to try to fix it on this ticket, and I'll let you create a new one if you want to tackle it. I really hope we can replace execution butler with something else before we ever consider disassembly in production, but of course we'll still want import/export to be able to handle disassembly anyway.