Run artifact transforms loaded from the configuration cache for external and file dependencies in parallel

[adammurdoch]

When using vintage builds and querying the contents of a dependency resolution view, then any transforms which still have to run will be executed in parallel and then the results collected.

However, when using builds loaded from the configuration cache, these transforms are run sequentially. This has a number of implications, such as potential performance issues. There should be as few differences as possible between the execution of transforms for vintage builds and for configuration cache builds.

Expected Behavior

Multiple transforms run in parallel regardless of whether configuration cache is enabled or not.

Current Behavior

Transforms are sometimes run sequentially. The underlying cause for this behaviour is that there is currently quite a lot of duplication of logic in the configuration cache, where some behaviours have been (temporarily) reimplemented because the original data structures are not very easy to recreate. There are some downsides to this:

• Performance may not be as good when the transforms actually need to do some work (rather than just doing an up-to-date check via the VFS). Usually this is not the case and all of these transforms are up-to-date. Performance may well be better in these cases.
• Only the first transform failure reported, rather than collecting all of the failures and reporting on them at the end of the query.
• There is a lot of complexity in the configuration cache implementation that shouldn't be there.
• The duplicated logic will inevitably introduce a new set of edge cases and bugs, that only apply when loading from the configuration cache.

We should address this by refactoring out some reusable pieces that can be more easily recreated by the configuration cache, rather than trying to improve the duplicated logic.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. If you're interested in how we try to keep the backlog in a healthy state, please read our blog post on how we refine our backlog. If you feel this is something you could contribute, please have a look at our Contributor Guide. Thank you for your contribution.