fix(pipelinetriggers): process mptv2 pipelines in parallel

[robert-blackman]

Hey

Several months ago we came across this issue whereby a large number of MPTv2 pipelines seemed to have a significant (negative) performance impact on pipeline triggers. Initial investigation indicated this was due to how the pipelines were planned. We originally developed a poc plan cache, this worked quite well however the scope of the fix grew as we discovered more code paths that called on the hydration method.

This fix takes a simpler approach by streaming the pipelines in parallel to orca for planning. This is implemented through the use of a ForkJoinPool using a custom parallelism argument, Runtime.getRuntime().availableProcessors() by default. We decided to use a ForkJoinPool over a cached thread pool due to its lower memory footprint and work stealing virtues.

The parallelism argument can be configured via the following spring config within the echo-local profile:

pipelineCache:
  parallelism: 128 # We are using this in production

Would be great to get some feedback on this. Thanks!

Fixes this issue

[spinnakerbot]

The following commits need their title changed:

• 144e6b7: add initial configuration

• c3fab7f: fix spel expression

• 8dc5dd5: change: camel case config

• 095a494: change: runtime ref

• e5b2c6d: should still get runtime

Please format your commit title into the form:

<type>(<scope>): <subject>, e.g. fix(kubernetes): address NPE in status check

This allows us to easily generate changelogs & determine semantic version numbers when cutting releases. You can read more about commit conventions here.

[robert-blackman]

Hey @dreynaud, just tagging you here for visibility 👍

[dreynaud]

Might be good to either log or throw the exception here, no? Not sure what the implication of returning null is here, are we sure the consumers are ok with it?

[robert-blackman]

@dreynaud Good point, perhaps it's best to log the exception for visibility, then return an empty list which should match the existing behaviour?

[dreynaud]

Hey @dreynaud, just tagging you here for visibility 👍

Looks like a good change @robert-blackman 👍 Just see my other comment around the exception handling, otherwise LGTM

Also tagging @robzienert 👀

[mattgogerly]

@Mergifyio update

[link108]

@Mergifyio update

[mergify]

update

❌ Pull request can't be updated with latest base branch changes

Mergify needs the author permission to update the base branch of the pull request.
monster-next needs to authorize modification on its head branch.
err-code: 09B30

[link108]

Hey @robert-blackman, do you mind allowing changes from maintainers on this PR? Eg https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork