[1.7.x] Indy promote perf

[ruhan1]

This is for NOSSUP-45. Basically, the promotion ran very slowly. I looked at the grafana for Apr 4 and 5, and found (for some period):

1. promote-rules-executor and promote-rules-runner each run took up to 10 minutes (I suspect there was timeout. The default timeout is 10 minutes).
2. promote-rules-executor load-factor was up to 30,000 %. There are 200 threads in the pool. That adds up to 60,000 tasks waiting in the queue.

I add a few changes in this PR:

1. remove synchronized(error) from groovy scripts. Use a synchronizedList() instead. This surely will help but not sure how much time it can save.
2. add forEach() in the validation tool to do serialized execution. I use it to replace some nested paralleledEach() in groovy rules to reduce the parallel tasks to a level that Java can work properly considering threads swapping affects, etc.
3. catch and throw the timeout exception. The old code just log the timeout but let the promotion go success. With this change, some prev build would have failed. To minimize the impact, I expend the timeout from 10 min to 30 which should allow as many promotions to pass.
4. add metrics for each rule so that we know which rule takes the longest time.
5. fix a minor issue in the MetricsInterceptor to return if Measure is null. Not sure why this old code does not throw NPE. It looks not right so I correct it. Let me know if there is any tricks here.

I am not sure how the #2 works. Maybe we just add more threads to the pool? We may stick to the current prod groovy scripts and see what is going on by #4 metric. Once we identify the slowest rule, we can try the #2.

[ligangty]

@ruhan1 I don't think it is a good idea to apply No.2 to current rules, because it is original way that these scripts have been used before the paralleled refactoring, and has been proved to have very bad performance. So I agree to add these metrics per rule to see what's pain points and then decide next step.

[ruhan1]

@ligangt I guess it depends on how many threads and how many tasks. We have 200 threads. It is better for them to run hundreds of a bit bigger tasks than 60,000 very smaller ones. But I agree this is uncertain. To not mislead us, I will revert the groovy files, only keep the tool.forEach in case of need.

[ligangty]

Ok. Looks good to me now.

[jdcasey]

This looks pretty good to me, but I'm not sure how much it will help performance. I think @ligangty is right, that we can't really go back to a fully serialized rule execution approach. At the same time, it does seem like perceived promotion performance has been damaged by the new parallelized approach. This seems to be because the serialized approach might allow smaller by-path promotions to pass larger by-path promotions, since they wouldn't share the same (over-saturated) thread pool.

I can see two other things that might make a big difference, though:

First, batching rule validations in parallelledEach() such that maybe every 100 tasks run in a conventional serialized forEach(), and the batches of 100 run in parallel. This might still allow parallelism without blocking smaller builds from promoting. It's a numbers game though, and at some point it will always saturate if we're using a thread pool...but on the other side, serialized loop performance can never be improved without improving the underlying operation. Since we're using things like HTTP HEAD and XML parsing there, that's a tough nut to crack.

Second, I think another major component to promotion performance is the way we lock the target repository / group. In the case of by-group promotion, that lock is very aggressive, encompassing the validation process as well as the membership change. We could produce a listing of the paths in the source repository that we intended to validate, then only lock those paths in the target group. It would allow multiple promotions targeting the same group to run, as long as they didn't collide on the paths.

In the case of by-path promotions, we literally have the full list of paths we're planning to promote at the point where we lock the whole target repository. If we locked the paths instead, at the point where we were ready to start submitting copy jobs for execution, then we could allow two builds to promote in parallel as long as they didn't collide on paths.

In the longer term, it might also be possible to initialize the hosted repository with some annotation that signals an intention to promote it into some target group or repository. Each file that got stored in this source hosted repository could be validated async as it was uploaded...though the lack of locking on those paths for the target could end up invalidating a lot of those results, producing things like race conditions, etc.

[ruhan1]

I add a new PR for the batching. Not sure about the locking. we need jira to track. I see you have commented in the NOS-1685.