Add forkCount option to parallelize build and tests#2278
Add forkCount option to parallelize build and tests#2278sijie merged 3 commits intoapache:masterfrom
Conversation
|
@eolivelli Looks like we've halved the total execution time required to run most CI tests on Github Actions! (it's even better locally, on my MBP its reduced by 2/3rds) |
eolivelli
left a comment
eolivelli
left a comment
There was a problem hiding this comment.
Sorry
I missed the notifications.
LGTM.
I have restarted GitHub tests
eolivelli
left a comment
eolivelli
left a comment
There was a problem hiding this comment.
LGTM
I have restarted failed jobs.
I sincerely hope that this change won't add flakyness to GitHub Actions jobs.
Please @Ghatage keep an eye these days on GitHub Action jobs,
we can add -DforkCount.variable=1 (or 2) in github workflows configurations
|
Will do. We can reduce it as you suggest if it causes too much ruckus. |
|
I have restarted bookie tests and integration tests. |
|
Fair enough @eolivelli. I have moved it to 1. |
|
I have restarted the failing Bookie Tests |
### Motivation Current code forks a new JVM per module. (bookkeeper-server, bookkeeper-proto etc) This means one fork per module for build and testing, no parallelism within the module where majority of the time goes. We need parallelism within a module during the test execution so we can have the builds complete faster and have the artifacts shipped out quicker. ### Changes We use the maven surefire plugin but don't define the `forkCount` and hence set it to default of 1. This means it executes each module with one thread. This change sets `forkCount` to 5, enabling parallelism in testing and drastically reducing total turnaround time. (by about 2/3rds!) *Total build+test time without this change* ``` [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:06 h [INFO] Finished at: 2020-03-05T02:01:29-08:00 [INFO] ------------------------------------------------------------------------ ``` *Total build+test time with this change* ``` [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 18:23 min [INFO] Finished at: 2020-03-05T02:38:22-08:00 [INFO] ------------------------------------------------------------------------ ``` ### Things to watch Added parallelism may cause some flappers but with much trial and error I have come to the number `5`. The flappers are usually only from conflict in obtaining the same port number. If needed, we can increase the retryCount, but as of now I consistently don't see any flappers at a `forkCount` of 5 Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Sijie Guo <None> This closes apache#2278 from Ghatage/parallelizeBuildAndTests
3 participants
Motivation
Current code forks a new JVM per module. (bookkeeper-server, bookkeeper-proto etc)
This means one fork per module for build and testing, no parallelism within the module where majority of the time goes.
We need parallelism within a module during the test execution so we can have the builds complete faster and have the artifacts shipped out quicker.
Changes
We use the maven surefire plugin but don't define the
forkCountand hence set it to default of 1.This means it executes each module with one thread.
This change sets
forkCountto 5, enabling parallelism in testing and drastically reducing total turnaround time. (by about 2/3rds!)Total build+test time without this change
Total build+test time with this change
Things to watch
Added parallelism may cause some flappers but with much trial and error I have come to the number
5. The flappers are usually only from conflict in obtaining the same port number.If needed, we can increase the retryCount, but as of now I consistently don't see any flappers at a
forkCountof 5