🌱 [UNDER TESTING] Increase parallelism #1345
Conversation
azeemshaikh38
requested review from
inferno-chromium,
laurentsimon and
naveensrinivasan
as code owners
.github/workflows/main.yml
Outdated
| @@ -38,7 +38,7 @@ jobs: | |||
| - name: Run presubmit tests | |||
| run: | | |||
| go env -w GOFLAGS=-mod=mod | |||
| make -j 5 all | |||
| make -j 12 all | |||
There was a problem hiding this comment.
have you tried using "make -jnproc" to dynamically retrieve the number of processors?
There was a problem hiding this comment.
Thanks for the suggestion. nproc worked great!
|
error |
azeemshaikh38
force-pushed
the
azeems/parallel
branch
from
89a6401
to
b8d0a30
Compare
|
Integration tests success for |
| @@ -38,7 +38,7 @@ jobs: | |||
| - name: Run presubmit tests | |||
| run: | | |||
| go env -w GOFLAGS=-mod=mod | |||
| make -j 5 all | |||
| make -j $(nproc) all | |||
There was a problem hiding this comment.
Instead of make -j all, split them into individual jobs with the actions and they run parallel.
There was a problem hiding this comment.
IIUC, we need to maintain order in our build process. For example, update-dependencies step should always run before any other step while tree-status should be the last step. Similarly, some steps which generate files like proto and mocks should run before we build. Explicitly specifying this ordering in the Makefile is the best thing I could come up with. It's not ideal, but if folks have better ideas, please do let me know.
There was a problem hiding this comment.
split them into individual jobs with the actions and they run parallel.
+1, example from Kubernetes RelEng: https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/release-utils/release-utils-presubmits.yaml
There we usually prefer to run separate Prow jobs to split them up along the following boundaries:
- unit/integration tests
- e2e tests
- verifies (linting,
go mod tidydiffs, boilerplate)
There was a problem hiding this comment.
I like boundaries too :-)
|
Stale pull request message |
|
Stale pull request message |
|
Stale pull request message |
|
@azeemshaikh38 Can we close this? Now that we have some parallelism. |
Done, thanks! |
What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
What is the current behavior? (You can also link to an open issue here)
What is the new behavior (if this is a feature change)?
Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
Other information: