Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate Nighthawk's CI to machines with more memory #820

Closed
mum4k opened this issue Mar 17, 2022 · 1 comment · Fixed by #844
Closed

Migrate Nighthawk's CI to machines with more memory #820

mum4k opened this issue Mar 17, 2022 · 1 comment · Fixed by #844
Assignees
@mum4k
Labels
enhancement New feature or request

Comments

@mum4k
Copy link
Collaborator

mum4k commented Mar 17, 2022

The current allocation only gives us 16GB of memory. test_gcc takes a very long time to finish and the integration tests can no longer execute.

We should migrate to machines with more memory.
@mum4k mum4k added the enhancement New feature or request label Mar 17, 2022
@mum4k mum4k self-assigned this Mar 17, 2022
This was referenced Mar 17, 2022
Merged
Merged
Merged
@jiajunye jiajunye mentioned this issue Mar 22, 2022
Merged
mum4k added a commit that referenced this issue Mar 31, 2022
@mum4k
Defining our first Azure Pipeline. (#821)
a247493 
This is first of a few PRs that will migrate Nighthawk's CI to Azure Pipelines. This is what Envoy uses currently and this is where we can get more resources which will enable the integration tests to pass again.

This defines a single step `build` that will execute in our `nighthawk-presubmit` pipeline.

Works on #820.

Signed-off-by: Jakub Sobon <mumak@google.com>
@mum4k mum4k mentioned this issue Apr 7, 2022
Merged
mum4k added a commit that referenced this issue Apr 8, 2022
@mum4k
Adding the test step into AZP (#825)
5f72e3d 
Also enabling local bazel cache to carry state between the CI steps and printing out the full bazel command line for debugging.

Works on #820.

Signed-off-by: Jakub Sobon <mumak@google.com>
@mum4k mum4k mentioned this issue Apr 12, 2022
Merged
mum4k added a commit that referenced this issue Apr 14, 2022
@mum4k
Adding all our CI jobs to AZP. (#827)
684a8af 
- Decreasing integration test coverage threshold from `75%` to `74.2%`. The integration test coverage wasn't executing on CircleCI due to memory limitations. Our coverage dropped despite my attempts to execute this locally before each submission. While I did verify that the tests passed locally, I did not notice that the script is unable to verify the coverage when compiling on RBE. Also removing a TODO linked to #564 which doesn't appear to be relevant anymore.
- Modified `ci/do_ci.sh` so that clang_tidy uses all available CPUs when executing on AZP. Included a logging statement that prints this out in the output for debugging.
- Setting a larger bazel timeout for the `python_test` that runs all our integration tests to avoid hitting the timeout on test execution.

Works on #820.

Signed-off-by: Jakub Sobon <mumak@google.com>
@mum4k mum4k mentioned this issue Apr 19, 2022
Merged
mum4k added a commit that referenced this issue Apr 19, 2022
@mum4k
Migrate our docker workflow to AZP. (#836)
de0d937 
Temporarily forking the `docker_push.sh` and `benchmark_push.sh` scripts to AZP specific versions as a simplification. The CircleCI versions will be cleared when we turn down the CircleCI pipeline. Adding new step `do_docker_azp` to `ci/do_ci.sh` to execute the forked scripts.

Modified `ci/run_envoy_docker.sh` to forward variables required by the docker scripts. Since we regularly copy this script from Envoy, the custom lines are marked as `# unique` and the maintenance steps in `MAINTAINERS.md` are updated to indicate the presence of these unique lines.

Works on #820.

Signed-off-by: Jakub Sobon <mumak@google.com>
@mum4k mum4k mentioned this issue Apr 21, 2022
Merged
mum4k added a commit that referenced this issue Apr 22, 2022
@mum4k
Deprecate CircleCI pipeline and configuration. (#837)
49453bd 
Works on #820.

- Deleting CircleCI config.
- Removing instructions related to CircleCI from MAINTAINERS.md.
- Removing reference to CircleCi in benchmarks/README.md.
- Folding previously forked docker scripts.
- Removing repokitteh retest support for CircleCI. Going to send separate PR that will add this functionality for AZP.
- Removing utility that detects CircleCi in integration tests.
- Removing references to CircleCI from ci/do_ci.sh.

Signed-off-by: Jakub Sobon <mumak@google.com>
@mum4k mum4k mentioned this issue Apr 22, 2022
Merged
mum4k added a commit that referenced this issue Apr 25, 2022
@mum4k
Fix the docker stage by renaming s/docker_azp/docker. (#840)
0f4774c 
The argument was renamed in #837, but I forgot to rename it in the pipeline configuration.

Works on #820.

Signed-off-by: Jakub Sobon <mumak@google.com>
@mum4k mum4k mentioned this issue Apr 26, 2022
Merged
@mum4k
Copy link
Collaborator Author

mum4k commented Apr 28, 2022

Nighthawk's CI now executes on AZP.

@mum4k mum4k closed this as completed Apr 28, 2022
mum4k added a commit that referenced this issue Apr 29, 2022
@mum4k
repokitteh: implement /retest on AZP (#844)
c1022d2 
Fixes #820.

Signed-off-by: Jakub Sobon <mumak@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mum4k mum4k

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

repokitteh: implement /retest on AZP mum4k/nighthawk
1 participant
@mum4k