Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ruby 3.1 tests in CircleCI should not require resource_class_to_use: xlarge #1791

Closed
ivoanjo opened this issue Nov 30, 2021 · 1 comment · Fixed by #1947
Closed

Ruby 3.1 tests in CircleCI should not require resource_class_to_use: xlarge #1791

ivoanjo opened this issue Nov 30, 2021 · 1 comment · Fixed by #1947
Assignees
@ivoanjo

Comments

@ivoanjo
Copy link
Member

ivoanjo commented Nov 30, 2021 •
edited
Loading

I've observed that installing gems in CircleCI on Ruby 3.1 was failing with something like:

Installing thin 1.8.1 with native extensions
Fetching qless 0.12.0
Installing qless 0.12.0
Killed

Exited with code exit status 1
CircleCI received exit code 1

This Killed is due to the container running out of memory while installing the grpc gem.

This happens because our container is running inside a bigger machine (with 36 cores), and the grpc gem spans a compilation process that uses 36*2 CPUs because Etc.nprocessors => 36 on this machine.

This issue can be reproduced by running the CircleCI job with ssh enabled (Rerun -> Rerun Job with SSH), ssh'ing to the machine and running gem install grpc.

For now, I've bumped the resource class for 3.1 tests to xlarge, but potential next steps are:

  1. Reporting the issue upstream, asking them to perhaps allow the number of cpus to be configured / overridden
  2. Add sysconfcpus when running our CI to lower the number of CPUs the Ruby process reports during gem installation
@ivoanjo ivoanjo self-assigned this Nov 30, 2021
@ivoanjo ivoanjo mentioned this issue Dec 1, 2021
Closed
@ivoanjo
Copy link
Member Author

ivoanjo commented Dec 2, 2021

A way to override this is being added in grpc/grpc#28250 :)

I'll wait until it gets merged/released, and then do a PR removing

dd-trace-rb/.circleci/config.yml

Line 491 in e9751f6

resource_class_to_use: xlarge # Temporary, see https://github.com/DataDog/dd-trace-rb/issues/1791

ivoanjo added a commit that referenced this issue Feb 2, 2022
@ivoanjo
Remove CI resource class workaround due to grpc issue
42c9373 
While being installed, the grpc gem compiles a bunch of native code.

In #1791 I documented an issue we were running into in Circle CI where
the CI worker was running out of memory because grpc was spawning
too many parallel processes for compilation.

I brought the issue up with upstream in
grpc/grpc#28244 and in
grpc/grpc#28250 a new flag was added to
enable us to limit the number of processes used.

So now we can remove our workaround of bumping the resource class
put in place in #1790 and instead use the `GRPC_RUBY_BUILD_PROCS`
flag to tell grpc how many processes to use.

Fixes #1791
ivoanjo added a commit that referenced this issue Mar 25, 2022
@ivoanjo
Remove CI resource class workaround due to grpc issue
4b19c29 
While being installed, the grpc gem compiles a bunch of native code.

In #1791 I documented an issue we were running into in Circle CI where
the CI worker was running out of memory because grpc was spawning
too many parallel processes for compilation.

I brought the issue up with upstream in
grpc/grpc#28244 and in
grpc/grpc#28250 a new flag was added to
enable us to limit the number of processes used.

So now we can remove our workaround of bumping the resource class
put in place in #1790 and instead use the `GRPC_RUBY_BUILD_PROCS`
flag to tell grpc how many processes to use.

Fixes #1791
@ivoanjo ivoanjo mentioned this issue Mar 25, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ivoanjo ivoanjo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove CI resource class workaround for Ruby 3.1 due to grpc issue DataDog/dd-trace-rb
1 participant
@ivoanjo