Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manually compact GC before fork #2093

Merged
merged 1 commit into from Dec 18, 2019
Merged

Manually compact GC before fork #2093

merged 1 commit into from Dec 18, 2019

Conversation

@schneems
Copy link
Contributor

schneems commented Dec 18, 2019

Rub 2.7.0 introduced GC.compact which allows manual compaction of Ruby slots. A good time to do this is before forking so that memory fragmentation can be reduced.

One issue with memory fragmentation when forking is that while a page in memory might have only one free slot, as that slot is written to after fork, the entire page is written so the benefit of CoW optimizations are greatly reduced. Manually compacting GC reduces the number of pages with empty slots.

This PR manually compacts memory right before Puma forks and after other before_fork hooks are called.

Description

Please describe your pull request. Thank you for contributing! You're the best.

Your checklist for this pull request

  • I have reviewed the guidelines for contributing to this repository.
  • I have added an entry to History.md if this PR fixes a bug or adds a feature. If it doesn't need an entry to HISTORY.md, I have added [changelog skip] the pull request title.
  • I have added appropriate tests if this PR fixes a bug or adds a feature.
  • My pull request is 100 lines added/removed or less so that it can be easily reviewed.
  • If this PR doesn't need tests (docs change), I added [ci skip] to the title of the PR.
  • If this closes any issues, I have added "Closes #issue" to the PR description or my commit messages.
  • I have updated the documentation accordingly.
  • All new and existing tests passed, including Rubocop.
@schneems schneems force-pushed the schneems/compact-gc branch from a62f551 to 2b4eb74 Dec 18, 2019
@MSP-Greg
Copy link
Contributor

MSP-Greg commented Dec 18, 2019

@liaden
Copy link

liaden commented Dec 18, 2019

Would it be worthwhile to add a GC.start to increase the resulting compactness? Edit: Nevermind, it already invokes GC https://github.com/ruby/ruby/blob/master/gc.c#L8438

At work, we often will use eye to monitor child processes for memory bloat and have eye kill them. It would be beneficial to have this only done on the first fork instead of every time we fork.

History.md Outdated Show resolved Hide resolved
@schneems schneems force-pushed the schneems/compact-gc branch 2 times, most recently from 9c46fea to 020e4d8 Dec 18, 2019
@schneems
Copy link
Contributor Author

schneems commented Dec 18, 2019

I'm getting failures on GitHub actions for 2.7.0 and 2.3.x and 2.3.8 with just

Run actions/setup-ruby@v1
  with:
    ruby-version: 2.7.0-preview3
##[error]Version 2.7.0-preview3 not found
##[error]Node run failed with exit code 1

Any idea how to debug or what needs to be changed @MSP-Greg ?

@MSP-Greg
Copy link
Contributor

MSP-Greg commented Dec 18, 2019

@schneems

I'm getting failures on GitHub actions for 2.7.0 and 2.3.x and 2.3.8

At present, standard GitHub actions just has Ruby 2.4.x, 2.5.x, & 2.6.x installed. Anything else needs code using a container, installing and using rvm, or a special action step.

@dentarg & PR #2087 adds Ruby 2.3 in a container. The Ruby/Psych code I linked to adds ruby-head using rvm. I can add 2.3 & ruby-head to Windows, but that won't run this code as fork is not available...

Rub 2.7.0 introduced `GC.compact` which allows manual compaction of Ruby slots. A good time to do this is before forking so that memory fragmentation can be reduced.

- https://www.ruby-lang.org/en/news/2019/12/17/ruby-2-7-0-rc1-released/
- https://www.youtube.com/watch?v=1F3gXYhQsAY

One issue with memory fragmentation when forking is that while a page in memory might have only one free slot, as that slot is written to after fork, the entire page is written so the benefit of CoW optimizations are greatly reduced. Manually compacting GC reduces the number of pages with empty slots.

This PR manually compacts memory right before Puma forks and after other `before_fork` hooks are called.
@schneems schneems force-pushed the schneems/compact-gc branch from 020e4d8 to abd3f35 Dec 18, 2019
@MSP-Greg
Copy link
Contributor

MSP-Greg commented Dec 18, 2019

All the Travis 2.7.0 tests passed, and all the Actions tests that are not broken passed.

@schneems
Copy link
Contributor Author

schneems commented Dec 18, 2019

All the Travis 2.7.0 tests passed, and all the Actions tests that are not broken passed.

Seems good!

@schneems schneems merged commit b646fc5 into master Dec 18, 2019
12 of 13 checks passed
12 of 13 checks passed
OS: ubuntu-16.04 Ruby: 2.3.x OS: ubuntu-16.04 Ruby: 2.3.x
Details
build
Details
OS: ubuntu-18.04 Ruby: 2.4.x
Details
OS: ubuntu-18.04 Ruby: 2.5.x
Details
OS: ubuntu-18.04 Ruby: 2.6.x
Details
OS: macos Ruby: 2.4.x
Details
OS: macos Ruby: 2.5.x
Details
OS: macos Ruby: 2.6.x
Details
OS: windows-latest Ruby: 2.4.x
Details
OS: windows-latest Ruby: 2.5.x
Details
OS: windows-latest Ruby: 2.6.x
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.