Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ThreadPool concurrency refactoring #2220

Merged
merged 2 commits into from Apr 14, 2020
Merged

Conversation

@wjordan
Copy link
Contributor

wjordan commented Apr 10, 2020

Description

This PR refactors some tricky concurrency parts of TestThreadPool, in order to make the unit tests more stable (all tests pass on jruby), faster (no more 1-second sleeps), and a bit more readable. Many of the tests now use a subclass of ThreadPool that overrides #<< to wait on the existing @not_full condition variable, to ensure work begins executing on a worker thread before continuing the test.

There are also some small changes to the ThreadPool class itself:

  • Add #with_mutex to make it easier to chain multiple @mutex.synchronize-wrapped methods together, and to extend existing mutex-wrapped methods in unit tests.
  • Wait for initially-spawned threads to enter the wait loop before returning from #initialize. This fixes a subtle edge-case concurrency bug mostly affecting unit-test reliability, demonstrated by the following simple test that fails on master:
    def test_waiting_on_startup
    pool = new_pool(1, 2)
    assert_equal 1, pool.waiting
    end
  • Fix a concurrency bug in #trim where a trim could still be requested even when enough work to utilize all threads has been queued but not yet picked up by workers (e.g., waiting > 0 but waiting - todo.size == 0). (I believe this was the cause of some jruby test flakiness in some trim unit tests.)
  • Refactor the main wait-loop in spawn_thread to make it a bit simpler and easier to read and reason about.

Your checklist for this pull request

  • I have reviewed the guidelines for contributing to this repository.
  • I have added an entry to History.md if this PR fixes a bug or adds a feature. If it doesn't need an entry to HISTORY.md, I have added [changelog skip] the pull request title.
  • I have added appropriate tests if this PR fixes a bug or adds a feature.
  • My pull request is 100 lines added/removed or less so that it can be easily reviewed.
  • If this PR doesn't need tests (docs change), I added [ci skip] to the title of the PR.
  • If this closes any issues, I have added "Closes #issue" to the PR description or my commit messages.
  • I have updated the documentation accordingly.
  • All new and existing tests passed, including Rubocop.
- Wait for threads to enter waiting loop on ThreadPool startup
- Simplify #spawn_thread inner threadpool loop
- Refactor TestThreadPool to make tests faster and more stable
@wjordan wjordan force-pushed the wjordan:thread_pool_refactor branch from 8d61027 to e1daf1b Apr 10, 2020
@@ -36,6 +36,7 @@
* Simplify `Runner#start_control` URL parsing (#2111)
* Removed the IOBuffer extension and replaced with Ruby (#1980)
* Update `Rack::Handler::Puma.run` to use `**options` (#2189)
* ThreadPool concurrency refactoring (#2220)

This comment has been minimized.

Copy link
@nateberkopec

nateberkopec Apr 12, 2020

Member

I think the trim bugfix could also be mentioned

@nateberkopec
Copy link
Member

nateberkopec commented Apr 12, 2020

I'll have to find some time to review this proper, but the test changes are gorgeous 😍

@@ -99,20 +102,13 @@ def spawn_thread
while true
work = nil

continue = true

mutex.synchronize do

This comment has been minimized.

Copy link
@nateberkopec

nateberkopec Apr 14, 2020

Member

Clever change and much more readable

end
Thread.pass until finish
pool.signal
sleep

This comment has been minimized.

Copy link
@nateberkopec

nateberkopec Apr 14, 2020

Member

typo or are you triggering a context switch?

@nateberkopec nateberkopec merged commit b16d8cb into puma:master Apr 14, 2020
dmarcotte added a commit to looker/puma that referenced this pull request Apr 23, 2020
Well, "fix" might overstate it, but we were able to get them passing
by backporting test skips that are currently in Puma master upstream.

With these skips, not only does tag `4.3.3.` all pass, but so does
our custom branch, so we can be pretty confident we haven't broken
anything vs. stock Puma.

Some notes on test runs:
- Invoke with `bundle exec rake test --trace`
- Ctrl-C'ing the tests usually left a zombified version running in the
background, which would *cause failures on subsequent runs*.  Ensure
after killing the a test run that no test processes remain in `jps -lm`
- `TestThreadPool` tests are flaky in this release, though this was a
known issue, and has been recently fixed here: puma#2220
JohnPhillips31416 added a commit to looker/puma that referenced this pull request May 29, 2020
Well, "fix" might overstate it, but we were able to get them passing
by backporting test skips that are currently in Puma master upstream.

With these skips, not only does tag `4.3.3.` all pass, but so does
our custom branch, so we can be pretty confident we haven't broken
anything vs. stock Puma.

Some notes on test runs:
- Invoke with `bundle exec rake test --trace`
- Ctrl-C'ing the tests usually left a zombified version running in the
background, which would *cause failures on subsequent runs*.  Ensure
after killing the a test run that no test processes remain in `jps -lm`
- `TestThreadPool` tests are flaky in this release, though this was a
known issue, and has been recently fixed here: puma#2220
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.