Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backoff not kicking in, 100% usage on all CPUs #103

Open
zevv opened this issue Feb 2, 2020 · 2 comments
Open

Backoff not kicking in, 100% usage on all CPUs #103

zevv opened this issue Feb 2, 2020 · 2 comments

Comments

@zevv
Copy link

@zevv zevv commented Feb 2, 2020

Running the following snippet makes my laptop hot, all CPUs spin at 100%

import weave, os

init(Weave)
os.sleep(10 * 1000)
exit(Weave)

Quick profiling indicates that CPU is spent in tryRecv / recvProxy

@zevv zevv changed the title Backoff not kicking, 100% usage on all CPUs Backoff not kicking in, 100% usage on all CPUs Feb 2, 2020
@zevv

This comment has been minimized.

Copy link
Author

@zevv zevv commented Feb 2, 2020

"Don't do that" is the proper answer, right? :)

Thanks for your talk today, and it was a pleasure meeting you!

@mratsim

This comment has been minimized.

Copy link
Owner

@mratsim mratsim commented Feb 8, 2020

Here is a way to do that:

import weave, os

init(Weave)
syncRoot(Weave)  # Empty the task queue
echo "Going to sleep"
os.sleep(10 * 1000)
echo "Waking up"
exit(Weave)

In your case what happens is:

  • You create threads
  • The main thread goes to sleep
  • Child threads send steal requests
  • They get stuck in the main thread which is sleeping
  • Child threads can't sleep unless they are notified that there is no work (but no notification since thread is sleeping)
  • The main thread wakes up
  • Now it can handle the incoming requests

In short, similar to async/io don't block without handing back control to the scheduler otherwise mayhem ensues.

Unfortunately this is not solvable with Weave message-passing design. It would work if thefts were shared memory based, sleeping threads would not prevent active threads from looking in their deques but this tradeoff also prevents many optimizations.

Alternatively I could change the backoff mechanism to have regular "maintenance" wakeups (for example exponential backoff) bu it's less efficient on a single computer (though it's the only way to have backoff in a distributed setting https://github.com/mratsim/weave/blob/v0.3.0/weave/channels/event_notifiers_and_backoff.md#distributed-backoff-and-contention-resolution-strategies

@mratsim mratsim removed the wontfix 🌋 label Feb 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.