Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize signal sending to processes with message_queue_data=off_heap enabled #5020

Conversation

kjellwinblad
Copy link
Contributor

Erlang guarantees that signals (i.e., message signals and non-message signals) sent from a single process to another process are ordered in send order. However, there are no ordering guarantees for signals sent from different processes to a particular process. Therefore, several processes can send signals in parallel to a specific process without synchronizing with each other. However, such signal sending was previously always serialized as the senders had to acquire the lock for the outer signal queue of the receiving process. This commit makes it possible for several processes to send signals to a process with the message_queue_data=off_heap setting(1) activated in parallel and without interfering with each other. This parallel signal sending optimization yields much better scalability for signal sending than what was previously possible(2).

(1) Information about how to enable the message_queue_data=off_heap setting can be found in the documentation of the functions erlang:process_flag/2 and erlang:spawn_opt/4.

(2) http://winsh.me/bench/erlang_sig_q/sigq_bench_result.html

Implementation

The parallel signal sending optimization works only on processes with the message_queue_data=off_heap setting enabled. For processes with the message_queue_data=off_heap setting enabled, the new optimization is activated and deactivated on demand based on heuristics to give a small overhead when the optimization is unnecessary. The optimization is activated when the contention on the lock for the outer message queue is high. It is deactivated when the number of enqueued messages per fetch operation (that fetch messages from the outer message queue to the inner) is low.

When the optimization is active, the outer message queue has an array of signal buffers where sending processes enqueue signals. When the receiving process needs to fetch messages from the outer message queue, the contents of the non-empty buffers are append to the outer message queue. Each process is assigned a particular slot in the buffer array (the process ID is used to hash to a particular slot). That way, the system can preserve the send order between messages coming from the same process.

@ranjanified
Copy link
Contributor

Nice.. And Brilliant step 👍 ... Hope it doesn't affect anything else...

Erlang guarantees that signals (i.e., message signals and non-message
signals) sent from a single process to another process are ordered in
send order. However, there are no ordering guarantees for signals sent
from different processes to a particular process. Therefore, several
processes can send signals in parallel to a specific process without
synchronizing with each other. However, such signal sending was
previously always serialized as the senders had to acquire the lock
for the outer signal queue of the receiving process. This commit makes
it possible for several processes to send signals to a process with
the message_queue_data=off_heap setting* activated in parallel and
without interfering with each other. This parallel signal sending
optimization yields much better scalability for signal sending than
what was previously possible**.

* Information about how to enable the message_queue_data=off_heap
  setting can be found in the documentation of the function
  erlang:process_flag/2.

** http://winsh.me/bench/erlang_sig_q/sigq_bench_result.html

Implementation
--------------

The parallel message sending optimization works only on processes with
the message_queue_data=off_heap setting enabled. For processes with
the message_queue_data=off_heap setting enabled, the new optimization
is activated and deactivated on demand based on heuristics to give a
small overhead when the optimization is unnecessary. The optimization
is activated when the contention on the lock for the outer message
queue is high. It is deactivated when the number of enqueued messages
per fetch operation (that fetch messages from the outer message queue
to the inner) is low.

When the optimization is active, the outer message queue has an array
of signal buffers where sending processes enqueue signals. When the
receiving process needs to fetch messages from the outer message
queue, the contents of the non-empty buffers are append to the outer
message queue. Each process is assigned a particular slot in the
buffer array (the process ID is used to hash to a particular
slot). That way, the system can preserve the send order between
messages coming from the same process.
@kjellwinblad kjellwinblad force-pushed the kjell/erts/parallel_signal_queue/g/OTP-16982 branch from 2476f90 to 658f864 Compare September 6, 2021 14:58
@kjellwinblad kjellwinblad merged commit 5b1bd6d into erlang:master Sep 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team:VM Assigned to OTP team VM
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants