Rate limiting techniques in io_service

[ct-clmsn]

The libfabric parcelport uses a strict control mechanism to manage thread suspension when the maximum amount of sends have been achieved through the compile-time variable(s) HPX_PARCELPORT_LIBFABRIC_THROTTLE_*. Would a token bucket rate limiting regime managing the number of sends (and thread suspension) provide performance improvements? Specifically in uses cases where input parameters could be provided to the runtime when the program is executed, or compiler tools could 'find' input parameters, or input parameters could be managed by autotuners (APEX)?

[biddisco]

The HPX_PARCELPORT_LIBFABRIC_THROTTLE is intended to limit the number of sends that are 'in existence' at a given moment. Consider the following

for (int i=0; i<1000000000000000000000; i++) {
  auto thing = hpx::async(remote_locality, action, blah, blah, blah);
}

and yes, I just held my finger down on the zero key to make a silly number.

As it stands, hpx will try to honour this code snippet and use all the memory in the machine, creating futures and putting args into send parcel buffers etc.

The idea behind the throttling is simply to block the thread that is sending, when the number of sends in the queue is beyond what the fabric can handle, to poll the network instead and when the queues have drained, to resume the thread and allow it to continue. @sithhell and I disagree about the way this should be handled and he removes my throttling code each time he sees it and I quietly put it back again when he is not looking (at least on my branch).

Internally, libfabric will perform some flow control of its own

The control messages that the GNI provider uses internally (GNI SMSG messages) use a circular queue mechanism with credits to protect from overrun of the queue. Also, for the RX hardware
CQs - which can get overrun if you are sending many small messages to a single endpoint - there's a plan B fallback to go through all of the mailboxes (the circular queues) and look for new messages until the RX CQ is no longer in overflow state. You can think of the RX CQ as an optimization for notifying us which mailbox to check for new messages. This procedure has been used for years in the Cray MPI (since the first Cray XE's came out), so its gotten lots of testing over the years.

Small messages injected via fi_tsend and friends use the SMSG protocol while larger messages go
through a combination of SMSG control message and RDMA reads.

Now, this is only true of the GNI provider, but for now, this is the only one we support, so we should revisit this issue and your question is the right way to go about it, however

Would a token bucket rate limiting regime managing the number of sends (and thread suspension) provide performance improvements?

I do not know. Could you ask the question again, but using different words - because I am not sure that I fully understand what it is you are suggesting and I'd rather not attempt to answer it until I do (insert smiley here).

[ct-clmsn]

@biddisco I probably should have held off on submitting this ticket until having a better understanding of the overall implementation. Thank you for the clarification!

I'm in the midst of sorting out how all of the pieces in the parcelport implementation work together - will provide more feedback soon. For what it's worth, given my current understanding of the implementation, what is there makes sense.

At the risk of burning credibility linking to wikipedia...the article on token bucket is about the most concise summary I can come up with at the moment.

[ct-clmsn]

@biddisco think I've zeroed in on a better description of this ticket. A token bucket scheduler would provide a different mechanism (potentially better) to control the way that boost::asio::io_service are accessed by hpx::iostreams::server::output_stream.

From what I've pieced together, hpx::iostreams::server::output_stream calls the io_service_pool::get_io_service method using the default call (index value of -1).

The default call to io_service_pool::get_io_service uses a round robin scheduling/assignment technique to pick a boost::asio::io_service (OS-threads?) that are used in hpx::iostreams::server::output_stream::write methods to call boost::asio::io_service::post.

[hkaiser]

@ct-clmsn is the way we use the io_service for the iostreams limiting in any way? I know the round robin algorithm used is trivial, perhaps even too simple, we have however not seen any bottlenecks on this end of the story yet.

[ct-clmsn]

@hkaiser after finding parcelport's commentary about throttling the creation of senders and receivers and some tickets referencing MPI's performance it seemed like an opportunity to propose an alternative technique that would provide tuning options and ask questions about the implementation. thank you both for the clarifications. i'll close out the ticket.

[biddisco]

Having read through the wikipedia article, I'd say that the answer is "no, not really". the token bucket simply provides a bucket from which we take a token when we want to send and when the bucket is empty, we have to wait. What we've implemented is actually, the opposite - we have a bucket that is empty, each time a send goes out, we put a token in the bucket and when it is full we wait. Once sends have been transmitted, the tokens are removed.

The result it identical as far as I can see unless I missed some crucial detail.