x/sync/semaphore: document ordering of Weighted.Acquire

[dsnet]

I was debugging a sluggish system and the problem came down to the ordering of how Acquire was satisfied.

The current behavior of Weighted.Acquire is that it serves acquisitions in order. This is a reasonable behavior, but it should be documented as it is significant performance implications. In particular, this behavior leads to head-of-line blocking where a large acquisition will prevent smaller acquisitions from making progress. The alternate behavior to satisfy any possible acquisition out-of-order, but that would lead to live-lock situations where a continuous stream of small acquisitions prevent a large acquisition from ever making progress.

My solution ended up splitting the semaphore into two parts:

• a semaphore for small requests
• a semaphore for large requests.

However, it was not obvious to me that I should do this since I had originally assumed that the semaphore allowed for out-of-order acquisitions.

[seankhliao]

cc @bcmills

[neilotoole]

I agree that the FIFO behavior of Weighted.Acquire should be documented. I have code that relies upon its FIFO-ness, and changing that behavior would break that code. I understand that Weighted.Acquire doesn't currently make that FIFO guarantee in its docs; I think it should.