Backport rescale bug

[arteam]

Backport of #1033 to the 3.1.* branch.

[ryantenney]

I'd like to look into this further before we backport it.

[arteam]

Thanks, ping me when you have more thoughts on the matter.

[ryantenney]

While it does seem to correct an issue, I have concerns about it blocking all threads on the rescale once every hour.

[ho3rexqj]

In both the original and updated implementations, while a thread is performing the rescale operation (within the write lock) all concurrent updates block until the rescale operation is complete. There are behavioral differences there, however nothing fundamental changes with respect to the impact on other concurrent updates while the rescale operation is ongoing - could you clarify your concern there?

Note that the race condition the patch fixes allowed updates to sometimes be added to the reservoir outside of the rescale window (that is, without a preceding rescale operation). The fix merely enforces that requirement; updates outside of the rescale window are forced to wait for the rescale operation to complete before updating the reservoir's state. There are other implementations that would accomplish that - the write lock barrier is more than is strictly required there:

One approach would be to add a second atomic to rescale (or similar) to guard the call to lockForRescale:

if (rescaleGuard.getAndSet(next) != next) {
    lockForRescale();
    try {
        if (nextScaleTime.compareAndSet(next, now + RESCALE_THRESHOLD)) {
            // ...
        }
    } finally {
        unlockForRescale();
    }
} else {
    // a spin lock is used in this illustration; ideally something else would be used in practice
    while (nextScaleTime.get() == next)
        ; // do nothing
}

Alternatively, rather than enforcing the above requirement, it could be relaxed and the reservoir could be refactored accordingly. For example, a cache could be added to the reservoir to collect updates added in update outside of the rescale window (value/timestamp pairs) - those would then needed to be processed as part of the rescale operation, and possibly as part of getSnapshot (although that could likely be omitted).

Note, however, both of the above would add significant complexity to the reservoir. I'm not certain it's at all worthwhile given that the window during which multiple concurrent calls to update will block on lockForRescale should either be small (two concurrent calls to update) or involve three or more competing threads (one thread executing getSnapshot and two or more executing update concurrently).