Reject PP/SR HTTP requests if cross shard semaphore has been exhausted

[michael-redpanda]

Fixes: #14116

Cross shard calls in SR/PP utilize an SMP resource group to limit the number of cross shard calls that
are made. However, if the semaphore has been fully consumed, the call simply hangs. Customers have
experienced issues where, under heavy load, SR appears to hang and not response. Now, instead, once the
semaphore has been completely exhausted, new HTTP requests that attempt cross shard calls will be
rejected with a 500 error.

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.3.x
• v23.2.x
• v23.1.x

Release Notes

Improvements

• SR/PP will now reply with a 500 error if an internal service semaphore has been completely exhausted

[michael-redpanda]

I don't believe we backport things if there's a new cluster config (right?). If we do want this backported, I can drop the commits that add this and submit the change in a separate PR.

[graphcareful]

Do we need the SMP resource group anymore?

[michael-redpanda]

Do we need the SMP resource group anymore?

I'd like @BenPope 's take on this question, I think it's a valid one.

[BenPope]

Do we need the SMP resource group anymore?

I'd like @BenPope 's take on this question, I think it's a valid one.

I'm not sure what the suggested alternative is?

[BenPope]

This doesn't seem to limit the number of in-flight requests; so does it fix #14116?

I.e., would it be better to limit the number of in-flight requests by applying backpressure rather than rejecting valid requests?

[michael-redpanda]

This doesn't seem to limit the number of in-flight requests; so does it fix #14116?

The issue that we had in relation to the incident in #14116 was that new requests were just hanging out in the semaphores wait list but we had no idea that that was happening. By checking whether the semaphore has been exhausted before initiating the cross-shard call, we can reject the new request.

I.e., would it be better to limit the number of in-flight requests by applying backpressure rather than rejecting valid requests?

I'm not sure how the behavior to the client would be any different. How would you limit the number of in-flight requests without rejecting new ones?

[BenPope]

This doesn't seem to limit the number of in-flight requests; so does it fix #14116?

The issue that we had in relation to the incident in #14116 was that new requests were just hanging out in the semaphores wait list but we had no idea that that was happening. By checking whether the semaphore has been exhausted before initiating the cross-shard call, we can reject the new request.

It's not just new requests, though, it's requests that may be half-processed.

I.e., would it be better to limit the number of in-flight requests by applying backpressure rather than rejecting valid requests?

I'm not sure how the behavior to the client would be any different. How would you limit the number of in-flight requests without rejecting new ones?

By not processing them until semaphore can be obtained (or timed-out), Unfortunately, we don't yet stream the body, so it's not ideal for something like a produce right now.

[michael-redpanda]

Force push 5a4d0ed:

• Refactored solution to add new adjustable sempahore that prevents too many inflight requests, per shard

[graphcareful]

Do we need the SMP resource group anymore?

I'd like @BenPope 's take on this question, I think it's a valid one.

I'm not sure what the suggested alternative is?

Within the smp resource config there is an option:

max_nonlocal_requests: The maximum number of non-local requests that execute on a shard concurrently

Correct me if i'm wrong but doesn't this configurable option solve the same issue that Mike is attempting to solve with the adjustable semaphore?

EDIT: I guess i'll try to answer my own question, maybe the logical difference here is the smp group enforces the max is number of futures vs what this PR is attempting to solve which is the max number of "requests"

[BenPope]

Within the smp resource config there is an option:

max_nonlocal_requests: The maximum number of non-local requests that execute on a shard concurrently

Correct me if i'm wrong but doesn't this configurable option solve the same issue that Mike is attempting to solve with the adjustable semaphore?

EDIT: I guess i'll try to answer my own question, maybe the logical difference here is the smp group enforces the max is number of futures vs what this PR is attempting to solve which is the max number of "requests"

I suspect that the problem is that cross-core requests are being made several times without "unwinding" first; a request can get stuck half-way waiting on the semaphore. Timing it out isn't ideal as only some of the work may have been applied.

Limiting requests much lower than the nonlocal limit should resolve this problem.

[BenPope]

Looks good.

[BenPope]

_inflight_sem could be destructed at this point (I don't see another mechanism that unsubscribes the watch, or otherwise sequences the destruction order.

[BenPope]

_inflight_sem could be destructed at this point (I don't see another mechanism that unsubscribes the watch, or otherwise sequences the destruction order.

[michael-redpanda]

Isn't destruction sequence order dictated by the class? _inflight_config_binding is constructed after _inflight_sem and therefor should be destructed before the semaphore?

[rockwotj]

BTW I find relying on destructor order is incredibly sneaky and usually deserves a comment somewhere

[BenPope]

Isn't destruction sequence order dictated by the class? _inflight_config_binding is constructed after _inflight_sem and therefor should be destructed before the semaphore?

You're correct.

[dotnwat]

lgtm

[BenPope]

Why is this considered "not a bug fix" and not backported?