stm/dist_kv_stm: fix locking sequence for remove_all

[bharathv]

Currently remove_all takes a write lock for the entire duration that
prevents replicate_and_wait() to make progress because it needs a read
lock. Fix this by making remove_all best effort, so it can just grab
a read lock.

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.3.x
• v23.2.x

Release Notes

Bug Fixes

• Fixes lock starvation during transform offset commits.

[rockwotj]

Nice catch, thanks for fixing this

[vbotbuildovich]

new failures in https://buildkite.com/redpanda/redpanda/builds/46822#018e7c71-1ba7-4479-823c-5dd8807187f5:

"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_time_based_retention.cloud_storage_type=CloudStorageType.ABS"

[bharathv]

Failures unrelated:
#11269
#16561

[dotnwat]

i was under the impression that this wouldn't work because the underlying semaphore attempts to prevent starvation by forcing new acquirers to queue up behind other waiters?

[bharathv]

ya you're right I think (nice catch), a sequence of events where this could fail.

u = write_lock (current), queue = [write_lock, ...] all the units deposited by current are grabbed by the first waiter.

let me fix this.

[rockwotj]

Don't forget to update the cover letter

[vbotbuildovich]

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46837#018e7d47-6be0-47d0-8e4e-7628af06898a

[vbotbuildovich]

/backport v23.3.x

[vbotbuildovich]

Failed to create a backport PR to v23.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-17402-v23.3.x-426 remotes/upstream/v23.3.x
git cherry-pick -x 2c7b25cd04ae1605e006a807f756cb17d507e524

Workflow run logs.

[rockwotj]

This code doesn't exist in 23.3.x so we don't need a backport IIRC

[dotnwat]

lgtm