Golang SDK: Support re-acquiring semaphore lock after application restart when using durable sessions

[francoposa]

User Story

As a user, I would like to use durable (long or infinte TTL, no heartbeat, no healthcheck) sessions to allow applications to re-acquire a previous semaphore lock

Why is this Needed?

We would like to use semaphores as a rate limiter for long-running workflow tasks which persist through retries, goroutine panics, and worker process restarts, etc. as in the temporal.io (successor of Cadence) framework. We do not want the lock to be released when the task dies, as the task monitors an external process. Releasing the lock would allow another sempahore slot to be claimed by another task resulting in too many in-process instances of the external task.

This is possible now using the bare check-and-set operations, but bare CAS does not have the quality-of-life features of the Semaphore interface available in the Golang Consul SDK.

In order to re-acquire the lock, the KV put operation used under the hood of the Sempahore must have the correct session ID, KV prefix, and the latest LockIndex value, as described here: https://www.consul.io/docs/dynamic-app-config/sessions#k-v-integration.
However the ability to set LockIndex on a Sempahore Acquire operation is not exposed in the SDK. As proof that this would work, the debugger can be used to suspend the session and set the LockIndex on the KV put request associated with the lock acquire operation before the KV put request was sent, and the lock will be re-acquired.

Description

Type: Feature

Current Behavior

An application cannot re-acquire its semaphore slot after restart using the Semaphore interface in the Golang SDK, even if the correct session ID is provided.

Expected Behavior

An application can re-acquire its semaphore slot after restart using the Semaphore interface in the Golang SDK, if the correct session ID and latest LockIndex is provided.

Steps to reproduce

using Golang SDK Semaphore and Session interfaces:

Start an application, create sempahore with 1 slot, acquire semaphore lock with a durable session created withSessionNoChecks, no, note session ID. End application without releasing lock.

Start an application and attempt to acquire the semaphore slot using the previous session ID. Acquire call will block forever or until session is deleted manually through UI.

Definition of Done

Update existing Semaphore Acquire interface in Golang SDK or create new interface like Reacquire in order to enable the desired behavior

(Apologies for odd format, I originally posted this mistakenly in the consul/consul repository which had quite a different new issue format)