make torchstore api concurrent safe

[casteryh]

This pull request introduces a robust concurrency control mechanism for RDMA operations in TorchStore by adding an asynchronous executor and refactoring the codebase to ensure safe, sequential execution of RDMA requests. The changes also update client and storage interfaces to support this executor, improve error handling, and clarify environment variable defaults.

Concurrency and RDMA execution improvements:

• Added OnceCell and SequentialExecutor classes in torchstore/_async_utils.py to provide async initialization and sequential execution for RDMA tasks, ensuring only one RDMA operation runs at a time.
• Refactored LocalClient and storage classes to accept and use SequentialExecutor for RDMA operations, including passing the executor through all relevant layers and enforcing concurrency limits via a semaphore and method decorator. [1] [2] [3] [4] [5] [6] [7] [8] [9]

API and initialization changes:

• Updated client cache in torchstore/api.py to use OnceCell[LocalClient] for lazy, single initialization and refactored related functions to support this pattern. [1] [2] [3]

Transport buffer interface updates:

• Modified TransportBuffer and RDMATransportBuffer to require an executor for RDMA read/write operations, enforcing sequential execution and updating method signatures. [1] [2] [3] [4]

Configuration and minor fixes:

• Changed the default for the TORCHSTORE_RDMA_ENABLED environment variable to "1", enabling RDMA by default.
• Minor typo and docstring corrections in storage volume and buffer classes.

These changes collectively ensure safe and efficient RDMA usage in TorchStore, improve initialization and resource management, and lay groundwork for future scalability.

[codecov-commenter]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️