Skip to content

Add AWS S3 support for py-key-value #178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 14 commits into from
Closed

Add AWS S3 support for py-key-value #178

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
da86ea2
Add AWS S3 support for py-key-value
github-actions[bot] Oct 29, 2025
1b1c3d7
fix: exclude S3 from sync library and improve test configuration
github-actions[bot] Oct 29, 2025
a8a093f
fix: address CodeRabbit review feedback for S3 store
github-actions[bot] Oct 29, 2025
401d947
fix: handle S3 key length limits by hashing long collection/key names
github-actions[bot] Oct 30, 2025
82d3f6a
refactor: update S3Store to use new serialization adapter pattern
github-actions[bot] Oct 31, 2025
d9c1d7b
feat: add length_is_bytes parameter to sanitize_string and hash_exces…
github-actions[bot] Nov 2, 2025
731afa1
fix: address review feedback for S3 store
github-actions[bot] Nov 2, 2025
d528933
fix: address CodeRabbit review feedback for S3 store
github-actions[bot] Nov 2, 2025
e0f4529
simplify setup/teardown in s3
strawgate Nov 2, 2025
065b551
refactor: align S3 store with sanitization strategy pattern
github-actions[bot] Nov 7, 2025
06228f0
Merge branch 'main' into claude/issue-161-20251029-0140
strawgate Nov 7, 2025
dbab902
Longer wait for s3 spinup
strawgate Nov 7, 2025
e5125c2
Small test changes
strawgate Nov 7, 2025
f8b3a70
Merge branch 'main' into claude/issue-161-20251029-0140
strawgate Nov 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ This monorepo contains two libraries:

## Why use this library?

- **Multiple backends**: DynamoDB, Elasticsearch, Memcached, MongoDB, Redis,
- **Multiple backends**: DynamoDB, S3, Elasticsearch, Memcached, MongoDB, Redis,
RocksDB, Valkey, and In-memory, Disk, etc
- **TTL support**: Automatic expiration handling across all store types
- **Type-safe**: Full type hints with Protocol-based interfaces
Expand Down Expand Up @@ -131,6 +131,7 @@ pip install py-key-value-aio
pip install py-key-value-aio[memory]
pip install py-key-value-aio[disk]
pip install py-key-value-aio[dynamodb]
pip install py-key-value-aio[s3]
pip install py-key-value-aio[elasticsearch]
# or: redis, mongodb, memcached, valkey, vault, registry, rocksdb, see below for all options
```
Comment on lines +134 to 137
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Install extra for S3 — suggest adding a minimal usage snippet.
Optional: add a short S3Store example right below to help users confirm setup quickly.

Example snippet to consider:

from key_value.aio.stores.s3 import S3Store
async with S3Store(bucket_name="my-bucket") as store:
    await store.put(key="k", value={"v": 1}, collection="c", ttl=60)
    assert await store.get(key="k", collection="c")
🤖 Prompt for AI Agents 
In README.md around lines 133 to 136, add a minimal S3 usage snippet immediately
below the pip install line for py-key-value-aio[s3]; include a short async
example that imports S3Store, opens it with a bucket_name (and mention
credentials are picked up from the environment), uses await store.put(...) to
write a key/value with collection and ttl, and uses await store.get(...) to
confirm the value; keep the snippet compact (show async context manager usage
and simple assertion) so users can quickly verify their S3 setup.

Expand Down Expand Up @@ -191,7 +192,7 @@ categories:
- **Local stores**: In-memory and disk-based storage (Memory, Disk, RocksDB, etc.)
- **Secret stores**: Secure OS-level storage for sensitive data (Keyring, Vault)
- **Distributed stores**: Network-based storage for multi-node apps (Redis,
DynamoDB, MongoDB, etc.)
DynamoDB, S3, MongoDB, etc.)

Each store has a **stability rating** indicating likelihood of
backwards-incompatible changes. Stable stores (Redis, Valkey, Disk, Keyring)
Expand Down
3 changes: 2 additions & 1 deletion key-value/key-value-aio/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ vault = ["hvac>=2.3.0", "types-hvac>=2.3.0"]
memcached = ["aiomcache>=0.8.0"]
elasticsearch = ["elasticsearch>=8.0.0", "aiohttp>=3.12"]
dynamodb = ["aioboto3>=13.3.0", "types-aiobotocore-dynamodb>=2.16.0"]
s3 = ["aioboto3>=13.3.0", "types-aiobotocore-s3>=2.16.0"]
keyring = ["keyring>=25.6.0"]
keyring-linux = ["keyring>=25.6.0", "dbus-python>=1.4.0"]
pydantic = ["pydantic>=2.11.9"]
Expand All @@ -67,7 +68,7 @@ env_files = [".env"]

[dependency-groups]
dev = [
"py-key-value-aio[memory,disk,redis,elasticsearch,memcached,mongodb,vault,dynamodb,rocksdb]",
"py-key-value-aio[memory,disk,redis,elasticsearch,memcached,mongodb,vault,dynamodb,s3,rocksdb]",
"py-key-value-aio[valkey]; platform_system != 'Windows'",
"py-key-value-aio[keyring]",
"py-key-value-aio[pydantic]",
Expand Down
13 changes: 13 additions & 0 deletions key-value/key-value-aio/src/key_value/aio/stores/s3/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
"""AWS S3-based key-value store."""

from key_value.aio.stores.s3.store import (
S3CollectionSanitizationStrategy,
S3KeySanitizationStrategy,
S3Store,
)

__all__ = [
"S3CollectionSanitizationStrategy",
"S3KeySanitizationStrategy",
"S3Store",
]
Loading
Loading