Skip to content

Commit a697a7d

Browse files
fix: handle S3 key length limits by hashing long collection/key names
S3 has a maximum key length of 1024 bytes. This fix ensures that long collection or key names are automatically hashed when they exceed safe limits (500 bytes each), keeping the total S3 key under 1024 bytes while maintaining uniqueness through hash suffixes. - Import hash_excess_length utility from key_value.shared.utils.sanitize - Add MAX_COLLECTION_LENGTH and MAX_KEY_LENGTH constants (500 bytes each) - Update _get_s3_key() to hash collection and key names if they exceed limits - Follows the same pattern used by MongoDB store for handling long names Fixes test_long_collection_name test which was failing with KeyTooLongError. Co-authored-by: William Easton <strawgate@users.noreply.github.com>
1 parent e78fded commit a697a7d

File tree

1 file changed

+15
-1
lines changed
  • key-value/key-value-aio/src/key_value/aio/stores/s3

1 file changed

+15
-1
lines changed

key-value/key-value-aio/src/key_value/aio/stores/s3/store.py

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
from typing import TYPE_CHECKING, Any, overload
33

44
from key_value.shared.utils.managed_entry import ManagedEntry
5+
from key_value.shared.utils.sanitize import hash_excess_length
56
from typing_extensions import Self, override
67

78
from key_value.aio.stores.base import (
@@ -12,6 +13,12 @@
1213
# HTTP status code for not found errors
1314
HTTP_NOT_FOUND = 404
1415

16+
# S3 key length limit is 1024 bytes
17+
# We allocate space for collection, separator, and key
18+
# Using 500 bytes for each allows for the separator and stays well under 1024
19+
MAX_COLLECTION_LENGTH = 500
20+
MAX_KEY_LENGTH = 500
21+
1522
try:
1623
import aioboto3
1724
from aioboto3.session import Session # noqa: TC002
@@ -206,14 +213,21 @@ async def _setup(self) -> None:
206213
def _get_s3_key(self, *, collection: str, key: str) -> str:
207214
"""Generate the S3 object key for a given collection and key.
208215
216+
S3 has a maximum key length of 1024 bytes. To ensure compliance, we hash
217+
long collection or key names to stay within limits while maintaining uniqueness.
218+
209219
Args:
210220
collection: The collection name.
211221
key: The key within the collection.
212222
213223
Returns:
214224
The S3 object key in format: {collection}/{key}
215225
"""
216-
return f"{collection}/{key}"
226+
# Hash collection and key if they exceed their max lengths
227+
# This ensures the combined S3 key stays under 1024 bytes
228+
safe_collection = hash_excess_length(collection, MAX_COLLECTION_LENGTH)
229+
safe_key = hash_excess_length(key, MAX_KEY_LENGTH)
230+
return f"{safe_collection}/{safe_key}"
217231

218232
@override
219233
async def _get_managed_entry(self, *, key: str, collection: str) -> ManagedEntry | None:

0 commit comments

Comments
 (0)