S3 blob layer and cache

S3 blob layer and cache

Upcoming. This page describes the s3 branch. It is not part of a release, and details may change.

Part of S3 implementation. The gateway keeps object data, and the runs and manifests of its namespace index, as immutable blobs. This page covers the interface to the store of blobs, how an object is made of blobs, and the local cache that sits in front of network storage.

The blob store

Two implementations of BlobStore serve a running node, one over the other:

Implementation	Module	Job
StorageBlobStore	application	Stores and reads blobs through the storage service of the node
CachingBlobStore	hedgehog-s3	Keeps local copies of the blobs of the store it wraps, see The cache

S3Gateway puts the cache over the network store before it opens anything else, so index runs and manifests go through the same cache as object data. The tests run the server over an in-memory blob store.

Network storage

StorageBlobStore stores through StorageService and answers the fingerprint of the stored file as blob id: a format byte and 32 random bytes in Base58Check. A blob id is thus a random capability, not a content hash. Its blobs implement only whole reads, so a range is efficient only through the cache.

Too few gridnodes, or storage that is disabled, become a BlobUnavailableException, which a client receives as 503 ServiceUnavailable with Retry-After: 5. An unknown fingerprint becomes a BlobNotFoundException.

Objects as blobs

The content of an object version is a list of parts, and each part is a list of slices. A slice names a blob, an offset into it and a length. Blobs are shared between versions, so the index counts the references to each, and a blob that nothing refers to any more is deleted after a grace period; see S3 namespace index.

Write	Blobs
PutObject	The body as one blob of at most 5 GiB (ObjectResource.MAX_SIZE); an empty body makes none
UploadPart	A blob of its own per part, at least 5 MiB except the last part, at most 10,000 parts
CompleteMultipartUpload	No new blob: the version refers to the slices of its parts
CopyObject, UploadPartCopy	No new blob: the copy refers to the slices of the source

• A refused write. A body whose index write is refused is deleted again.
• Copies. The source of a copy is read under the writer lock of the index, so its blobs cannot be deleted before the copy refers to them. No data is copied.
• Completion. The part numbers must ascend and the ETags must match those of the uploaded parts. The ETag of the completed object is the MD5 over the binary MD5s of its parts, followed by - and the number of parts.
• Reads. A read computes the slices that lie within the requested range, trimming the first and last, and opens the first blob before the response starts, so a refusal is still answered as an XML error.

The cache

CachingBlobStore keeps decoded blobs in local files within a byte budget. The content of these files is decrypted, so the cache directory is created rwx------, readable by the owner only. Each file is named by the SHA-256 of its blob id. A blob never changes, so a cached copy never goes stale.

Option	Default	Effect
--s3cachedir	<data>/s3/cache	Directory of the cache
--s3cachesize	1073741824 (1 GiB)	Byte budget of the cache
--s3cachehalflife	P1D	Time after which a read counts half as much toward keeping a blob cached

On write. store() tees the upload into a .partial file while it passes the stream to the store below. When the copy fails, would pass the budget, or the stream skips bytes, the cache gives up the copy and never the store. A delete of the blob during a store keeps that store from admitting its copy.

On read. open() claims an entry. The first claimer loads the blob from the store below, and others wait for it. A blob larger than the whole budget is handed back uncached.

Memory mapping. CacheFile maps a cached file read-only in pieces of 1 GiB, since one mapping cannot span more than 2 GiB, and copies out in pieces of 64 KiB. A file system that cannot map, such as one held in memory, leaves no mapping, and the file is then read by position.

Readers. A lease pins an entry while it is read. Eviction only takes idle entries, and the file of an entry that has left the cache is deleted once nobody reads it.

Popularity. Each entry counts its reads, and the count decays by half every half-life:

hits now = hits then × 0.5 ^ (time since / half-life)

Eviction. When the cache passes its budget, it ranks the idle entries by decayed hits per byte and drops the lowest first until it fits, so small blobs read often outlast large ones read rarely.

Restart. Nothing is known of how often an earlier run read a blob. After a restart the entries are rebuilt from the names and sizes of the files, each counted as read once.

In the source

• s3/src/main/java/org/unigrid/hedgehog/s3/spi/BlobStore.java: the interface, with Blob.java, BlobId.java and CredentialStore.java
• s3/src/main/java/org/unigrid/hedgehog/s3/spi/BlobUnavailableException.java: storage that cannot be reached, with BlobNotFoundException.java
• application/src/main/java/org/unigrid/hedgehog/service/storage/StorageBlobStore.java: blobs in network storage
• s3/src/main/java/org/unigrid/hedgehog/s3/resource/Content.java: the slices of a range
• s3/src/main/java/org/unigrid/hedgehog/s3/resource/Multipart.java: parts and completion
• s3/src/main/java/org/unigrid/hedgehog/s3/cache/CachingBlobStore.java: the cache, with CacheEntry.java and CacheLease.java
• s3/src/main/java/org/unigrid/hedgehog/s3/cache/CacheFile.java: the memory mapping
• s3/src/main/java/org/unigrid/hedgehog/s3/cache/Popularity.java: the decaying read count