feat: add presigned URL redirect for blob storage streaming#194
Merged
raymondjacobson merged 5 commits intomainfrom Apr 2, 2026
Merged
feat: add presigned URL redirect for blob storage streaming#194raymondjacobson merged 5 commits intomainfrom
raymondjacobson merged 5 commits intomainfrom
Conversation
When OPENAUDIO_BLOB_STORAGE_STREAMING=true and the storage backend supports presigned URLs (S3/GCS), redirect stream requests directly to the storage backend via HTTP 307 instead of proxying bytes through the server. This reduces CPU, memory, and I/O load on content nodes. - Duration-aware expiry: looks up track duration from uploads table FFProbe data (with functional index for fast lookup), computes expiry as duration * 1.1 with a 5-minute floor and 2-hour default - Graceful fallback: if SignedURL fails, falls through to normal proxy - Skips presigned URLs for ID3-tagged streams and download requests - Feature is off by default, only activates for s3://, https://, gs:// Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests for supportsPresignedURLs(), presignedURLExpiry(), and the combined feature flag + DSN computation. Covers all storage backends, duration edge cases (unknown, short, long, DJ mixes), and floor/default behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…te handleConsensusCheck The 3-minute track test expected 198s but 198s < 5min floor, so the correct expectation is the 5-minute minimum. Also removes the duplicate handleConsensusCheck function from main.go (already extracted to consensus_check.go) which was causing build failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The `openaudio` gitignore pattern causes files in cmd/openaudio/ to be ignored. consensus_check.go was created but never force-added to git, so CI only had main.go which referenced the missing function. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rickyrombo
reviewed
Apr 2, 2026
| // supportsPresignedURLs returns true if the storage backend supports generating presigned URLs. | ||
| // S3, S3-compatible (https), and GCS backends support this. File and Azure backends do not. | ||
| func supportsPresignedURLs(dsn string) bool { | ||
| for _, prefix := range []string{"s3://", "https://", "gs://"} { |
Contributor
There was a problem hiding this comment.
is this definitely true of all implementations of S3-compatible blob storage? wondering if this should instead be a separate config w/ some validation to fail fast
Contributor
Author
There was a problem hiding this comment.
there's a separate env var to gate the field so i think it's enough, but maybe just remove this and keep the env var?
Contributor
There was a problem hiding this comment.
ah yeah i see that now. and yeah i think if you enable that env var but are using an "invalid" url, we should either throw or do nothing and let it "fail" vs silently not doing the thing you configured it to do
Contributor
Author
There was a problem hiding this comment.
good call. will make that change
rickyrombo
approved these changes
Apr 2, 2026
…errors Instead of checking the DSN prefix via supportsPresignedURLs(), the BlobStorageStreaming config flag now directly controls whether presigned URL redirects are attempted. If SignedURL() fails at runtime, the node returns a 500 error instead of silently falling back to proxying, making misconfiguration immediately visible. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: add presigned URL redirect for blob storage streaming
When OPENAUDIO_BLOB_STORAGE_STREAMING=true and the storage backend
supports presigned URLs (S3/GCS), redirect stream requests directly
to the storage backend via HTTP 307 instead of proxying bytes through
the server. This reduces CPU, memory, and I/O load on content nodes.
FFProbe data (with functional index for fast lookup), computes
expiry as duration * 1.1 with a 5-minute floor and 2-hour default
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
test: add unit tests for presigned URL logic
Tests for supportsPresignedURLs(), presignedURLExpiry(), and the
combined feature flag + DSN computation. Covers all storage backends,
duration edge cases (unknown, short, long, DJ mixes), and floor/default
behavior.
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com