Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Add S3 backend #156
This new backend currently isn't hooked up; new and existing installs will continue to use the localfs backend. * Rework torrent generation to be backend-dependent so we can use S3's existing torrent API. * Remove the torrent test cases, which broke with this torrent rework; they will need to be added back later. * Use `http.MaxBytesReader` for better max size handling. * Allow backends to return errors in `ServeFile` if needed.
* Bail out on files that are too large earlier if possible. * Return 400 instead of 500 for empty files and files that are too large (when we can bail out early).
Although S3 offers a GetObjectTorrent API call to generate a torrent file on their end, it doesn't look like any similar systems with S3 compatible APIs actually support it. Notably, Minio and Wasabi do not. In order to remain compatible with these, it's better to not rely on the storage backend to handle creation.
Previously, missing files would return a "corrupt metadata" error because errors were not being properly handled in the S3 backend. This change catches not found errors and passes them up to be handled accordingly. Alpine Linux puts the CA cert bundle at /etc/ssl/cert.pem by default; to ensure that Go looks in that location, we now set the SSL_CERT_FILE environment variable.
When generating a torrent, we need to get the SHA1 hash of each chunk of the file. Because we stream the data for S3, Read doesn't always fill the chunk buffer, so to fix this, we can use ReadFull.
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments.