pip install tc_aws
Authentication is handled by botocore, see Boto3 documentation.
First you need to fork this project. Clone your repo at your computer Next, inside a folder for project:
make setup make test
If all test passed, you have an environment ready to start. We recommend to use python-virtualevn (virtualenv and virtualenv-wrapper)
- tc_aws.loaders.s3_loader - takes a S3 key path and optional bucket name, and downloads the file through the S3 API.
- tc_aws.loaders.presigning_loader - instead of downloading via the API, generates a signed link to the file on S3, then feeds it to the Thumbor's regular http loader. This will likely be more performant, as it avoids async issues with the boto library (see #22 and #14.
What is the purpose of the S3 loader?
You might ask yourself why the S3 loaders are necessary? Aren't files on S3 already available through HTTP already? Why wouldn't you just give the S3 url of your file to Thumbor and let it query the file through HTTP?
If your S3 assets are not public, you'll need to generate a signed URL. This url will be different everytime you sign it. Thumbor will be unable to understand that these urls all refer to the same file, and thus won be able to cache it.
The S3 loader avoids this problem, since you'll only be including the S3 key name in the Thumbor url. Thumbor itself will have the AWS authorization keys to fetch the file.
Additional Configuration values used:
# AWS Region the bucket is located in. TC_AWS_REGION='eu-west-1' # A custom AWS endpoint. TC_AWS_ENDPOINT=''
When using either
TC_AWS_STORAGE_BUCKET='' # S3 bucket for Storage TC_AWS_STORAGE_ROOT_PATH='' # S3 path prefix for Storage bucket # S3 bucket for Loader. If given, source urls are interpreted as keys # within this bucket. If not given, source urls are expected to contain # the bucket name, such as 's3-bucket/keypath'. TC_AWS_LOADER_BUCKET='' # S3 path prefix for Loader bucket. If given, this is prefixed to # all S3 keys. TC_AWS_LOADER_ROOT_PATH='' # Enable HTTP Loader as well? # This would allow you to load watermarks in over your images dynamically through a URI # E.g. # http://your-thumbor.com/unsafe/filters:watermark(http://example.com/watermark.png,0,0,50)/s3_bucket/photo.jpg TC_AWS_ENABLE_HTTP_LOADER=False TC_AWS_ALLOWED_BUCKETS=False # List of allowed bucket to be requested
tc_aws.storages.s3_storage is enabled.
TC_AWS_STORAGE_BUCKET='' # S3 bucket for Storage TC_AWS_STORAGE_ROOT_PATH='' # S3 path prefix for Storage bucket # put data into S3 using the Server Side Encryption functionality to # encrypt data at rest in S3 # https://aws.amazon.com/about-aws/whats-new/2011/10/04/amazon-s3-announces-server-side-encryption-support/ TC_AWS_STORAGE_SSE=False # put data into S3 with Reduced Redundancy # https://aws.amazon.com/about-aws/whats-new/2010/05/19/announcing-amazon-s3-reduced-redundancy-storage/ TC_AWS_STORAGE_RRS=False
Result storage settings
tc_aws.result_storages.s3_storage is enabled.
TC_AWS_RESULT_STORAGE_BUCKET='' # S3 bucket for result Storage TC_AWS_RESULT_STORAGE_ROOT_PATH='' # S3 path prefix for Result storage bucket TC_AWS_MAX_RETRY=0 # Max retries for get image from S3 Bucket. Default is 0 TC_AWS_STORE_METADATA=False # Store result with metadata (for instance content-type)
TC_AWS_RANDOMIZE_KEYS=False # Adds some randomization in the S3 keys for the Storage and Result Storage. Defaults to False for Backwards Compatibility, set it to True for performance. TC_AWS_ROOT_IMAGE_NAME='root_image' # Sets a default name for requested images ending with a trailing /. Those images will be stored in result_storage and storage under the name set in this configuration.
Check your configuration
You may have errors due to unproperly formatted configuration. For instance, if you've set the value "None" or "", this will default to the string value, and not the False or None value exepected, which can lead to issues later on. So if you're running into issues, try to re-read the configuration, taking care of the formatting.
Make it work with riak
You'll need to tweak a bit your aws configuration (see boto doc), which should in the end look as follows:
[default] aws_access_key_id = (KEY) aws_secret_access_key =(SECRET) s3 = signature_version = s3