Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
app
 
 
res
 
 
 
 
 
 
 
 

Python version MIT license

Upload files or folders (even with subfolders) to Amazon S3 in a totally automatized way taking advantage of:

  • Amazon S3 Multipart Upload: The uploaded files are processed transparently in parts improving the throughput and the quick recovery from any network issues.

  • Resilent Retry System: Intelligent S3 Upload has been built to detect any error during the uploading process and to perform any retries whenever is necessary.

  • User Friendly Interface: Just check the demo to see with your own eyes how the upload process is performed.

Checkout the demo

Installation

Clone the repository

git clone https://github.com/polius/intelligent-s3-upload.git

Install the dependencies

python3 -m pip install boto3 --user
python3 -m pip install requests --user

Setup

Before executing the Intelligent S3 Upload, modify the credentials.json file.

{
    "aws_access_key_id": "",
    "aws_secret_access_key": "",
    "region_name": "",
    "bucket_name": "",
    "bucket_prefix": "",
    "storage_class": "",
    "skip_s3_existing_files": true,
    "server_side_encryption": true,
    "slack_url": ""
}
  • aws_access_key_id | aws_secret_access_key: Credentials generated by Amazon IAM.
  • region_name: The AWS Region Code where the bucket is located.
  • bucket_name: The bucket name created by Amazon S3.
  • bucket_path: (Optional) The bucket path to store the uploaded objects.
  • storage_class: The type of storage to use for the uploaded object. These are the possible values:
storage_class
STANDARD
REDUCED_REDUNDANCY
STANDARD_IA
ONEZONE_IA
INTELLIGENT_TIERING
GLACIER
DEEP_ARCHIVE
OUTPOSTS
  • skip_s3_existing_files: Skip uploading objects if these already exists in S3. Possible values: [ true | false ]
  • server_side_encryption: Enable Server-side encryption using the Amazon S3 key (SSE-S3). Possible values: [ true | false ]
  • slack_url: (Optional) Enter a Webhook URL to send a message to Slack when a upload finishes.

AWS Policy

To be able to run the script check that your policy meets the following requirements:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucketMultipartUploads",
                "s3:AbortMultipartUpload",
                "s3:ListBucket",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::yourbucket/*",
                "arn:aws:s3:::yourbucket"
            ]
        }
    ]
}

Execution

python3 upload.py --path "{PATH}"

Replace the {PATH} string with the absolute file/folder path.

License

This project is licensed under the MIT license. See the LICENSE file for more info.

About

A complete Python application to automatize the process of uploading files to Amazon S3

Topics

Resources

License

Packages

No packages published