Provides endpoints for AwsS3Multipart Uppy plugin
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib Use content_disposition gem for :content_disposition create_multipart… Dec 19, 2018
test Use content_disposition gem for :content_disposition create_multipart… Dec 19, 2018
.gitignore Initial commit Aug 27, 2018
CODE_OF_CONDUCT.md Initial commit Aug 27, 2018
Gemfile Initial commit Aug 27, 2018
LICENSE.txt
README.md README tweaks Dec 30, 2018
Rakefile Initial commit Aug 27, 2018
uppy-s3_multipart.gemspec Bump to 0.3.0 Dec 19, 2018

README.md

Uppy::S3Multipart

Provides a Rack application that implements endpoints for the AwsS3Multipart Uppy plugin. This enables multipart uploads directly to S3, which is recommended when dealing with large files, as it allows resuming interrupted uploads.

Installation

Add the gem to your Gemfile:

gem "uppy-s3_multipart", "~> 0.2"

Setup

In order to allow direct multipart uploads to your S3 bucket, we need to update the bucket's CORS configuration. In the AWS S3 Console go to your bucket, click on "Permissions" tab and then on "CORS configuration". There paste in the following:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <CORSRule>
    <AllowedOrigin>https://my-app.com</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>Authorization</AllowedHeader>
    <AllowedHeader>x-amz-date</AllowedHeader>
    <AllowedHeader>x-amz-content-sha256</AllowedHeader>
    <AllowedHeader>content-type</AllowedHeader>
    <ExposeHeader>ETag</ExposeHeader>
  </CORSRule>
  <CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
  </CORSRule>
</CORSConfiguration>

Replace https://my-app.com with the URL to your app (in development you can set this to *). Once you've hit "Save", it may take some time for the new CORS settings to be applied.

Usage

This gem provides a Rack application that you can mount inside your main application. If you're using Shrine, you can initialize the Rack application via the uppy_s3_multipart Shrine plugin, otherwise you can initialize it directly.

Shrine

In your Shrine initializer load the uppy_s3_multipart plugin:

require "shrine"
require "shrine/storage/s3"

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", ...),
  store: Shrine::Storage::S3.new(...),
}

# ...
Shrine.plugin :uppy_s3_multipart # load the plugin

The plugin will provide a Shrine.uppy_s3_multipart method that creates a new Uppy::S3Multipart::App instance, which is a Rack app that you can mount inside your main application:

# Rails (config/routes.rb)
Rails.application.routes.draw do
  mount Shrine.uppy_s3_multipart(:cache) => "/s3/multipart"
end

# Rack (config.ru)
map "/s3/multipart" do
  run Shrine.uppy_s3_multipart(:cache)
end

Now in your Uppy configuration point serverUrl to your app's URL:

// ...
uppy.use(Uppy.AwsS3Multipart, {
  serverUrl: '/',
})

In the upload-success Uppy callback you can then construct the Shrine uploaded file data (this example assumes your temporary Shrine S3 storage has prefix: "cache" set):

uppy.on('upload-success', function (file, data, uploadURL) {
  var uploadedFileData = JSON.stringify({
    id: uploadURL.match(/\/cache\/([^\?]+)/)[1], // extract key without prefix
    storage: 'cache',
    metadata: {
      size:      file.size,
      filename:  file.name,
      mime_type: file.type,
    }
  })
  // ...
})

See Adding Direct S3 Uploads for an example of a complete Uppy setup with Shrine. From there you can swap the presign_endpoint + AwsS3 code with the uppy_s3_multipart + AwsS3Multipart setup.

Note that Shrine won't extract metadata from directly upload files on assignment by default. Instead, it will just copy metadata that was extracted on the client side. See this section for the rationale and instructions on how to opt in.

App

You can also use uppy-s3_multipart without Shrine, by initializing the Uppy::S3Multipart::App directly:

require "uppy/s3_multipart"

resource = Aws::S3::Resource.new(
  access_key_id:     "...",
  secret_access_key: "...",
  region:            "...",
)

bucket = resource.bucket("my-bucket")

UPPY_S3_MULTIPART_APP = Uppy::S3Multipart::App.new(bucket: bucket)

You can mount it inside your main app in the same way:

# Rails (config/routes.rb)
Rails.application.routes.draw do
  mount UPPY_S3_MULTIPART_APP => "/s3/multipart"
end

# Rack (config.ru)
map "/s3/multipart" do
  run UPPY_S3_MULTIPART_APP
end

This will add the routes that the AwsS3Multipart Uppy plugin expects:

POST   /s3/multipart
GET    /s3/multipart/:uploadId
GET    /s3/multipart/:uploadId/:partNumber
POST   /s3/multipart/:uploadId/complete
DELETE /s3/multipart/:uploadId

Now in your Uppy configuration point serverUrl to your app's URL:

// ...
uppy.use(Uppy.AwsS3Multipart, {
  serverUrl: '/',
})

Configuration

This section describe various configuration options that you can pass to Uppy::S3Multipart::App.

:bucket

The :bucket option is mandatory and accepts an instance of Aws::S3::Bucket. It's easiest to create an Aws::S3::Resource, and call #bucket on it.

require "uppy/s3_multipart"

resource = Aws::S3::Resource.new(
  access_key_id:     "<ACCESS_KEY_ID>",
  secret_access_key: "<SECRET_ACCESS_KEY>",
  region:            "<REGION>",
)

bucket = resource.bucket("<BUCKET>")

Uppy::S3MUltipart::App.new(bucket: bucket)

If you want to use Minio, you can easily configure your Aws::S3::Bucket to point to your Minio server:

resource = Aws::S3::Resource.new(
  access_key_id:     "<MINIO_ACCESS_KEY>", # "AccessKey" value
  secret_access_key: "<MINIO_SECRET_KEY>", # "SecretKey" value
  endpoint:          "<MINIO_ENDPOINT>",   # "Endpoint"  value
  region:            "us-east-1",
  force_path_style:  true,
)

bucket = resource.bucket("<MINIO_BUCKET>") # name of the bucket you created

See the Aws::S3::Client#initialize docs for all supported configuration options. In the Shrine plugin this option is inferred from the S3 storage.

:prefix

The :prefix option allows you to specify a directory which you want the files to be uploaded to.

Uppy::S3Multipart::App.new(bucket: bucket, prefix: "cache")

In the Shrine plugin this option is inferred from the S3 storage:

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", **options),
  store: Shrine::Storage::S3.new(**options),
}

:options

The :options option allows you to pass additional parameters to Client operations. With the Shrine plugin they can be passed when initializing the plugin:

Shrine.plugin :uppy_s3_multipart, options: { ... }

or when creating the app:

Shrine.uppy_s3_multipart(:cache, options: { ... })

In the end they are just forwarded to Uppy::S3Multipart::App#initialize:

Uppy::S3Multipart::App.new(bucket: bucket, options: { ... })

In the :options hash keys are Client operation names, and values are the parameters. The parameters can be provided statically:

options: {
  create_multipart_upload: { cache_control: "max-age=#{365*24*60*60}" },
  prepare_upload_part:     { expires_in: 10 },
}

or generated dynamically for each request, in which case a Rack::Request object is also passed to the block:

options: {
  create_multipart_upload: -> (request) {
    { key: SecureRandom.uuid }
  }
}

The initial request to POST /s3/multipart (which calls the #create_multipart_upload operation) will contain type and filename query parameters, so for example you could use that to make requesting the URL later force a download with the original filename (using the content_disposition gem):

options: {
  create_multipart_upload: -> (request) {
    filename = request.params["filename"]

    { content_disposition: ContentDisposition.attachment(filename) }
  }
}

See the Client section for list of operations and parameters they accept.

:public

The :public option sets the ACL of uploaded objects to public-read, and makes sure the object URL returned at the end is a public non-expiring URL without query parameters.

Uppy::S3Multipart::App.new(bucket: bucket, public: true)

It's really just a shorthand for:

Uppy::S3Multipart::App.new(bucket: bucket, options: {
  create_multipart_upload: { acl: "public-read" },
  object_url: { public: true },
})

In the Shrine plugin this option is inferred from the S3 storage (available from Shrine 2.13):

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", public: true, **options),
  store: Shrine::Storage::S3.new(**options),
}

Client

If you would rather implement the endpoints yourself, you can utilize the Uppy::S3Multipart::Client to make S3 requests.

require "uppy/s3_multipart/client"

client = Uppy::S3Multipart::Client.new(bucket: bucket)

#create_multipart_upload

Initiates a new multipart upload.

client.create_multipart_upload(key: "foo", **options)
# => { upload_id: "MultipartUploadId", key: "foo" }

Accepts:

Returns:

  • :upload_id – id of the created multipart upload
  • :key – object key

#list_parts

Retrieves currently uploaded parts of a multipart upload.

client.list_parts(upload_id: "MultipartUploadId", key: "foo", **options)
# => [ { part_number: 1, size: 5402383, etag: "etag1" },
#      { part_number: 2, size: 5982742, etag: "etag2" },
#      ... ]

Accepts:

Returns:

  • array of parts

    • :part_number – position of the part
    • :size – filesize of the part
    • :etag – etag of the part

#prepare_upload_part

Returns the endpoint that should be used for uploading a new multipart part.

client.prepare_upload_part(upload_id: "MultipartUploadId", key: "foo", part_number: 1, **options)
# => { url: "https://my-bucket.s3.amazonaws.com/foo?partNumber=1&uploadId=MultipartUploadId&..." }

Accepts:

Returns:

  • :url – endpoint that should be used for uploading a new multipart part via a PUT request

#complete_multipart_upload

Finalizes the multipart upload and returns URL to the object.

client.complete_multipart_upload(upload_id: upload_id, key: key, parts: [{ part_number: 1, etag: "etag1" }], **options)
# => { location: "https://my-bucket.s3.amazonaws.com/foo?..." }

Accepts:

Returns:

  • :location – URL to the uploaded object

#object_url

Generates URL to the object.

client.object_url(key: key, **options)
# => "https://my-bucket.s3.amazonaws.com/foo?..."

This is called after #complete_multipart_upload in the app and returned in the response.

Accepts:

Returns:

  • URL to the object

#abort_multipart_upload

Aborts the multipart upload, removing all parts uploaded so far.

client.abort_multipart_upload(upload_id: upload_id, key: key, **options)
# => {}

Accepts:

Contributing

You can run the test suite with

$ bundle exec rake test

This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

The gem is available as open source under the terms of the MIT License.