Ruby Gherkin
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
features
lib
test
.gitignore
.travis.yml
CHANGELOG.md
CODE_OF_CONDUCT.md
Gemfile
LICENSE.txt
README.md
Rakefile
tus-server.gemspec

README.md

tus-ruby-server

A Ruby server for the tus resumable upload protocol. It implements the core 1.0 protocol, along with the following extensions:

Installation

# Gemfile
gem "tus-server", "~> 2.0"

Usage

The gem provides a Tus::Server Rack app, which you can mount inside your main application. If you're using Rails, you can mount it in config/routes.rb:

# config/routes.rb (Rails)
Rails.application.routes.draw do
  mount Tus::Server => "/files"
end

Otherwise you can run it in config.ru:

# config.ru (Rack)
require "tus/server"

map "/files" do
  run Tus::Server
end

Now you can tell your tus client library (e.g. tus-js-client) to use this endpoint:

// using tus-js-client
new tus.Upload(file, {
  endpoint: "/files",
  chunkSize: 5*1024*1024, // required unless using Goliath
  // ...
})

By default uploaded files will be stored in the data/ directory. After the upload is complete, you'll probably want to attach the uploaded file to a database record. Shrine is currently the only file attachment library that integrates well with tus-ruby-server, see this walkthrough that adds resumable uploads from scratch, and for a complete example you can check out the demo app.

Goliath

Running the tus server alongside your main app using classic web servers like Puma or Unicorn is probably fine for most cases, however, it does come with a few gotchas. First, since these web servers don't accept partial requests (request where the request body hasn't been fully received), the tus client must be configured to split the upload into multiple requests. Second, since web workers are tied for the duration of the request, serving uploaded files through the tus server app could significantly impact request throughput, so you need to be careful to avoid that.

There is an alternative. Goliath is an asychronous web server built on top of EventMachine, which supports streaming requests and streaming responses.

  • Asynchronous streaming requests allows the tus server to begin saving uploaded data while it's still being received. If the request is interrupted, the tus server will attempt to save as much of the data that was received so far. This means it's not necessary for the tus client to split the upload into multiple smaller requests.

  • Asynchronous streaming responses allows the tus server to stream large files with very small impact to the request throughput.

Since Goliath is web server, to run tus server on it we'll have to run it as a standalone web app. It's recommended that you use goliath-rack_proxy for running your tus server app:

# Gemfile
gem "tus-server", "~> 2.0"
gem "goliath-rack_proxy", "~> 1.0"
# tus.rb
require "tus/server"
require "goliath/rack_proxy"

# any additional Tus::Server configuration you want to put in here

class GoliathTusServer < Goliath::RackProxy
  rack_app Tus::Server
  rewindable_input false # set to true if you're using checksums
end
$ ruby tus.rb --stdout # enable logging

This will run the tus server app on the root URL; if you want to run it on some path you can use Rack::Builder:

class GoliathTusServer < Goliath::RackProxy
  rack_app Rack::Builder.new {
    map("/files") { run Tus::Server }
  }
  rewindable_input false # set to true if you're using checksums
end

In this case you'll have to configure the tus client to point to the standalone Goliath app:

// using tus-js-client
new tus.Upload(file, {
  endpoint: "http://localhost:9000/files",
  // ...
})

Storage

Filesystem

By default Tus::Server stores uploaded files in the data/ directory. You can configure a different directory:

require "tus/storage/filesystem"

Tus::Server.opts[:storage] = Tus::Storage::Filesystem.new("public/tus")

If the configured directory doesn't exist, it will automatically be created. By default the UNIX permissions applied will be 0644 for files and 0755 for directories, but you can set different permissions:

Tus::Storage::Filesystem.new("data", permissions: 0600, directory_permissions: 0777)

One downside of filesystem storage is that it doesn't work by default if you want to run tus-ruby-server on multiple servers, you'd have to set up a shared filesystem between the servers. Another downside is that you have to make sure your servers have enough disk space. Also, if you're using Heroku, you cannot store files on the filesystem as they won't persist.

All these are reasons why you might store uploaded data on a different storage, and luckily tus-ruby-server ships with two more storages.

Serving files

If your retrieving uploaded files through the download endpoint, by default the files will be served through the Ruby application. However, that's very inefficient, as web workers are tied when serving download requests and cannot serve additional requests for that duration.

Therefore, it's highly recommended to delegate serving uploaded files to your frontend server (Nginx, Apache). This can be achieved with the Rack::Sendfile middleware, see its documentation to learn more about how to use it with popular frontend servers.

If you're using Rails, you can enable the Rack::Sendfile middleware by setting the config.action_dispatch.x_sendfile_header value accordingly:

Rails.application.config.action_dispatch.x_sendfile_header = "X-Sendfile" # Apache and lighttpd
# or
Rails.application.config.action_dispatch.x_sendfile_header = "X-Accel-Redirect" # Nginx

Otherwise you can add the Rack::Sendfile middleware to the stack in config.ru:

use Rack::Sendfile, "X-Sendfile" # Apache and lighttpd
# or
use Rack::Sendfile, "X-Accel-Redirect" # Nginx

Amazon S3

You can switch to Tus::Storage::S3 to uploads files to AWS S3 using the multipart API. For this you'll also need to add the aws-sdk-s3 gem to your Gemfile.

# Gemfile
gem "aws-sdk-s3", "~> 1.2"
require "tus/storage/s3"

# You can omit AWS credentials if you're authenticating in other ways
Tus::Server.opts[:storage] = Tus::Storage::S3.new(
  bucket:            "my-app", # required
  access_key_id:     "abc",
  secret_access_key: "xyz",
  region:            "eu-west-1",
)

One thing to note is that S3's multipart API requires each chunk except the last to be 5MB or larger, so that is the minimum chunk size that you can specify on your tus client if you want to use the S3 storage.

If you'll be retrieving uploaded files through the tus server app, it's recommended to set Tus::Server.opts[:redirect_download] to true. This will avoid tus server downloading and serving the file from S3, and instead have the download endpoint redirect to the direct S3 object URL.

Tus::Server.opts[:redirect_download] = true

You can customize how the S3 object URL is being generated by passing a block to :redirect_download, which will then be evaluated in the context of the Tus::Server instance (which allows accessing the request object). See Aws::S3::Object#get for the list of options that Tus::Storage::S3#file_url accepts.

Tus::Server.opts[:redirect_download] = -> (uid, info, **options) do
  storage.file_url(uid, info, response_expires: 10, **options) # link expires after 10 seconds
end

If you want to files to be stored in a certain subdirectory, you can specify a :prefix in the storage configuration.

Tus::Storage::S3.new(prefix: "tus", **options)

You can also specify additional options that will be fowarded to Aws::S3::Client#create_multipart_upload using :upload_options.

Tus::Storage::S3.new(upload_options: {content_disposition: "attachment"}, **options)

All other options will be forwarded to Aws::S3::Client#initialize, so you can for example change the :endpoint to use S3's accelerate host:

Tus::Storage::S3.new(endpoint: "https://s3-accelerate.amazonaws.com", **options)

MongoDB GridFS

MongoDB has a specification for storing and retrieving large files, called "GridFS". Tus-ruby-server ships with Tus::Storage::Gridfs that you can use, which uses the Mongo gem.

# Gemfile
gem "mongo", "~> 2.3"
require "tus/storage/gridfs"

client = Mongo::Client.new("mongodb://127.0.0.1:27017/mydb")
Tus::Server.opts[:storage] = Tus::Storage::Gridfs.new(client: client)

You can change the database prefix (defaults to fs):

Tus::Storage::Gridfs.new(client: client, prefix: "fs_temp")

By default MongoDB Gridfs stores files in chunks of 256KB, but you can change that with the :chunk_size option:

Tus::Storage::Gridfs.new(client: client, chunk_size: 1*1024*1024) # 1 MB

Note that if you're using the concatenation tus feature with Gridfs, all partial uploads except the last one are required to fill in their Gridfs chunks, meaning the length of each partial upload needs to be a multiple of the :chunk_size number.

Other storages

If none of these storages suit you, you can write your own, you just need to implement the same public interface:

def create_file(uid, info = {})            ... end
def concatenate(uid, part_uids, info = {}) ... end
def patch_file(uid, io, info = {})         ... end
def update_info(uid, info)                 ... end
def read_info(uid)                         ... end
def get_file(uid, info = {}, range: nil)   ... end
def file_url(uid, info = {}, **options)    ... end # optional
def delete_file(uid, info = {})            ... end
def expire_files(expiration_date)          ... end

Maximum size

By default the size of files the tus server will accept is unlimited, but you can configure the maximum file size:

Tus::Server.opts[:max_size] = 5 * 1024*1024*1024 # 5GB

Expiration

Tus-ruby-server automatically adds expiration dates to each uploaded file, and refreshes this date on each PATCH request. By default files expire 7 days after they were last updated, but you can modify :expiration_time:

Tus::Server.opts[:expiration_time] = 2*24*60*60 # 2 days

Tus-ruby-server won't automatically delete expired files, but each storage knows how to expire old files, so you just have to set up a recurring task that will call #expire_files.

expiration_time = Tus::Server.opts[:expiration_time]
tus_storage     = Tus::Server.opts[:storage]
expiration_date = Time.now.utc - expiration_time

tus_storage.expire_files(expiration_time)

Download

In addition to implementing the tus protocol, tus-ruby-server also comes with a GET endpoint for downloading the uploaded file, which by default streams the file from the storage. It supports Range requests, so you can use the tus file URL as src in <video> and <audio> HTML tags.

It's highly recommended not to serve files through the app, but offload it to your frontend server if using disk storage, or if using S3 storage have the download endpoint redirect to the S3 object URL. See the documentation for the individual storage for instructions how to set this up.

The endpoint will automatically use the following Upload-Metadata values if they're available:

  • type -- used to set Content-Type response header
  • name -- used to set Content-Disposition response header

The Content-Disposition header will be set to "inline" by default, but you can change it to "attachment" if you want the browser to always force download:

Tus::Server.opts[:disposition] = "attachment"

Checksum

The following checksum algorithms are supported for the checksum extension:

  • SHA1
  • SHA256
  • SHA384
  • SHA512
  • MD5
  • CRC32

Tests

Run tests with

$ bundle exec rake test # unit tests
$ bundle exec cucumber  # acceptance tests

Set MONGO=1 environment variable if you want to also run MongoDB tests.

Inspiration

The tus-ruby-server was inspired by rubytus.

License

MIT