The official server implementation of the tus resumable upload protocol.
Go Shell
Clone or download
Acconut Merge pull request #195 from egalpin/update-upload-length-logic
Updates uploadLength logic in PatchFile
Latest commit f17a393 Jun 22, 2018
Permalink
Failed to load latest commit information.
.hooks Squashed commit of the following: Jul 19, 2017
.infra Fixed deletion script @Acconut Jun 11, 2018
.scripts Change cluster name Jun 11, 2018
cmd/tusd Allow AWS-SDK to handle gathering credentials. Jun 14, 2018
consullocker Use new consul test server functions Apr 12, 2017
docs Add ProxyPreserveHose to apache2 example Feb 22, 2018
filestore Add length deferrer support to FileStore Jun 3, 2018
gcsstore Handle GCS object-not-found error May 25, 2018
limitedstore Format code in limitedstore Oct 13, 2016
memorylocker Make minor improvements to memorylocker internals Feb 6, 2017
prometheuscollector Correct metrics types Mar 30, 2017
s3store Avoid incorrectly returning errors when upload length is deferred Jun 3, 2018
uid Correct linting issue and misspellings Sep 27, 2016
vendor Update dependencies Nov 20, 2017
.gitignore Remove frey references Mar 14, 2018
.travis.yml Add Go 1.10 to testing setup May 13, 2018
Dockerfile Commit permissions fix Jan 4, 2018
LICENSE.txt Update year Jan 19, 2017
README.md A note about shared config file to README Jun 21, 2018
appveyor.yml Add support for Google Cloud storage as a DataStore (#106) Sep 16, 2017
composer.go Add length deferrer support to composer Jun 3, 2018
composer.mgo Add length deferrer support to composer Jun 3, 2018
composer_test.go Use memorylocker in example for composer Sep 29, 2016
concat_test.go Remove creation-defer-length declaration from tests that don't use it Jun 3, 2018
config.go Squashed commit of the following: Jul 19, 2017
config_test.go Add tests for Config structure Mar 11, 2016
cors_test.go Add Upload-Defer-Length to access-control-{allow,expose}-headers Jun 11, 2018
datastore.go Add LengthDeferrerDataStore interface Jun 3, 2018
doc.go Add documentation about using tusd Mar 4, 2016
get_test.go Allow certain file types to be shown inline in browsers Feb 28, 2018
handler.go Embed UnroutedHandler into Handler Mar 12, 2016
handler_mock_test.go Regenerate handler mock Jun 3, 2018
head_test.go Add HEAD tests Jun 3, 2018
log.go Add proper, formatted logging Sep 23, 2016
metrics.go Move network timeout handling back in UnroutedHandler Mar 1, 2017
options_test.go Remove creation-defer-length declaration from tests that don't use it Jun 3, 2018
patch_test.go Specify MaxSize in PATCH tests that refer to it Jun 3, 2018
post_test.go Add POST tests Jun 3, 2018
subtest_go17_test.go Enable subtests for environments prior to Go 1.7 Oct 13, 2016
subtest_test.go Improve output of subtests when emulated Oct 13, 2016
terminate_test.go Remove creation-defer-length declaration from tests that don't use it Jun 3, 2018
tusd.code-workspace Remove frey references Mar 14, 2018
unrouted_handler.go Updates uploadLength logic in PatchFile Jun 22, 2018
utils_test.go Add length deferrer support to FullDataStore Jun 3, 2018

README.md

tusd

Tus logo

tus is a protocol based on HTTP for resumable file uploads. Resumable means that an upload can be interrupted at any moment and can be resumed without re-uploading the previous data again. An interruption may happen willingly, if the user wants to pause, or by accident in case of an network issue or server outage.

tusd is the official reference implementation of the tus resumable upload protocol. The protocol specifies a flexible method to upload files to remote servers using HTTP. The special feature is the ability to pause and resume uploads at any moment allowing to continue seamlessly after e.g. network interruptions.

It is capable of accepting uploads with arbitrary sizes and storing them locally on disk, on Google Cloud Storage or on AWS S3 (or any other S3-compatible storage system). Due to its modularization and extensibility, support for nearly any other cloud provider could easily be added to tusd.

Protocol version: 1.0.0

Getting started

Download pre-builts binaries (recommended)

You can download ready-to-use packages including binaries for OS X, Linux and Windows in various formats of the latest release.

Compile from source

The only requirement for building tusd is Go 1.5 or newer. If you meet this criteria, you can clone the git repository, install the remaining depedencies and build the binary:

git clone git@github.com:tus/tusd.git
cd tusd

go get -u github.com/aws/aws-sdk-go/...
go get -u github.com/prometheus/client_golang/prometheus

go build -o tusd cmd/tusd/main.go

Running tusd

Start the tusd upload server is as simple as invoking a single command. For example, following snippet demostrates how to start a tusd process which accepts tus uploads at http://localhost:1080/files/ and stores them locally in the ./data directory:

$ tusd -dir ./data
[tusd] Using './data' as directory storage.
[tusd] Using 0.00MB as maximum size.
[tusd] Using 0.0.0.0:1080 as address to listen.
[tusd] Using /files/ as the base path.
[tusd] Using /metrics as the metrics path.

Alternatively, if you want to store the uploads on an AWS S3 bucket, you only have to specify the bucket and provide the corresponding access credentials and region information using environment variables (if you want to use a S3-compatible store, use can use the -s3-endpoint option):

$ export AWS_ACCESS_KEY_ID=xxxxx
$ export AWS_SECRET_ACCESS_KEY=xxxxx
$ export AWS_REGION=eu-west-1
$ tusd -s3-bucket my-test-bucket.com
[tusd] Using 's3://my-test-bucket.com' as S3 bucket for storage.
[tusd] Using 0.00MB as maximum size.
[tusd] Using 0.0.0.0:1080 as address to listen.
[tusd] Using /files/ as the base path.
[tusd] Using /metrics as the metrics path.

tusd is also able to read the credentials automatically from a shared credentials file (~/.aws/credentials) as described in https://github.com/aws/aws-sdk-go#configuring-credentials.

Furthermore, tusd also has support for storing uploads on Google Cloud Storage. In order to enable this feature, supply the path to your account file containing the necessary credentials:

$ export GCS_SERVICE_ACCOUNT_FILE=./account.json
$ tusd -gcs-bucket my-test-bucket.com
[tusd] Using 'gcs://my-test-bucket.com' as GCS bucket for storage.
[tusd] Using 0.00MB as maximum size.
[tusd] Using 0.0.0.0:1080 as address to listen.
[tusd] Using /files/ as the base path.
[tusd] Using /metrics as the metrics path.

Besides these simple examples, tusd can be easily configured using a variety of command line options:

$ tusd -help
Usage of tusd:
  -base-path string
    	Basepath of the HTTP server (default "/files/")
  -behind-proxy
    	Respect X-Forwarded-* and similar headers which may be set by proxies
  -dir string
    	Directory to store uploads in (default "./data")
  -expose-metrics
    	Expose metrics about tusd usage (default true)
  -gcs-bucket string
    	Use Google Cloud Storage with this bucket as storage backend (requires the GCS_SERVICE_ACCOUNT_FILE environment variable to be set)
  -hooks-dir string
    	Directory to search for available hooks scripts
  -hooks-http string
    	An HTTP endpoint to which hook events will be sent to
  -hooks-http-backoff int
    	Number of seconds to wait before retrying each retry (default 1)
  -hooks-http-retry int
    	Number of times to retry on a 500 or network timeout (default 3)
  -host string
    	Host to bind HTTP server to (default "0.0.0.0")
  -max-size int
    	Maximum size of a single upload in bytes
  -metrics-path string
    	Path under which the metrics endpoint will be accessible (default "/metrics")
  -port string
    	Port to bind HTTP server to (default "1080")
  -s3-bucket string
    	Use AWS S3 with this bucket as storage backend (requires the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_REGION environment variables to be set)
  -s3-endpoint string
    	Endpoint to use S3 compatible implementations like minio (requires s3-bucket to be pass)
  -store-size int
    	Size of space allowed for storage
  -timeout int
    	Read timeout for connections in milliseconds.  A zero value means that reads will not timeout (default 30000)
  -version
    	Print tusd version information

Using tusd manually

Besides from running tusd using the provided binary, you can embed it into your own Go program:

package main

import (
	"fmt"
	"net/http"

	"github.com/tus/tusd"
	"github.com/tus/tusd/filestore"
)

func main() {
	// Create a new FileStore instance which is responsible for
	// storing the uploaded file on disk in the specified directory.
	// This path _must_ exist before tusd will store uploads in it.
	// If you want to save them on a different medium, for example
	// a remote FTP server, you can implement your own storage backend
	// by implementing the tusd.DataStore interface.
	store := filestore.FileStore{
		Path: "./uploads",
	}

	// A storage backend for tusd may consist of multiple different parts which
	// handle upload creation, locking, termination and so on. The composer is a
	// place where all those seperated pieces are joined together. In this example
	// we only use the file store but you may plug in multiple.
	composer := tusd.NewStoreComposer()
	store.UseIn(composer)

	// Create a new HTTP handler for the tusd server by providing a configuration.
	// The StoreComposer property must be set to allow the handler to function.
	handler, err := tusd.NewHandler(tusd.Config{
		BasePath:      "/files/",
		StoreComposer: composer,
	})
	if err != nil {
		panic(fmt.Errorf("Unable to create handler: %s", err))
	}

	// Right now, nothing has happened since we need to start the HTTP server on
	// our own. In the end, tusd will start listening on and accept request at
	// http://localhost:8080/files
	http.Handle("/files/", http.StripPrefix("/files/", handler))
	err = http.ListenAndServe(":8080", nil)
	if err != nil {
		panic(fmt.Errorf("Unable to listen: %s", err))
	}
}

Please consult the online documentation for more details about tusd's APIs and its sub-packages.

Implementing own storages

The tusd server is built to be as flexible as possible and to allow the use of different upload storage mechanisms. By default the tusd binary includes filestore which will save every upload to a specific directory on disk.

If you have different requirements, you can build your own storage backend which will save the files to S3, a remote FTP server or similar. Doing so is as simple as implementing the tusd.DataStore interface and using the new struct in the configuration object. Please consult the documentation about detailed information about the required methods.

Packages

This repository does not only contain the HTTP server's code but also other useful tools:

  • s3store: A storage backend using AWS S3
  • filestore: A storage backend using the local file system
  • gcsstore: A storage backend using Google cloud storage
  • memorylocker: An in-memory locker for handling concurrent uploads
  • consullocker: A locker using the distributed Consul service
  • limitedstore: A storage wrapper limiting the total used space for uploads

Running the testsuite

Build Status Build status

go test -v ./...

FAQ

How can I access tusd using HTTPS?

The tusd binary, once executed, listens on the provided port for only non-encrypted HTTP requests and does not accept HTTPS connections. This decision has been made to limit the functionality inside this repository which has to be developed, tested and maintained. If you want to send requests to tusd in a secure fashion - what we absolutely encourage, we recommend you to utilize a reverse proxy in front of tusd which accepts incoming HTTPS connections and forwards them to tusd using plain HTTP. More information about this topic, including sample configurations for Nginx and Apache, can be found in issue #86 and in the Apache example configuration.

Can I run tusd behind a reverse proxy?

Yes, it is absolutely possible to do so. Firstly, you should execute the tusd binary using the -behind-proxy flag indicating it to pay attention to special headers which are only relevent when used in conjunction with a proxy. Furthermore, there are addtional details which should be kept in mind, depending on the used software:

  • Disable request buffering. Nginx, for example, reads the entire incoming HTTP request, including its body, before sending it to the backend, by default. This behavior defeats the purpose of resumability where an upload is processed while it's being transfered. Therefore, such as feature should be disabled.

  • Adjust maximum request size. Some proxies have default values for how big a request may be in order to protect your services. Be sure to check these settings to match the requirements of your application.

  • Forward hostname and scheme. If the proxy rewrites the request URL, the tusd server does not know the original URL which was used to reach the proxy. This behavior can lead to situations, where tusd returns a redirect to a URL which can not be reached by the client. To avoid this confusion, you can explicitly tell tusd which hostname and scheme to use by supplying the X-Forwarded-Host and X-Forwarded-Proto headers.

Explicit examples for the above points can be found in the Nginx configuration which is used to power the master.tus.io instace.

Can I run custom verification/authentication checks before an upload begins?

Yes, this is made possible by the hook system inside the tusd binary. It enables custom routines to be executed when certain events occurs, such as a new upload being created which can be handled by the pre-create hook. Inside the corresponding hook file, you can run your own validations against the provided upload metadata to determine whether the action is actually allowed or should be rejected by tusd. Please have a look at the corresponding documentation for a more detailed explanation.

License

This project is licensed under the MIT license, see LICENSE.txt.