Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Software distribution by way of S3 signed URLs
Haskell Python Shell
Pull request Compare This branch is 75 commits ahead of airbnb:master.
Fetching latest commit...
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.



       SSSP is an HTTP proxy for S3 that can generate short-lived, signed URLs
       for stored objects. By providing a server separate from S3 that can  be
       placed  behind an authenticating proxy or firewall, SSSP allows a vari-
       ety of common security mechanisms to be used  to  limit  access  to  S3
       objects over HTTP while taking advantage of S3's considerable bandwidth
       and parallelism.

       Use-cases for SSSP include:

          o sharing of large files within an organization,

          o media service for public facing web applications,

          o distribution of internal software.

       SSSP supports configuration via environment variables or STDIN.

       These settings can be passed as environment variables  or  fed  to  the
       server  on  STDIN in colon separated format. Both the new and old forms
       of  the  AWS  credential  environment  variables  are  supported.   The
       AWS_DEFAULT_REGION  variable, shared with AWS Python CLI tools, is sup-
       ported as well. The AWS_REGION is an old, SSSP specific version of  the
       same functionality.

       # AWS Settings
       AWS_ACCESS_KEY              = account access key
       AWS_ACCESS_KEY_ID           = account access key
       AWS_SECRET_KEY              = secret
       AWS_SECRET_ACCESS_KEY       = secret
       AWS_DEFAULT_REGION          = eu-west-1, classic, us-east-1, ...
       AWS_REGION                  = eu-west-1, classic, us-east-1, ...

       # Storage settings
       SSSP_BUCKET                 = DNS friendly bucket name

       # Server settings
       SSSP_CONN                   = <ip>:<port> pair
       PORT                        = port to connect to, on localhost

       SSSP  is  fairly  liberal when parsing STDIN. In fact, Bourne shell .rc
       files, like the follow example, are parsed without error:

       export SSSP_BUCKET=dist
       export SSSP_CONN=*:6000

       However, SSSP skips over lines that contain quotes ("') or that  appear
       to  require  shell interpolation for their correct interpolation (lines
       containing $`{}).

       URLs in SSSP point to one of two objects: an item or a  listing.  Items
       correspond  to  S3  objects;  a  GET retrieves a signed redirect to the
       object.  Listings are a sequence of URLs, in  ascending  order;  a  GET
       retrieves the listing as a plaintext document, one URL per line.

       GET         # Signed for the default time (10s).
       GET     # Signed for n seconds.

       A  PUT  to  an item sets the item's content. DELETEs can be singular or
       plural. A plural DELETE removes only the objects generated by  a  list-

       URLs  are  divided syntactically in to listings and items. A URL ending
       with a slash is always a listing.

       GET   # Signed redirect to an object called dist.
       GET  # Listing of items below the key `dist'.

       To make it easier to work with versioned or  timestamped  assets,  SSSP
       supports the @hi and @lo meta-paths. These correspond to the names that
       sort highest and lowest  according  to  semantic  version  sort,  where
       non-digit chars serve to delimit arrays of numbers. For common forms of
       dates, these have the same effect as ASCII sort. (ASCII sort may speci-
       fied, as well; please the section WILDCARDS, below.)


       # Retrieval with @hi and @lo.
       GET  -307->
       GET  -307->

       Wildcards  @hi  and  @lo used together with a count specify a set wild-
       card; the result is a listing:

       GET  -200->  dist/x/x-0.1.1.tgz

       Counts are the natural numbers starting at 0. The wildcard @* refers to
       "all the items".

       A counted wildcard, like @hi2, can be suffixed with a tilde to form its
       complement -- so @hi2~ is everything but the highest  two  items.  This
       can be useful for bulk deletion of old/new things.

       Wildcards can also be used with a string prefix and postfix, e.g.


       Predictably files not matching the prefix or postfix will not be

       If a set wildcard (a * or a counted wildcard) is followed by a *,
       the string following said wildcard will be used as a postfix
       regardless of its content and SSSP will match *any* file with a
       path that matches the pattern preceding the wildcard and a
       postfix equal to the one following the wildcard. The matched
       files will be returned as a listing.

       For example


       Will match and return a list of .tgz file.

          @hi.semver, @lo.semver
                 Key  with  highest or lowest version, according to a liberal-
                 ized form of "semantic versioning", where version  components
                 are delimited by any non-digit characters.

          @hi.ascii, @lo.ascii
                 Keys sorted ASCIIbetically, in the C locale (sorted purely by
                 byte value).

          @hi, @lo
                 The default sort, which is semantic version sort.

          @*, @*.semver, @*.ascii
                 All the items, in the default order (semantic version) or  in
                 a specified order.

       ASCII  sort  can be substantially more performant than semantic version
       sort, because S3 returns data in ASCII order and thus no  real  sorting
       is necessary.

       # Start web application.
       sssp < conf

       # Start web application with configuration provided by the environment.
       export AWS_ACCESS_KEY_ID=...
       export AWS_SECRET_ACCESS_KEY=...
       sssp <<CONF
       SSSP_BUCKET: dist

       Listing  results  should really be URLs. The time to sign should really
       be configurable; or at least settable with a query parameter.

Something went wrong with that request. Please try again.