Skip to content

Commit

Permalink
Reverted the pulling out of various middleware:
Browse files Browse the repository at this point in the history
RateLimit
StaticWeb
TempURL/FormPOST

Change-Id: I988e93e6f4aacb817a2e354d43a04e47516fdf88
  • Loading branch information
gholt committed May 16, 2012
1 parent 3d3ed34 commit 1c3b75c
Show file tree
Hide file tree
Showing 18 changed files with 5,497 additions and 6 deletions.
70 changes: 70 additions & 0 deletions bin/swift-form-signature
@@ -0,0 +1,70 @@
#!/usr/bin/env python

import hmac
from hashlib import sha1
from os.path import basename
from sys import argv, exit
from time import time


if __name__ == '__main__':
if len(argv) != 7:
prog = basename(argv[0])
print 'Syntax: %s <path> <redirect> <max_file_size> ' \
'<max_file_count> <seconds> <key>' % prog
print
print 'Where:'
print ' <path> The prefix to use for form uploaded'
print ' objects. For example:'
print ' /v1/account/container/object_prefix_ would'
print ' ensure all form uploads have that path'
print ' prepended to the browser-given file name.'
print ' <redirect> The URL to redirect the browser to after'
print ' the uploads have completed.'
print ' <max_file_size> The maximum file size per file uploaded.'
print ' <max_file_count> The maximum number of uploaded files'
print ' allowed.'
print ' <seconds> The number of seconds from now to allow'
print ' the form post to begin.'
print ' <key> The X-Account-Meta-Temp-URL-Key for the'
print ' account.'
print
print 'Example output:'
print ' Expires: 1323842228'
print ' Signature: 18de97e47345a82c4dbfb3b06a640dbb'
exit(1)
path, redirect, max_file_size, max_file_count, seconds, key = argv[1:]
try:
max_file_size = int(max_file_size)
except ValueError:
max_file_size = -1
if max_file_size < 0:
print 'Please use a <max_file_size> value greater than or equal to 0.'
exit(1)
try:
max_file_count = int(max_file_count)
except ValueError:
max_file_count = 0
if max_file_count < 1:
print 'Please use a positive <max_file_count> value.'
exit(1)
try:
expires = int(time() + int(seconds))
except ValueError:
expires = 0
if expires < 1:
print 'Please use a positive <seconds> value.'
exit(1)
parts = path.split('/', 4)
# Must be four parts, ['', 'v1', 'a', 'c'], must be a v1 request, have
# account and container values, and optionally have an object prefix.
if len(parts) < 4 or parts[0] or parts[1] != 'v1' or not parts[2] or \
not parts[3]:
print '<path> must point to a container at least.'
print 'For example: /v1/account/container'
print ' Or: /v1/account/container/object_prefix'
exit(1)
sig = hmac.new(key, '%s\n%s\n%s\n%s\n%s' % (path, redirect, max_file_size,
max_file_count, expires), sha1).hexdigest()
print ' Expires:', expires
print 'Signature:', sig
59 changes: 59 additions & 0 deletions bin/swift-temp-url
@@ -0,0 +1,59 @@
#!/usr/bin/env python

import hmac
from hashlib import sha1
from os.path import basename
from sys import argv, exit
from time import time


if __name__ == '__main__':
if len(argv) != 5:
prog = basename(argv[0])
print 'Syntax: %s <method> <seconds> <path> <key>' % prog
print
print 'Where:'
print ' <method> The method to allow, GET or PUT.'
print ' Note: HEAD will also be allowed.'
print ' <seconds> The number of seconds from now to allow requests.'
print ' <path> The full path to the resource.'
print ' Example: /v1/AUTH_account/c/o'
print ' <key> The X-Account-Meta-Temp-URL-Key for the account.'
print
print 'Example output:'
print ' /v1/AUTH_account/c/o?temp_url_sig=34d49efc32fe6e3082e411e' \
'eeb85bd8a&temp_url_expires=1323482948'
print
print 'This can be used to form a URL to give out for the access '
print 'allowed. For example:'
print ' echo https://swift-cluster.example.com`%s GET 60 ' \
'/v1/AUTH_account/c/o mykey`' % prog
print
print 'Might output:'
print ' https://swift-cluster.example.com/v1/AUTH_account/c/o?' \
'temp_url_sig=34d49efc32fe6e3082e411eeeb85bd8a&' \
'temp_url_expires=1323482948'
exit(1)
method, seconds, path, key = argv[1:]
if method not in ('GET', 'PUT'):
print 'Please use either the GET or PUT method.'
exit(1)
try:
expires = int(time() + int(seconds))
except ValueError:
expires = 0
if expires < 1:
print 'Please use a positive <seconds> value.'
exit(1)
parts = path.split('/', 4)
# Must be five parts, ['', 'v1', 'a', 'c', 'o'], must be a v1 request, have
# account, container, and object values, and the object value can't just
# have '/'s.
if len(parts) != 5 or parts[0] or parts[1] != 'v1' or not parts[2] or \
not parts[3] or not parts[4].strip('/'):
print '<path> must point to an object.'
print 'For example: /v1/account/container/object'
exit(1)
sig = hmac.new(key, '%s\n%s\n%s' % (method, expires, path),
sha1).hexdigest()
print '%s?temp_url_sig=%s&temp_url_expires=%s' % (path, sig, expires)
140 changes: 139 additions & 1 deletion doc/manpages/proxy-server.conf.5
Expand Up @@ -90,7 +90,7 @@ are acceptable within this section.
.IP "\fBpipeline\fR"
It is used when you need apply a number of filters. It is a list of filters
ended by an application. The default should be \fB"catch_errors healthcheck
cache tempauth proxy-server"\fR
cache ratelimit tempauth proxy-server"\fR
.RE
.PD

Expand Down Expand Up @@ -209,6 +209,53 @@ Default for memcache_servers is to try to read the property from /etc/swift/memc



.RS 0
.IP "\fB[filter:ratelimit]\fR"
.RE

Rate limits requests on both an Account and Container level. Limits are configurable.

.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the ratelimit middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#ratelimit\fR.
.IP "\fBset log_name\fR"
Label used when logging. The default is ratelimit.
.IP "\fBset log_facility\fR"
Syslog log facility. The default is LOG_LOCAL0.
.IP "\fBset log_level\fR "
Logging level. The default is INFO.
.IP "\fBset log_headers\fR "
Enables the ability to log request headers. The default is False.
.IP \fBclock_accuracy\fR
This should represent how accurate the proxy servers' system clocks are with each other.
1000 means that all the proxies' clock are accurate to each other within 1 millisecond.
No ratelimit should be higher than the clock accuracy. The default is 1000.
.IP \fBmax_sleep_time_seconds\fR
App will immediately return a 498 response if the necessary sleep time ever exceeds
the given max_sleep_time_seconds. The default is 60 seconds.
.IP \fBlog_sleep_time_seconds\fR
To allow visibility into rate limiting set this value > 0 and all sleeps greater than
the number will be logged. If set to 0 means disabled. The default is 0.
.IP \fBrate_buffer_seconds\fR
Number of seconds the rate counter can drop and be allowed to catch up
(at a faster than listed rate). A larger number will result in larger spikes in
rate but better average accuracy. The default is 5.
.IP \fBaccount_ratelimit\fR
If set, will limit PUT and DELETE requests to /account_name/container_name. Number is
in requests per second. If set to 0 means disabled. The default is 0.
.IP \fBaccount_whitelist\fR
Comma separated lists of account names that will not be rate limited. The default is ''.
.IP \fBaccount_blacklist\fR
Comma separated lists of account names that will not be allowed. Returns a 497 response.
The default is ''.
.IP \fBcontainer_ratelimit_size\fR
When set with container_limit_x = r: for containers of size x, limit requests per second
to r. Will limit PUT, DELETE, and POST requests to /a/c/o. The default is ''.
.RE



.RS 0
.IP "\fB[filter:catch_errors]\fR"
.RE
Expand All @@ -228,6 +275,97 @@ Enables the ability to log request headers. The default is False.



.RS 0
.IP "\fB[filter:cname_lookup]\fR"
.RE

Note: this middleware requires python-dnspython

.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the cname_lookup middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#cname_lookup\fR.
.IP "\fBset log_name\fR"
Label used when logging. The default is cname_lookup.
.IP "\fBset log_facility\fR"
Syslog log facility. The default is LOG_LOCAL0.
.IP "\fBset log_level\fR "
Logging level. The default is INFO.
.IP "\fBset log_headers\fR"
Enables the ability to log request headers. The default is False.
.IP \fBstorage_domain\fR
The domain to be used by the middleware.
.IP \fBlookup_depth\fR
How deep in the CNAME chain to look for something that matches the storage domain.
The default is 1.
.RE



.RS 0
.IP "\fB[filter:staticweb]\fR"
.RE

Note: Put staticweb just after your auth filter(s) in the pipeline

.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the staticweb middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#staticweb\fR.
.IP \fBcache_timeout\fR
Seconds to cache container x-container-meta-web-* header values. The default is 300 seconds.
.IP "\fBset log_name\fR"
Label used when logging. The default is staticweb.
.IP "\fBset log_facility\fR"
Syslog log facility. The default is LOG_LOCAL0.
.IP "\fBset log_level\fR "
Logging level. The default is INFO.
.IP "\fBset log_headers\fR"
Enables the ability to log request headers. The default is False.
.IP "\fBset access_log_name\fR"
Label used when logging. The default is staticweb.
.IP "\fBset access_log_facility\fR"
Syslog log facility. The default is LOG_LOCAL0.
.IP "\fBset access_log_level\fR "
Logging level. The default is INFO.
.RE



.RS 0
.IP "\fB[filter:tempurl]\fR"
.RE

Note: Put tempurl just before your auth filter(s) in the pipeline

.RS 3
.IP \fBincoming_remove_headers\fR
The headers to remove from incoming requests. Simply a whitespace delimited list of header names and names can optionally end with '*' to indicate a prefix match. incoming_allow_headers is a list of exceptions to these removals.
.IP \fBincoming_allow_headers\fR
The headers allowed as exceptions to incoming_remove_headers. Simply a whitespace delimited list of header names and names can optionally end with '*' to indicate a prefix match.
.IP "\fBoutgoing_remove_headers\fR"
The headers to remove from outgoing responses. Simply a whitespace delimited list of header names and names can optionally end with '*' to indicate a prefix match. outgoing_allow_headers is a list of exceptions to these removals.
.IP "\fBoutgoing_allow_headers\fR"
The headers allowed as exceptions to outgoing_remove_headers. Simply a whitespace delimited list of header names and names can optionally end with '*' to indicate a prefix match.
.IP "\fBset log_level\fR "
.RE



.RS 0
.IP "\fB[filter:formpost]\fR"
.RE

Note: Put formpost just before your auth filter(s) in the pipeline

.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the formpost middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#formpost\fR.
.RE



.RS 0
.IP "\fB[filter:name_check]\fR"
.RE
Expand Down
4 changes: 0 additions & 4 deletions doc/source/associated_projects.rst
Expand Up @@ -52,8 +52,4 @@ Content Distribution Network Integration
Other
-----

* `Domain Remap <https://github.com/notmyname/swift-domainremap>`_ - Translates subdomains on the Host header to path elements that are appropriate for swift.
* `Glance <https://github.com/openstack/glance>`_ - Provides services for discovering, registering, and retrieving virtual machine images (for OpenStack Compute [Nova], for example).
* `Rate Limit <https://github.com/dpgoetz/swift-ratelimit>`_ - Enforces limits on the request rates to accounts and containers.
* `StaticWeb <http://gholt.github.com/swift-staticweb/>`_ - Allows serving static websites from Swift containers using ACLs and other metadata on those containers.
* `TempURL/FormPOST <http://gholt.github.com/swift-tempurl/>`_ - Temporary, Expiring URLs and Form POSTing middleware.
1 change: 1 addition & 0 deletions doc/source/index.rst
Expand Up @@ -47,6 +47,7 @@ Overview and Concepts
overview_reaper
overview_auth
overview_replication
ratelimit
overview_large_objects
overview_object_versioning
overview_container_sync
Expand Down
42 changes: 42 additions & 0 deletions doc/source/misc.rst
Expand Up @@ -133,9 +133,51 @@ Manager
:members:
:show-inheritance:

Ratelimit
=========

.. automodule:: swift.common.middleware.ratelimit
:members:
:show-inheritance:

Swift3
======

.. automodule:: swift.common.middleware.swift3
:members:
:show-inheritance:

StaticWeb
=========

.. automodule:: swift.common.middleware.staticweb
:members:
:show-inheritance:

TempURL
=======

.. automodule:: swift.common.middleware.tempurl
:members:
:show-inheritance:

FormPost
========

.. automodule:: swift.common.middleware.formpost
:members:
:show-inheritance:

Domain Remap
============

.. automodule:: swift.common.middleware.domain_remap
:members:
:show-inheritance:

CNAME Lookup
============

.. automodule:: swift.common.middleware.cname_lookup
:members:
:show-inheritance:

0 comments on commit 1c3b75c

Please sign in to comment.