Skip to content

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also .

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also .
This comparison is big! We’re only showing the most recent 250 commits
Commits on Nov 16, 2012
Jenkins Merge "Refactor DiskFile to hide temp file names and exts" d13869e
Commits on Nov 17, 2012
@smerritt smerritt Remove IP-based container-sync ACLs from auth middlewares.
The determination of the client IP looked at the X-Cluster-Client-Ip
and X-Forwarded-For headers in the incoming HTTP request. This is
trivially spoofable by a malicious client, so there's no security
gained by having the check there.

Worse, having the check there provides a false sense of security to
cluster operators. It sounds like it's based on the client IP, so an
attacker would have to do IP spoofing to defeat it. However, it's
really just a shared secret, and there's already a secret key set
up. Basically, it looks like 2-factor auth (IP+key), but it's really
1-factor (key).

Now, the one case where this might provide some security is where the
Swift cluster is behind an external load balancer that strips off the
X-Cluster-Client-Ip and X-Forwarded-For headers and substitutes its
own. I don't think it's worth the tradeoff, hence this commit.

Fixes bug 1068420 for very small values of "fixes".

DocImpact

Change-Id: I2bef64c2e1e4df8a612a5531a35721202deb6964
357b12d
Commits on Nov 19, 2012
Jenkins Merge "Fix lazy-listing of object segments." cb55f0c
@dhersam dhersam Fix for container sync not syncing last rowid
Bug 1079439

Change-Id: I1985a7176f34931ecb0f24c0289f18df5e934598
923b8b9
Commits on Nov 20, 2012
@notmyname notmyname pep8 utils.py
Change-Id: I0466fd2a462e94e5a1e0f88adf901de9f5ac131d
f74ef1e
@notmyname notmyname pep8 direct client
Change-Id: I1ed6afe4336c238044174c93ae1e0078fbcef971
3aab248
@notmyname notmyname pep8 db.py
Change-Id: I2ae277b6f435f56549560684697486b56d24d46d
c46033a
@notmyname notmyname pep8 middleware
Change-Id: Ieef65f05e66b12347752c0e02648858e3242d8e7
1f7be20
Commits on Nov 21, 2012
Jenkins Merge "pep8 middleware" 2fc9716
@smerritt smerritt Add CONTRIBUTING file.
If the CONTRIBUTING[.md] file exists, Github will show a link to it to
anyone who files an issue on Github or opens a pull request on
Github. We don't want people to do that, so this file points people at
the OpenStack wiki page with instructions on how to contribute
properly. This should cut down on the number of pull requests and
Github issues that we then have to spend our valuable time ignoring.

See also <https://github.com/blog/1184-contributing-guidelines>.

Change-Id: Icd23b65c642c5ae748ca1f7f397e2c8d63173492
fdf55c2
@smerritt smerritt Improve container-sync docs.
Two improvements: first, document that the container-sync process
connects to the remote cluster's proxy server, so outbound
connectivity is required.

Second, rewrite the behind-the-scenes container-sync example and add
some ASCII-art diagrams.

Fixes bug 1068430.

Bonus fix of docstring in wsgi.py to squelch a sphinx warning.

Change-Id: I85bd56c2bd14431e13f7c57a43852777f14014fb
89a871d
Commits on Nov 26, 2012
Jenkins Merge "pep8 direct client" 67201c3
Jenkins Merge "pep8 db.py" 2ad23a2
Commits on Nov 27, 2012
@smerritt smerritt Upgrade pep8 to 1.3.3.
This required a bunch of whitespace-poking of the scripts in bin, but
that's all. Now every file in swift/ and bin/ is pep8-1.3.3-compliant,
so hopefully we can be done with this pep8 stuff for a good long time.

Change-Id: I44fdb41d219c57400a4c396ab7eb0ffa9dcd8db8
35f4d29
@jajohnson jajohnson Replace hard-coded test accounts with user-configured values
Change-Id: I824ba5f231e252b923abc895f757137855a93d3a
48ebd67
@notmyname notmyname added GoLang binding and CDMI to associated projects
Change-Id: I720a0e0e886d25eec52556e8ce93bf9c49f6b452
46f26b3
Jenkins Merge "added GoLang binding and CDMI to associated projects" 0b00d0d
Commits on Nov 28, 2012
@gholt gholt Fix bug with swob.Request.path_info_pop
path_info_pop didn't behave as the webob one did with single segment
paths like /one and with root-only paths /

Now it should.

Change-Id: Ib88344de386ab9e8975e7f48c1afc47731992ee2
4063123
Jenkins Merge "Upgrade pep8 to 1.3.3." da8578f
@dprince dprince Updates to use new keystoneclient middleware.
Updates the proxy-server.conf-sample and docs to use
the new Keystoneclient middleware class name.

Change-Id: I3727f7b7328a2513347b8ef257c270126df36d7b
abcd872
Jenkins Merge "Fix bug with swob.Request.path_info_pop" 0242985
Commits on Nov 29, 2012
Jenkins Merge "Fix for container sync not syncing last rowid" a830b6f
Jenkins Merge "Updates to use new keystoneclient middleware." 871f552
@gholt gholt Added --top option to swift-recon -d
When showing the disk usage dispersion graph it is often useful to
know what those top full drives are so you can do further research.
Now you can run 'swift-recon -d --top 10' to list the top ten devices
by fullness after the usual dispersion graph.

Change-Id: I7ddb2141e55e0613f69750fabe544940192c2d48
18f755a
Commits on Nov 30, 2012
@gholt gholt Swob bugfixes; for ? in names specifically
It was discovered that uploading items with ? in their names (encoded
with %3F of course) made Swob fail in that it trimmed off everything
after the ? as if it were a query string.

Change-Id: Ie686db9a2177aafad2e77c307ffc3f446646fbb5
6743e4d
@guangyee guangyee bp/cross-tenant-acls: allow tenantId:user, tenantName:user, and *:use…
…r ALCs

Change-Id: I7cfe77b3f03172814814f2e2bae04a3ae184efb0
795acd4
Commits on Dec 01, 2012
Jenkins Merge "Improve container-sync docs." 4ac6904
Commits on Dec 03, 2012
@redbo redbo 406 if we can't satisfy Accept
The container and account servers should respond with 406 if the Accept header
isn't satisfiable.  This behavior is defined in RFC 2616 section 14.1.

Change-Id: I8a67ccafe33dc70ef4f7794686a54fbc8581f4dc
064ee2b
Jenkins Merge "Added --top option to swift-recon -d" 4f617f4
Jenkins Merge "406 if we can't satisfy Accept" 8a6922b
Commits on Dec 04, 2012
@jkff jkff drop_privileges should also set HOME
E.g. if HOME is not set, swift-proxy will create the
keystone_signing file not in HOME but in /root.
This is because the swift user doesn't have a shell
in /etc/passwd and so it doesn't set environment variables
when impersonating.

Change-Id: I3013007e0dadf6ddccc176e142b7c78c5d63a351
772da0a
@portante portante Ignore pycscope files
Change-Id: I5fe38f28896d086e56ba9e718b02227d87dd4a2d
Signed-off-by: Peter Portante <peter.portante@redhat.com>
37da61b
@dbishop dbishop Allow optional, temporary healthcheck failure.
A deployer may want to remove a Swift node from a load balancer for
maintenance or upgrade.  This patch provides an optional mechanism for
this.  The healthcheck filter config can specify "disable_path" which is
a filesystem path.  If a file is present at that location, the
healthcheck middleware returns a 503 with a body of "DISABLED BY FILE".

So a deployer can configure "disable_path" and then touch that
filesystem path, wait for the proxy to be removed from the load balancer
pool, perform maintenance/upgrade, and then remove the "disable_path"
file.

Also cleaned up the conf file man pages a bit.

Change-Id: I1759c78c74910a54c720f298d4d8e6fa57a4dab4
b8e3e9e
Jenkins Merge "Allow optional, temporary healthcheck failure." db6d62a
Jenkins Merge "drop_privileges should also set HOME" 7e8959f
Commits on Dec 05, 2012
Jenkins Merge "bp/cross-tenant-acls: allow tenantId:user, tenantName:user, an…
…d *:user ALCs"
13937ad
Commits on Dec 07, 2012
@clayg clayg capture resetswift output in probetests
This change makes the dots prettier during probetests

When calling the resetswift script, the probetests will use subprocess
to redirect stderr to stdout and capture stdout into a buffer.  We print
the captured buffer from resetswift's combined stdout/stderr and let
nosetests stdout capturing handle printing the output for debug only if a
test fails.

Change-Id: I022512f2ef5a4c43b0e49264bad1bca98c1f0299
b852f22
@clayg clayg Allow dot test runners from any dir
Currently the .*test files in the source tree root seem to assume you're
in that directory when you run them.  This change should work exactly
the same except for removing that restriction.

Change-Id: I9b037016490dd6d5b515cb2c3cad2f99ee40bb6d
1f83f00
@clayg clayg Add config of server start timeouts for probetests
Currently the timeout for a wsgi server successfully binding to a port
and for a probetest background service to finish starting are hard coded
to 30 seconds.  While a reasonable default for most configurations, a
small virtualized environment may need a little more time in order for
probe tests to complete successfully.

This patch adds a 'bind_timeout' option to the DEFAULT section of the
main wsgi servers' config.  Also a new [probe_test] section and
'check_server_timeout' option to test.conf

DocImpact

Change-Id: Ibcaff153c7633bbf32e460fd9dbf04932eddb56f
3a70112
Commits on Dec 10, 2012
@pandemicsyn pandemicsyn Add dispersion report flags to limit reports
- Add two optional flags that let you limit swift-dispersion-report to only
reporting on containers OR objects.
- Also make dispersion.conf and swift-dispersion-report manpages
  current.

DocImpact

Change-Id: Iad56133cad261241db27d0e2103098e3c2f3c245
e474dfb
@creiht creiht Add config option to turn eventlet debug on/off
By default, this will be turned off.  This will cause eventlet to not
print stack traces to stderr which can be very annoying on production
systems.  It is still recommended to turn it on for development or
debuging purposes.

DocImpact
Change-Id: I5e5b902d3d9ed85f784549e53f2ee2fc87cbe2e5
4c6a354
Jenkins Merge "Add dispersion report flags to limit reports" 067335a
Jenkins Merge "Add config of server start timeouts for probetests" 1619cee
Jenkins Merge "capture resetswift output in probetests" a6ee9b4
Jenkins Merge "Add config option to turn eventlet debug on/off" 8b770aa
Commits on Dec 11, 2012
Jenkins Merge "Allow dot test runners from any dir" 1135c9e
@dbishop dbishop Avoid infinite recursion in swift.obj.replicator.get_hashes.
Fixes bug 1089140.

Turns out that if an exception bails out of the pickle loading (eg.
zero-byte hahes_file), the if clause to determine whether or not to
write out a fresh hashes_file can evaluate to false, leading to an
infinite loop.

This patch fixes this infinite loop generally, by ensuring that if any
exception is thrown, a new hashes_file is written.

Change-Id: I344c5f8e261ce7c667bdafe1687263a4150b21dc
ea95d00
Commits on Dec 12, 2012
@jd jd Use install_requires in setup.py
Using install_requires makes sure that if some code is using Swift internal
and wants to require Swift, all its dependencies have been already pulled.
This is really useful on test automation for example, otherwise, code might
be failing because on of the module listed in pip-requires is not instaled.

Also, we change the pip-requires files to use >= rather than ==, so the
requirements are easily fulfiled with distribution provided packages.

Change-Id: I65814bcd8ce798da21a5c17b4d5916a23f59e962
Signed-off-by: Julien Danjou <julien@danjou.info>
dcff5ff
@creiht creiht Fix needed for keystone middleware logging
This is required for a fix in keystone middleware to log to the proxy
logs.  Since the keystone middleware doesn't initialize the log with
swift.commong.utils.config_logger, it doesn't have the server and
transaction id setup.  This is required to fix bug #1089664

Change-Id: Icbb44076495a4d56065445c7a4d972ebede5d1e0
a064fa4
Commits on Dec 13, 2012
@saschpe saschpe Add a license header to functionalnosetests
Change-Id: I413d8bb9cbdea1f67ab2649ab0cc4a05a7a65883
08a110b
Commits on Dec 14, 2012
@clayg clayg document correct config in dispersion-report help
The --help message of swift-dispersion-report used to say
/etc/swift/stats.conf instead of the correct and consistent
/etc/swift/dispersion.conf - this change updates that
commandline help message.

Change-Id: I69ad64d31bb86eb0d36fcf5b17aa8bf42f646ed1
3dbc1a9
@gholt gholt Fix permissions from last commit
Change-Id: I3bad8c2995d4daf1e7798fae2c59b409608cbc45
9302c5f
Jenkins Merge "Fix permissions from last commit" 2ed1d3b
Commits on Dec 15, 2012
Jenkins Merge "Fix needed for keystone middleware logging" 7f60997
Commits on Dec 17, 2012
@dpgoetz dpgoetz one dot, 5% increase in coverage
Change-Id: I1a2d603be983aa3291d22f7258437ee264e6c559
2a9bf20
@gholt gholt Added override for swift-init's KILL_WAIT
You can now give swift-init a -k N (or --kill-wait N) option to
override the default 15 second wait for a process to die after
sending it the die signal. This is useful for boxes that are awfully
slow for whatever reason.

DocImpact

Change-Id: I328ec254f6e0ee1cd423c1d062ba4c5331bd8337
4fcbeff
Commits on Dec 18, 2012
@gholt gholt Made 507s report drive, if known.
This functionality was lost with the swob change, but is back now.

Change-Id: I13b3154080a7c601235711b274e4899efb6adc93
52a2a65
Jenkins Merge "Made 507s report drive, if known." c840bd4
@wu-wenxiang wu-wenxiang Refactor Ring::get_nodes() and Ring::get_part_nodes()
Add a _get_part_nodes() method in Ring class, then call it in Ring::get_nodes() and Ring::get_part_nodes().

In order to reduce duplicate codes.

Change-Id: Id92f8dfd08388305638a2ba2ab64e77545c14cba
b85f69e
Jenkins Merge "Use install_requires in setup.py" 9543a71
Jenkins Merge "Refactor Ring::get_nodes() and Ring::get_part_nodes()" 74038dc
Commits on Dec 19, 2012
@dprince dprince Convert LICENSE to use unix style line endings.
Ran dos2unix on LICENSE to convert to unit style line endings.
This makes building packages which contain the LICENSE file
a bit nicer... and matches other LICENSE files under the openstack
umbrella which have unix style line endings as well.

Change-Id: Id724d1ba402a590725a2d200f5e0599bce696e5a
97f79b9
@redbo redbo suggest fallocate instead of dd in saio
Change-Id: I4d5ba8e58216749c2858dc1c0b8225b3619a7287
e81ca8d
@creiht creiht Fixes console logging with non-swift middleware
Fixes logging if logging to the console with non-swift middleware (such
as keystone) enabled.  This also fixes issues with swift in devstack

Change-Id: Ib8b691b62b657a6d4ecdb1648d1fc2f3a0479982
53ed90b
Commits on Dec 20, 2012
@jd jd Move InputProxy to utils
This class is being used at least by Ceilometer in its Swift middleware, and
since it's a general one anyway, it looks good to move it to common.utils.

This is a follow-up to Chmouel suggestion in
https://review.openstack.org/#/c/18231

Change-Id: I8d0ed8600c4152b91be9a88a3b396c3967d0add2
Signed-off-by: Julien Danjou <julien@danjou.info>
6df2854
@redbo redbo Revert "suggest fallocate instead of dd in saio"
Change-Id: I3742c4580d920e09de434e86b75b134c62f1732e
5d5b262
Jenkins Merge "Fixes console logging with non-swift middleware" 032f448
Jenkins Merge "Revert "suggest fallocate instead of dd in saio"" c1964e5
Commits on Dec 21, 2012
@redbo redbo saio truncate instead of dd (for reals)
Changing that complex dd to a functionally equivalent truncate that's easier
to read and modify.

DocImpact

Change-Id: I64404318364608e62a1d80f6a0550271eb4cd03a
c37954d
Jenkins Merge "Move InputProxy to utils" 0d6fb2e
@gholt gholt Fix for subtle bug from 5c8f9c5
Change-Id: If2bd1d4a850936b2e575a96073c116a8b9522602
5f19ccf
Jenkins Merge "Fix for subtle bug from 5c8f9c5" 9c80093
Jenkins Merge "saio truncate instead of dd (for reals)" 6b8ae93
Commits on Jan 03, 2013
@notmyname notmyname Functional tests use x-auth-token rather than x-storage-token
This change allows other auth systems to test against the functests but not
require them to support x-storage-token (which was deprecated before swift
was even open-sourced). All other places in the code where x-storage-token is
used is only to populate x-auth-token if x-auth-token doesn't already exist.

Change-Id: Ie903d0f7135fc643353b74a61c14becf7dfc4d9d
dc8d0e0
@FlaPer87 FlaPer87 Check whether poll is available or not and let eventlet chose the best
hub when it isn't.

Change-Id: I5fb05a71e8c62d84cee1da9ea05f67ffbad18b9d
b155da4
Commits on Jan 04, 2013
Jenkins Merge "Check whether poll is available or not and let eventlet chose …
…the best hub when it isn't."
6e9b1a5
@dpgoetz dpgoetz Allow actual paths to work for swift-get-nodes
Change-Id: I29f9ab73c8ca2a3c52247fb92ef951dd5b5246cb
ded5046
Commits on Jan 08, 2013
Jenkins Merge "Allow actual paths to work for swift-get-nodes" b8626f9
Commits on Jan 09, 2013
@corywright corywright consistency and grammatical tweaks
Change-Id: I0a1dc9e07f246db3e294f5c62f4ffbd3560b0afd
c69e130
Commits on Jan 10, 2013
@dbishop dbishop Fix HEAD request response when request not given to response.
If a middleware (swift3, I'm looking at you), doesn't pass a Request
object into the Response constructor, Response._response_iter cannot
know to send zero bytes in the body of the HEAD response.

This patch fixes this usage of swob by making Response.__call__
helpfully reify self.request from env if it wasn't already set by the
Response object's constructor.

This fixes a bug in swift3 + swob-enabled-Swift where HEAD requests to
swift3 resulted in a response with a body in violation of the relevant
RFC and confusing clients.

Thanks to kostecky for finding the bug and describing it accurately.

Change-Id: I2bdb098052b161e1cddf1e4e482ab4dfafeb18c0
e2929ec
@gholt gholt Fixed version req for netifaces to 0.5
Change-Id: Ic1c67ced0169658c9aa7f78ed09bf24863cc9fb6
46c5b09
Commits on Jan 12, 2013
@gholt gholt swift-recon: Added oldest and most recent repl
I've been doing this with cluster-wide log searches for far too long.
This adds support for reporting the oldest replication pass
completion as well as the most recent. This is quite useful for
finding those odd replicators that have hung up for some reason and
need intervention.

Change-Id: I7fd7260eca162d6b085f3e82aaa3cf90670f2d53
a88b412
@gholt gholt Reject names with NULL characters
Unfortunately, SQLite truncates strings with null characters.
Additionally, XML pretty much hates them too.

Change-Id: Id9a8eaa27b841db6350d6959c202d3e3d6462b35
592d895
@gholt gholt Fixed bug in object replicator
If the object replicator couldn't create a device's object directory
(due to permissions or whatever) it wouldn't do any work at all. This
fixes that.

Change-Id: I6a30439d036b29c9cfdb660428d13668e0dc8632
95d5cf8
Jenkins Merge "Reject names with NULL characters" 34bda97
Jenkins Merge "swift-recon: Added oldest and most recent repl" d904fd4
@chmouel chmouel Add container-sync to container-server.conf doc.
- In howto install multi nodes container-sync was not included in the
  config example.

Change-Id: I2682d47c5cae28ff321ba03d70dae2fea00c276f
0df6fe8
Commits on Jan 14, 2013
@dbishop dbishop Some optimizations for ring rebalancing.
For a part-power 18 ring with a small number of devices (12), this drops
"swift-ring-builder ... rebalance" from ~7.2 seconds to ~3.6s on OS X
with Python 2.7 (2.7 GHz i7 processor).

Under the profiler, that part-power 18 rebalance now spends only
slightly more time rebalancing the ring than it does writing out the
results.

I haven't done more extensive before/after comparison with larger
numbers of devices and larger part-power values...

Change-Id: I25735bc71da2f11617cc436d4f8e0c4f3f82bfec
0bba071
Hodong Hwang Make object-auditor to use one logger
This commit makes that AuditorWorker gets a logger from ObjectAuditor class
(instead of creating new one) so the auditor uses minimum unix-sockets.

Fixes: bug #1089215
Change-Id: Ia47d862cbe7505db821784b01fcce6f22196e79f
d46f90e
Jenkins Merge "Make object-auditor to use one logger" 586076f
Jenkins Merge "Add container-sync to container-server.conf doc." c9f9ac1
Jenkins Merge "Some optimizations for ring rebalancing." 7e62056
@smerritt smerritt Allow for multiple X-(Account|Container)-* headers.
When the number of account/container or container/object replicas are
different, Swift had a few misbehaviors. This commit fixes them.

* On an object PUT/POST/DELETE, if there were 3 object replicas and
  only 2 container replicas, then only 2 requests would be made to
  object servers. Now, 3 requests will be made, but the third won't
  have any X-Container-* headers in it.

* On an object PUT/POST/DELETE, if there were 3 object replicas and 4
  container replicas, then only 3/4 container servers would receive
  immediate updates; the fourth would be ignored. Now one of the
  object servers will receive multiple (comma-separated) values in the
  X-Container-* headers and it will attempt to contact both of them.

  One side effect is that multiple async_pendings may be written for
  updates to the same object. They'll have differing timestamps,
  though, so all but the newest will be deleted unread. To trigger
  this behavior, you have to have more container replicas than object
  replicas, 2 or more of the container servers must be down, and the
  headers sent to one object server must reference 2 or more down
  container servers; it's unlikely enough and the consequences are so
  minor that it didn't seem worth fixing.

The situation with account/containers is analogous, only without the
async_pendings.

Change-Id: I98bc2de93fb6b2346d6de1d764213d7563653e8d
6ff644b
@dpgoetz dpgoetz swob refactors needed for sos
Change-Id: I839c2af9e524f712e6fbeb8bf74d860af1d243e2
c8b28ba
Jenkins Merge "consistency and grammatical tweaks" eff1862
Jenkins Merge "Fixed bug in object replicator" d69509a
Commits on Jan 15, 2013
Jenkins Merge "swob refactors needed for sos" 4a909cf
Commits on Jan 16, 2013
Jenkins Merge "Fix HEAD request response when request not given to response." b6d48c6
@dpgoetz dpgoetz get swob.Request.blank to parse path
Change-Id: I3327c915b3b868bb1829103adb718632e58b1b4a
54b7594
Jenkins Merge "get swob.Request.blank to parse path" 0fce08a
Commits on Jan 18, 2013
@notmyname notmyname authors and changelog update for swift 1.7.6
Change-Id: Iad00e51fb17c9e829bd5ba7b56a0d24253cfe8b2
d87ce03
@notmyname notmyname version bump to 1.7.7-dev
I expect the next release to actually be 1.8, but to prevent issues
if we decide to release 1.7.7, I'm only moving the rev number.

Change-Id: Ia9e05c5f036b8fbf67972470f240e84f7595223a
633d1aa
Jenkins Merge "version bump to 1.7.7-dev" 6c5fc3c
Commits on Jan 19, 2013
@dbishop dbishop Make statsd sample rate behave better.
As Dieter pointed out in bug 1090495
(https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics
can vary wildly between StatsD metrics.

This patch implements a partial solution by reducing the sample_rate
used for known high-volume metrics (operational experience will need to
inform this over time) and introducing a new tunable,
log_statsd_sample_rate_factor which is multiplied by the sample_rate for
every statsd stat.  This tunable can be used to reduce StatsD traffic
proportionally for all metrics and is intended to replace
log_statsd_default_sample_rate, which is left alone for
backward-compatibility, should anyone be using it.

This patch also includes a drive-by fix for log_udp_port which wasn't
being converted to an int (I didn't verify that actually causes trouble
in SysLogHandler(), but it's definitely an improvement regardles).

Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
8801b74
Commits on Jan 21, 2013
@chmouel chmouel Add --run-dir switch to swift-init
- Add ability to specify an alternative run_dir than the default
  /var/run/swift.
- DocImpact

Change-Id: I17677588f2c8da563b7fec2dc4fdc52da87126ed
3282e38
Jenkins Merge "Add --run-dir switch to swift-init" 3814f9f
@smerritt smerritt Fix superfluous GET requests in swift-bench.
If you specified num_gets = 0 for a benchmarking run (say, if you're
benchmarking PUT rate), you'd still get each swift-bench-client
process doing 1 GET request. Now you don't.

This should also fix the case where you've got more objects than
swift-bench-client processes, for example when you're uploading a few
large objects and then doing lots of parallel GETs of those
objects. Now you'll get the number requested, not
max(number-requested, number-of-bench-clients).

Change-Id: Ied9eb733dd9af51a3c6af8b815ad6cff0ff746b7
035e49c
Commits on Jan 22, 2013
@bloodeagle40234 bloodeagle40234 prevent irregular auth_prefix config in tempauth
Tempauth handles a authorization request by request path.
If a request's path start with auth_prefix, tempauth middleware
handles that request to authorization method.
Therefore, when configuring auth_prefix to '/', all requests
handle to authorization method.

This change enables tempauth to prevent invalid auth_prefix
config '/' and similar empty auth_prefix in initialization method.

Fixes bug #1096538.

Change-Id: I20b157e2a0809c17409fc65a8eff0858fe4aff29
161aa89
Jenkins Merge "prevent irregular auth_prefix config in tempauth" d54a5a9
@notmyname notmyname Update auto-generated docs
Removed sidebar with broken (static) links referencing out-of-date docs.

Added an external link to the Swift API docs

fixes bug #1025099

Change-Id: I7f3106175b84b1063f74aa6c5693ab1e422cdb59
ee98288
Commits on Jan 23, 2013
Eohyung Lee Fixed bug in domain_remap and cname_lookup middleware
If domain_remap and cname_lookup received request which has no host header,
then returns 500 error. This fixes that problem.

Change-Id: Ibb457e9b4cb21181d8243858c04ce255365690da
Fixes: bug #1100632
04037ae
@smerritt smerritt Add more command-line options to swift-bench.
These settings:
  1. are already exposed via config file
  2. were not exposed as command-line options, and
  3. are things I wanted on the command line while benchmarking recently.

I didn't include short options for --(get|put|delete)-concurrency
since I couldn't think of good single-letter ones, and optparse won't
take "-gc", "-pc", or "-dc".

Change-Id: I0bd7c7ae2892244eed37b8c10c9357dc7e1190d3
68479cc
@smerritt smerritt Validate numericness of ports in builder files.
You can't really goof this up using bin/swift-ring-builder, but if you
have code that uses swift.common.ring.RingBuilder directly, you can
stuff e.g. "6002" in where you mean 6002, resulting in some fairly
baffling failures. (Yes, I have done this.)

Change-Id: I87b7b7066b9ea2ce6f82255605da99cf0d283689
f2941b0
Jenkins Merge "Update auto-generated docs" bf55ae7
Jenkins Merge "Add more command-line options to swift-bench." 5a4d488
Adrian Smith Add handler for CORS "actual requests"
Fix for bug 1095130

* Added a wrapper function around public methods to handle
  CORS actual requests. These requests need to return some
  extra headers to be valid responses to a CORS request.
  Access-Control-Expose-Headers and Access-Control-Allow-Origin.

* Added support for the CORS header Access-Control-Expose-Headers.

* Some refactoring of the OPTIONS method so the
  "is_origin_allowed" logic can be reused.

* Added a little extra detail to the CORS documentation.

DocImpact

Change-Id: I68538e472a900775427f21a8a59e738a83dcc8bc
89ee10b
Jenkins Merge "Add handler for CORS "actual requests"" 77a562d
Commits on Jan 24, 2013
Jenkins Merge "Allow for multiple X-(Account|Container)-* headers." 64270fa
@chmouel chmouel Deprecate is_admin feature.
- This has been confusing since the beginning of keystoneauth.
- Do not remove the feature but just print a large warning when
  used.

Change-Id: I13990d7ab6760bb6479cf4f0717067522e1ef920
c7cf74f
@chmouel chmouel Fix keystoneauth with OPTIONS headers (eg: CORS)
- Fixes bug 1101143.

Change-Id: I2a82269e4aed2c8926e0aa736c38120784e25de2
0dc32d3
@dpgoetz dpgoetz Bulk Requests: auto extract archive and bulk delete middleware.
Fix small problem in ratelimiting middleware.

DocImpact

Change-Id: Ide3e0b9f4887626c30cae0b97eb7e2237b1df3ed
2f663ff
Commits on Jan 25, 2013
@redbo redbo Add wsgify and split_path utilities to swob
And refactor some of the code to use them.

Remove unused imports.

Change-Id: Ica479c10247fa85c740bb99cf7d1db7fbb1b2c80
c45e435
@pandemicsyn pandemicsyn Add optional locking to swift-ring-builder
If invoked as 'swift-ring-builder-safe' the directory containing the builder
file provided will be locked (via lock_parent_directory()). This provides a
small safe guard against multiple instances of the swift-ring-builder (or
other utilities that observe this lock) from attempting to write to or read
the builder/ring files while operations are in progress.

This is particularly useful in environments where ring management has been
automated (via Chef or custom solutions) but the operator still occasionally
needs to manually interact with the ring.

DocImpact

Change-Id: Ia362744a8151a91bfb586d01da582906726852e6
00dbad0
Commits on Jan 28, 2013
@famao famao Handle access_log_headers in proxy_logging
 * in etc/proxy-server.conf-sample, there is an access_log_headers directive in proxy_logging middleware.
   but current code doesn't handle it. This patch enables proxy_logging to handle access_log_headers.

Change-Id: I03337ccc4f48625af3f578ed543da992d3eec2bd
f28914c
Jenkins Merge "Handle access_log_headers in proxy_logging" 988b1ba
Commits on Jan 29, 2013
@gholt gholt Added fallocate_reserve option
Some systems behave badly when they completely run out of space. To
alleviate this problem, you can set the fallocate_reserve conf value
to a number of bytes to "reserve" on each disk. When the disk free
space falls at or below this amount, fallocate calls will fail, even
if the underlying OS fallocate call would succeed. For example, a
fallocate_reserve of 5368709120 (5G) would make all fallocate calls
fail, even for zero-byte files, when the disk free space falls under
5G.

The default fallocate_reserve is 0, meaning "no reserve", and so the
software behaves exactly as it always has unless you set this conf
value to something non-zero.

Also fixed ring builder's search_devs doc bugs.

Related: To get rsync to do the same, see
https://github.com/rackspace/cloudfiles-rsync
Specifically, see this patch:
https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff

DocImpact

Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
87a42ab
Jenkins Merge "Fix keystoneauth with OPTIONS headers (eg: CORS)" 5c93cb5
Jenkins Merge "Add wsgify and split_path utilities to swob" c9b24df
Commits on Jan 30, 2013
Christopher MacGown Allow rebalance to take a seed.
Passing a seed into rebalance makes the rebalance deterministic
which allows us to generate identical rings across disparate
nodes without having to copy the ring files around.

Change-Id: Ie5ae46ac030e61284bc501fdef9d77eeb5243afd
e189723
Jenkins Merge "Added fallocate_reserve option" 0dfd705
Commits on Jan 31, 2013
@sileht sileht Allow change the endpoint_type when use swift-dispersion tools
Fixes bug 1102319
DocImpact

Change-Id: I8fb0417ab9468e97ed01a6cb1e262630905e7f29
a1395ec
@gholt gholt Remove tempauth allowed_sync_hosts conf option
Seems we missed these references when committing
357b12d

DocImpact

Change-Id: Ia226ce1d63e52769bc067d50ec4704cea4e11c5c
8552953
@emonty emonty Cleanup based on pyflakes.
pyflakes itself can't be used in any automated gating way, because there are
two sets of false errors it raises. However, as an exercise, cleaning up the
'valid' ones uncovered three actual bugs. The other changes (mostly unused
variables) are included here for fun.

Command run: pyflakes swift | grep -v "undefined name '_'"

Change-Id: I18696bf047dedad1a9fdbde3463e214fba95f7c6
abe70e8
Jenkins Merge "Cleanup based on pyflakes." c0d497c
Commits on Feb 01, 2013
@smerritt smerritt Deterministic, repeatable serialization for rings.
The (account|container|object).ring.gz files contain, among other
things, a JSON-encoded dictionary. This change simply makes the JSON
serializer sort the keys of that dictionary so that two
Python-identical rings will result in two bytewise-identical ring
files. Also, to get repeatable compression, we lock down the timestamp
in the gzip output stream to a fixed value. (There's a timestamp value
in a gzip stream header; by default, gzip.GzipFile sticks time.time()
in there.)

This only works on Python 2.7; on 2.6, the 'mtime' argument to
gzip.GzipFile() is unsupported. Don't worry, serialization still works
on 2.6. It just doesn't always produce the same bytes for the same
ring.

Change-Id: Ide446413d0aeb78536883933fd0caf440b8f54ad
156cdc8
Jenkins Merge "Allow rebalance to take a seed." 65baec3
Commits on Feb 02, 2013
@zyluo zyluo Fixes 'not in' operator usage
Fixes bug 1110150

Change-Id: I6c48553564b1809a7ae5de9f778d93dccbe157ae
c3d4a63
Jenkins Merge "Deterministic, repeatable serialization for rings." 62e71a2
@dbishop dbishop Optimize the ring builder's _reassign_parts() method.
Another ring builder optimization.  Profiling revealed hotspots in many
calls to min() and list.sort() in _reassign_parts().  That method didn't
get exercised in my last optimization pass because that pass targeted a
rebalance where nothing really moved around.

This time, I wrote a script which created a fresh ring, added a bunch of
devices, did the initial balance, deleted some devices, balanced, and
added some more back in.

Results from homebrew Python 2.7.3 on OS X 10.8.2 Macbook Pro
(bare-metal):

 BEFORE:
 Using part-power = 18, adding 600 devices, removing 100, then adding 300 more...
 NOT Profiling to 'initial_balance.prof'
   wall-time delta: 131.33s
 NOT Profiling to 'deleting_200_rebalance.prof'
   wall-time delta: 25.67s
 NOT Profiling to 'first_rebalance.prof'
   wall-time delta: 62.00s

 AFTER:
 Using part-power = 18, adding 600 devices, removing 100, then adding 300 more...
 NOT Profiling to 'initial_balance.prof'
   wall-time delta: 28.04s
 NOT Profiling to 'deleting_200_rebalance.prof'
   wall-time delta: 9.35s
 NOT Profiling to 'first_rebalance.prof'
   wall-time delta: 16.41s

The driver script I used is available here:
https://gist.github.com/adb982aec6f0709f1273

Change-Id: I17e270acb12b5e4d4bbb1e34d8867dea90678961
ec084de
Commits on Feb 04, 2013
@greglange greglange Made internal client handle failed requests better
In two places, the internal client was not doing what it should with failed requests.

Change-Id: I49e566108fe920d85ca60fe76d24debe8ec7ffad
07d85d3
Commits on Feb 05, 2013
@redbo redbo Basic container quotas
Add a new middleware implementing some basic container quotas.

Quotas are subject to several limitations: eventual consistency, the timeliness
of the cached container_info (60 second ttl by default), and it’s unable to
reject chunked transfer uploads that exceed the quota (though once the quota
is exceeded, new chunked transfers will be refused).

However, they get most of the way to container quotas fairly inexpensively.

Quotas are set by adding meta values to the container, and are validated when
set:

  X-Container-Meta-Quota-Bytes: Maximum size of the container, in bytes.
  X-Container-Meta-Quota-Count: Maximum object count of the container.

DocImpact

Change-Id: I77cfbf6dc231a2e522bd67328e4c082424a93eee
24ef120
Commits on Feb 06, 2013
Jenkins Merge "Basic container quotas" 946c7a5
Commits on Feb 07, 2013
Jenkins Merge "Fixes 'not in' operator usage" 9f14161
@dpgoetz dpgoetz Change the flag set to specify bulk delete and expand archives from a…
… HTTP

header to a query parameter.

This is needed because query parameters show up in proxy logs and headers do
not. With this change it will be easy to determine from any log line that gets
created from the original request (of which there is currently none) that the
request was a bulk action.

Note: This is not backwards compatible with the previous method of setting a
header. Because the bulk middleware has not been included in an openstack swift
release this should be fine.

Change-Id: I0297fa2de9e491bf0b8c430c0781e2e12316ed4b
08c0174
Jenkins Merge "Fixed bug in domain_remap and cname_lookup middleware" f6b224e
Commits on Feb 08, 2013
Jenkins Merge "Change the flag set to specify bulk delete and expand archives…
… from a HTTP header to a query parameter."
9de643e
Jenkins Merge "Deprecate is_admin feature." f7ded57
@gholt gholt Fixed some swift-recon typos
Change-Id: Ibe4b59a90b82742b70b00353d22aedd972bbeb59
0640a61
Jenkins Merge "Allow change the endpoint_type when use swift-dispersion tools" c0e8ad6
Commits on Feb 09, 2013
@davidhIBM davidhIBM minor development doc changes
Reviewed development saio and guidelines while installing
a new dev and test node.

Change-Id: I59238a4d43fa08a89be14f43681ad4693f1261dd
83d8922
Jenkins Merge "Fixed some swift-recon typos" 2dca894
Commits on Feb 11, 2013
@dpgoetz dpgoetz Use a doubled proxy-logging instead of each middleware handling it
differently (if at all)

Adding a swift.source to wsgi pre_auth funcs and all middleware that makes
subrequests to proxy server.

NOTE: This change will result in a change in the number of proxy logs made for
staticweb, formpost, tempurl, and any other middleware that performs sub
requests (including swauth and SOS).

Please see docs for details.

DocImpact

Change-Id: I80cf2806add1c3d34054147e2515944be340455b
a622349
Jenkins Merge "Made internal client handle failed requests better" 89d4a51
@chmouel chmouel Add generated keystone documentation to misc.rst.
- Change a bit the formatting of the documention as well.
- Fix WARNING: Title underline too short. in misc.rst.

Change-Id: I2f4e36bcb5e01e984f0af0152bc5b3b9f7e942ce
1b86380
annegentle Replaces Copyright statements for LLC with Foundation, removes date.
	Replaced GA code for cross-domain tracking.

Patchset addresses reviewer's comments
and follows new guidance from Foundation:
http://wiki.openstack.org/Documentation/Copyright

Adds current year to each Sphinx-built page.

Addresses only the docs copyright attribution, not code files.

Change-Id: Ib90fd1c92c8fafce2db821bc2b17cef1377cfc1e
7242843
Commits on Feb 12, 2013
Jenkins Merge "Use a doubled proxy-logging instead of each middleware handlin…
…g it differently (if at all)"
3df9229
Jenkins Merge "Replaces Copyright statements for LLC with Foundation, removes…
… date."
e4e5558
Jenkins Merge "Add generated keystone documentation to misc.rst." af3bd46
jola-mirecka Change in swift-drive-audit handling log rotation.
Change supports kern.log rotation in order to avoid loss
of significant information.

There is a year change functionality added as kern.log
does not keep record of year.

There is also backwards function added which allows
reading logs from the back to the front, speeding up the
execution along with the unit test for it

Fixes Bug 1080682

Change-Id: I93436c405aff5625396514000cab774b66022dd0
902b66d
Commits on Feb 13, 2013
@jogo jogo Fix spelling mistakes
git ls-files | misspellings -f -
Source: https://github.com/lyda/misspell-check

Change-Id: I4132e6a276e44e2a8985238358533d315ee8d9c4
45f0502
@pyKun pyKun log message in exception level when path encoding errors happen.
Original code is kind of except-pass. This will ignore the error
occurs in process of encoding path. And if such error happened
without any information in logs, developer have to debug in a
long distance from error stack. That's wasting time, so there
should be a logging message here.

Fixes: bug #1123048
Change-Id: I7941b524e936241fb8d62fad5ad90de3daffe5e7
f4a09eb
Jenkins Merge "Make statsd sample rate behave better." 23f33b2
@litong01 litong01 Swift MemcacheRing (set) interface is incompatible fixes
This patch fixes the Swift MemcacheRing set and set_multi
interface incompatible problem with python memcache. The fix
added two extra named parameters to both set and set_multi
method. When only time or timeout parameter is present, then one
of the value will be used. When both time and timeout are present,
the time parameter will be used.

Named parameter min_compress_len is added for pure compatibility
purposes. The current implementation ignores this parameter.

To make swift memcached methods all consistent cross the board,
method incr and decr have also been changed to include a new
named parameter time.

In future OpenStack releases, the named parameter timeout will be
removed, keep the named parameter timeout around for now is
to make sure that mismatched releases between client and server
will still work.

From now on, when a call is made to set, set_multi, decr, incr
by using timeout parametner, a warning message will be logged to
indicate the deprecation of the parameter.

Fixes: bug #1095730
Change-Id: I07af784a54d7d79395fc3265e74145f92f38a893
2267b03
@dpgoetz dpgoetz fix large objects with unicode
Change-Id: I2a088e6cfe354d18cb2107a42bc626c2e7822e42
3b341e3
@dbishop dbishop Adds first-byte latency timings for GET requests.
This was an oustanding TODO for StatsD Swift metrics.  A new timing
metric is tracked for (only) GET requests for accounts, containers,
and objects:

  proxy-server.<req_type>.GET.<status_int>.first-byte.timing

Also updated StatsD documentation in the Admin Guide to clarify that
timing metrics are sent in units of milliseconds.

Change-Id: I5bb781c06cefcb5280f4fb1112a526c029fe0c20
bce8443
Commits on Feb 14, 2013
Jenkins Merge "log message in exception level when path encoding errors happen." 70385d3
Jenkins Merge "fix large objects with unicode" ce49777
Jenkins Merge "Swift MemcacheRing (set) interface is incompatible fixes" b66232a
Commits on Feb 15, 2013
@creiht creiht Cleanup of file permissions
Mostly removed execute on non-executable files

Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9
e88ff34
Commits on Feb 16, 2013
@chmouel chmouel Account and container info fixes and improvement.
- Fixes bug 1119282.
- Allow middleware accessing metadata of an account without having to
  store it separately in a new memcache namespace.
- Add tests for get_container_info that was previously missed.
- Add get_account_info method based on get_container_info, a function
  for other middleware to query accounts.
- Rename container_info['count'] as container_info['object_count'].

Change-Id: I43787916c7a812cb08d278edf45370521f12c912
b622993
Commits on Feb 18, 2013
@zaitcev zaitcev Documentation fixups
These are mostly cosmetic fixes for irritating imperfections:
- "separated with commas" was duplicated, leave just one
- extra whitespace here and there, man pages are not PEP8, drop
- weird extra commas, drop
- Fedora logs to /var/log/messages
- "drive is has failed", drop "is"

Change-Id: I5ceba2e61b16db4855d76c92cbc83663b9b2a0da
93ea7c6
Commits on Feb 19, 2013
@notmyname notmyname timing-based affinity sorting for primary replicas
This changes the way primary replicas can be sorted on GET requests.
Previously, replicas were shuffled. Now, if configured, the replicas
are sorted based on the most recent connection time data to that node.
This patch adds a config value that changes the sorting method.

get_more_nodes() (ie handoffs) is unaffected by this patch because
sorting by affinity would break the durability provided by the current
as-unique-as-possible handoff selection.

Timing data is collected for each node each time the proxy makes a
connection to that node (IP address). If timing data for a node doesn't
exist, then it is assumed at -1 (ie will sort earlier) so that timing
data can be collected for that node.

Change-Id: I837fa21c3a566b10cce33eb75788665e1d01cd8a
6991734
@gholt gholt Fixed formpost QUERY_STRING bugs.
Ensures that any QUERY_STRING to FormPost is not passed onward.
Handles a redirect with a query string properly.

Change-Id: If0a7d9b0a17314c6cd3852175362d4633f828d81
ffada71
Commits on Feb 20, 2013
@zhouyuan zhouyuan Adding speed limit options for DB auditor
Fix bug 1129760

Without speed limit, DB auditor will likely consume high CPU% on
storage node. That will highly impact the cluster's performance.

This patch adds two options for account/container auditor:
 - containers_per_second: Maximum containers audited per second
 - accounts_per_second: Maximum accounts audited per second

DocImpact

Change-Id: I9faa506438185a83ca77db4906969328624d015f
0937086
@chmouel chmouel Allow acl with a valid token.
- When a user as a valid token it would go to authorize but the acl
  check was after the reseller_check and due fail. Check this before
  reseller_check and add a test for it.
- Fixes bug 1020722.

Change-Id: Iaff9f35f5ee690e9b729c36d05fb9adf3368dc79
0f284e0
Commits on Feb 21, 2013
Jenkins Merge "Change in swift-drive-audit handling log rotation." 44e3915
Jenkins Merge "Documentation fixups" d14c0c0
@chmouel chmouel Add tests on keystone for invalid path requests.
- Add missing tests to make it coverage to 100%.

Change-Id: Ic182ad379e2c48cc4151b2532c509bd62d44bd00
a7e31eb
@chmouel chmouel Add cache=swift.cache for authtoken example.
- Things swill go badly with swift if we leave the default to authtoken
  to use its own memcache cache connection based python-memcache c based
  binding.

Change-Id: I293b875acdcb06e5a7a0cfa9a9bb5d7678675da0
e69f3be
Commits on Feb 22, 2013
@smerritt smerritt Make rings' replica counts adjustable.
Example:

$ swift-ring-builder account.builder set_replicas 4
$ swift-ring-builder rebalance

This is a prerequisite for supporting globally-distributed clusters,
as operators of such clusters will probably want at least as many
replicas as they have regions. Therefore, adding a region requires
adding a replica. Similarly, removing a region lets an operator remove
a replica and save some money on disks.

In order to not hose clusters with lots of data, swift-ring-builder
now allows for setting of fractional replicas. Thus, one can gradually
increase the replica count at a rate that does not adversely affect
cluster performance.

Example:

$ swift-ring-builder object.builder set_replicas 3.01
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>

$ swift-ring-builder object.builder set_replicas 3.02
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>...

Obviously, fractional replicas are nonsensical for a single
partition. A fractional replica count is for the whole ring, not for
any individual partition, and indicates the average number of replicas
of each partition. For example, a replica count of 3.2 means that 20%
of partitions have 4 replicas and 80% have 3 replicas.

Changes do not take effect until after the ring is rebalanced. Thus,
if you mean to go from 3 replicas to 3.01 but you accidentally type
2.01, no data is lost.

Additionally, 'swift-ring-builder X.builder create' can now take a
decimal argument for the number of replicas.

DocImpact

Change-Id: I12b34dacf60350a297a46be493d5d171580243ff
7548cb9
Commits on Feb 23, 2013
Jenkins Merge "Adds first-byte latency timings for GET requests." d1381ba
Commits on Feb 24, 2013
@pyKun pyKun Improve bin/swift-account-audit
1.Correct the usage of initializing a Ring object.
2.Fix typo.
3.Align lines of print_status part. (keep origin author mind)

note: Adam Kijak has commited it at early time, but didn't put it
      in Gerrit. I import his commit and send it in right Workflow.

Fixes: bug #1131901

Change-Id: If56316cc424045f7df662d60088382f7fd40e8b1
13eabdc
Commits on Feb 25, 2013
@pyKun pyKun Correct docstring for swift.common.ring.utils.build_tier_tree and add
unit test for it.

Some mistakes is in original docstring of that method. There's no unit
test for two methods in swift.common.ring.utils.

Fixes: bug #1070621

Change-Id: I6f4f211ea67d7fb8ccfe659f30bb0f5d394aca6b
d9130d7
@smerritt smerritt Fix descriptions in sample configs.
Change-Id: I7aca3c6cafd9391031f7a10cc233f99e81ee0393
a4a047c
Commits on Feb 26, 2013
Jenkins Merge "Adding speed limit options for DB auditor" 249a654
@pyKun pyKun Add debug level logs for TempAuth.authorize.
The function authorize in tempauth.py and keysthoneauth.py is very
important and frequently called in swift. But authorize in tempauth
seems hard to debug with nearly no logs here.

Change-Id: I3300a5a8d02743ff76e2ff86c51338ca24ddddcb
Fixes: bug #1129769
c52a992
@zaitcev zaitcev Move the swift.1 manpage to python-swiftclient
When someone split away python-swiftclient, he left behind the
client manpage.

Apparently all we need now is to delete it from swift and
add it to python-swiftclient. Except that of course the releases
have to be synchronized somehow or else files conflict.

Change-Id: I8b5f7b5557c28818048d8941df36473dacfb1d57
9a3e0f4
@daeskp daeskp Account Server: Refactor HEAD request handler
Deleted unused container checks. As method
swift::common::db::AccountBroker::get_container_timestamp becomes
unused, it is deleted too, along with the corresponding tests.

Change-Id: I61de4549b0abd7103226d6a13f1d9844abaa92d3
89ab090
Commits on Feb 27, 2013
Jenkins Merge "Account and container info fixes and improvement." 5e1139b
Jenkins Merge "Account Server: Refactor HEAD request handler" ac4dd56
Jenkins Merge "Improve bin/swift-account-audit" ca7468f
Jenkins Merge "Correct docstring for swift.common.ring.utils.build_tier_tree …
…and add unit test for it."
b63620c
Jenkins Merge "timing-based affinity sorting for primary replicas" 1dc38b4
Jenkins Merge "Add debug level logs for TempAuth.authorize." d3232d4
@gholt gholt Force log entries to be one line
Different versions of syslog-ng and probably other syslog services
handle multi line log messages differently and sometimes quite
poorly. This patch collapses multi line log messages into single
lines before sending them on to syslog.

It's just a copy of what was already in Python's logging.Formatter
but altered to replace the newlines with #012. I used #012 since
that's a convention we've already used elsewhere in Swift.

Change-Id: I8d0509b7cf48e45c2cf6480b51c67eec5bc94fe2
86220ba
Jenkins Merge "Add cache=swift.cache for authtoken example." a06c71c
Jenkins Merge "Move the swift.1 manpage to python-swiftclient" 2a119b6
@dpgoetz dpgoetz Adding a quick note to helper functions so its clear what they do and
they don't get misused.

Change-Id: Ie0292df9d9e565ccd608b6b61ead57dfff1d3797
7e3adf5
@creiht creiht Updating classifiers status to stable and clean up
Change-Id: I72943a9575b873cef1bf193f197b390b544ed6bf
5d2b70b
Jenkins Merge "Force log entries to be one line" 569bd1e
Commits on Feb 28, 2013
@redbo redbo make swift fsync
Swift never fsyncs, it only fdatasyncs.  That is dumb, we have important
metadata we need to save.  Also, the code was weird and had no tests.

Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
8bc065e
Jenkins Merge "Add tests on keystone for invalid path requests." 61f4d07
Jenkins Merge "Allow acl with a valid token." b6b5d66
@Jian-Zhang Jian-Zhang Added per disk PUT timing monitoring support.
Fixes bug 1104708

There could be severe performance drop for swift is one disk of one
storage node is problematic due to the tragic state of async disk I/O.

This patch provided PUT timing per kB transfered (ms/kB) monitoring
support for each non-zero-byte request of each disk and report to
statsD for alert.
-adding "object-server.PUT.<device>.timing" metrics for object-server.

DocImpact.

Change-Id: Ie94bddad28e8be52e71683bf6c9db988664abe47
1d8a02f
Jenkins Merge "make swift fsync" 9e00618
Commits on Mar 01, 2013
@ywang19 ywang19 Remove reduant check of list.
Remove len() check and parenthesis.

Fixes Bug #1136893

Change-Id: Ib47ec4890c2f6a50e316a7fef204ef818c6c4d6e
b715eb5
@gholt gholt TempURL filename options; bug fixes
- Prior to this commit, a Content-Disposition header was always set
  on responses to GET requests, with the filename based on the object
  name. Now, the header will only be set for 2xx responses and the
  filename can be overridden with a filename query parameter on the
  request.

- Fixed a bug where all query parameters on the request were being
  passed down the WSGI pipeline. Now, just the query parameters
  useful in log-based debugging are included. This becomes important
  with things like the Bulk middleware that act upon query
  parameters.

- Fixed bug where the Content-Disposition header wasn't following RFC
  spec.

DocImpact

Change-Id: I66ad809321dcdd03444324973c8b76869e3b0c8e
c4c66d8
Jenkins Merge "Fixed formpost QUERY_STRING bugs." cc63f8d
Commits on Mar 02, 2013
@dpgoetz dpgoetz Static Large Object Support
DocImpact

Change-Id: I7edaa5e44208ab451f7f7566b64bb571b8eea1f9
5d73da1
Jenkins Merge "TempURL filename options; bug fixes" 9808a87
Jenkins Merge "Remove reduant check of list." 84bd434
Commits on Mar 04, 2013
Jenkins Merge "Make rings' replica counts adjustable." 457ff96
@gholt gholt Updated get_more_nodes algorithm
The handoff nodes will try to be in zones other than the primary
zones, will take into account the device weights, and will usually
keep the same sequences of handoffs even with ring changes.

On a real ring test the old get_more_nodes placed data mostly evenly
across zones, which is a problem for differently weighted zones. But
the real problem was that the extra partitions given to each device
was 0% to 0.77% with only 46.05% of the candidate devices getting
anything. Some of the devices increased in effective weight over 50%
in the test.

The new get_more_nodes placed closer to what the zone weights were
and the extra partitions given to each device was 0% to 0.24% with
90.58% of the candidate devices getting something. The worst off
device only increased in effective weight by 10.71%.

Change-Id: Iffb133a22db69074acaa2b90854cbfa92e4c2b9e
e064ba1
Commits on Mar 05, 2013
Jenkins Merge "Static Large Object Support" 7117b74
@litong01 litong01 David Hadas email address has been changed.
Change-Id: Ic3e9ad50f123eecb4abbe91340b0acff4a2d580c
7fd4323
@smerritt smerritt Spread handoffs out better around zones.
Before, you'd get your 3* primary nodes in 3 different zones, and then
get_more_nodes would give you everything it could from a non-primary
zone, and then finish up with stuff from the primary zones. It would
sort of look like this:

P: device in a primary node's zone
N: device not in a primary node's zone

PPPNNNNNNNNNNNNNNNNNNN...NNNNNNNNNPPP...PPPPPP

(The first three Ps are the primary nodes; they don't actually come
out of get_more_nodes(), but they're included for clarity.)

Now, the first few handoffs from get_more_nodes are in non-primary
zones, but only one per zone, and then the rest of the handoffs ignore
zones. It's still sampling the ring, so it's still taking weights into
consideration, but the zone distribution is more even early in the
handoff chain. It looks like this, assuming 10 zones:

P: device in a primary node's zone
N: device not in a primary node's zone
D: zone doesn't matter

PPPNNNNNNNDDDDDDDDDDD...DDD

* or whatever your replica count is

Change-Id: I31d2a2bc2cd6038386a2df85cd4fa37ccf2f650e
27dcaf2
Commits on Mar 06, 2013
Jenkins Merge "Added per disk PUT timing monitoring support." 28ac46d
@smerritt smerritt Fix misspelled variable name
Change-Id: Ied6ed3cad16e9797df73a05f3df3ac9cc64299e0
71a20d0
Jenkins Merge "Spread handoffs out better around zones." 34beb92
@Frostman Frostman Support listing endpoints for an object.
Implements blueprint list-endpoints.

DocImpact: new middleware list_endpoints.

Change-Id: I0c4911ff726abd4cb8ce2b6245c99786ad46b410
7d5095c
@davidhIBM davidhIBM Support tests for Apache
Add support for functional tests that work with Apache web front end

Change-Id: I72358a12016eeccc842d834461dbebaa188aa117
Implements: blueprint wsgi-application-interface
40782ed
Commits on Mar 07, 2013
@redbo redbo simplify the chexor function
Replace all that map(operator) nonsense.

It changes the error raised on invalid hashes, but we don't handle that
anywhere, and it shouldn't ever happen in real life.

Change-Id: Ib8cb549fac05e0b2725b4ea295326ac0c5e1f035
0219b08
Jenkins Merge "Support tests for Apache" 09f1387
Jenkins Merge "simplify the chexor function" 48380c5
@smerritt smerritt Fix some unreadable code.
I thought it was readable back when I wrote it; turns out it's
not. Oops.

Change-Id: I1e1020935356522d6e07409aa867a5ffc8919787
Bonus: remove unused import.
d9fb84d
Commits on Mar 08, 2013
@cschwede cschwede Account quotas
Add a new middleware implementing account quotas.

This middleware blocks write requests (PUT, POST) if a given quota (in bytes)
is exceeded while DELETE requests are still allowed.

Quotas are stored in the x-account-meta-quota-bytes metadata entry.
Write requests to this metadata setting are only allowed for resellers.

Change-Id: I57fd7c6209f34cc79d4bab72d500d43ba2a62083
28c75db
Jenkins Merge "Account quotas" a8af383
jola-mirecka Add exception to swift-drive-audit if no date in line.
Sometimes there is no date at the beginning of a line in kern.log.
Although it does not happen often, there should be a check ensuring
the program doesn't crash in case it happens.

Added try-except block surrounding parsing string to date format.

Change-Id: I44a101266582eea2199189a006afa1037a9bd4ea
Fixes: bug #1152658
5e427e5
Commits on Mar 11, 2013
Vladimir Vechkanov Add cryptography functional
All files, which are saving are crypting with crypto algorithm
chosen in configuration files.

Keys for encryption/decryption are saved in keystore. As keystore
can be used sql database or you own driver. For writing you own
keystore driver you should implement KeyManager class.

For cryptography used different crypto drivers. Each of them
can realise different crypto algorithms. Also can be written
any new crypto driver by implementation of CryptoDriver class.

For now M2CryptoDriver driver exists, which is wrapper ower
2crypto library. From M2CryptoDriver aes 128 cryptography
algorithms can be used.

Change-Id: Idf4190f9535609dcdcf01304c1d37770643ac361
a6c8824
@akscram akscram Changed .gitreview for Mirantis Gerrit.
Change-Id: I70fb770fc970bf43ff57d089ea59d62da6814168
8b2379f
Commits on Mar 21, 2013
@skraynev skraynev Management of an encryption keys is common.
Some key changes within this commit:
- Common part of storing encryption keys moved to
  swift.common.key_manager. Also, the package have interface function
  which named as 'get_driver' to initialize instances of drivers.
- Object server encryption driver have similar interface to get driver
  instances.
- Changed sample configuration files for object and proxy servers.
- Improved unittests for encryption implementation.

Change-Id: I82beb7d383ae2cd2f7d20a26e4adcb39c943ba1a
97f2f22
@akscram akscram Improved method of loading drivers.
* Add function create_instance which support to import class by path,
  validate its subclass and create instance. Wrote unit tests.
* Replace mechanism of importing key manager and crypto drivers.
* Add descriptions to samples of configuration.
* Add dummy key manager driver.

Change-Id: I7370f4dd9c58b346667ee7baab851997ae0a7700
ff9f60e
@akscram akscram Improved pluggable mechanism of encryption.
* Cleaned interface of drivers. Added encryption context to exchange
  of security data between unified encryption/decryption methods.
  Instance of key manager created externally towards drivers.
* Renamed FakeDriver to DummyDriver.
* ObjectController and DiskFile used new interface of drivers.

Change-Id: I4acb253b9a0f4c348fc6f27326e4219d086efbef
556d8e9
@akscram akscram Key manager add header on put object request.
* X-Meta-Object-key_id header is added if it's request to PUT object.

Change-Id: If437dd9af6ff0d500b3c18e9dc6a90411114e29c
350bfd2
@akscram akscram Key manager drivers support schema migration.
* Script to run migration swift-key-manager-sync.
* KeyDriver interface extended by sync method.
* Intergration with SQLAlchemy Migration for SQLDriver.

Change-Id: I2c482e994668fcb55c6a3b11158ae3c65d43d33e
6b2a175
@skraynev skraynev Fixing pep8
Change-Id: I05d6f12fa8d88e2cd57f9dc0f0faa76bfd26874e
fe9bf37
@skraynev skraynev Fix migration script.
Change-Id: I39cf20d3817d73cfad45f8f2d091197cd003cb06
8f7a709
@skraynev skraynev Changing lenght of encoding key string.
Change-Id: Ia7fd41f02959c063d4c6db4efef1a1e742902179
005733d
@skraynev skraynev Removing function create_table
Change-Id: I9e960867dcf56a2c142cdbfb6aadff105b8cd104
c29ae05
@ogelbukh ogelbukh Initial documentation commit for swift-init
This is the initial commit of documentation extension for
Swift with on-disk object encryption support

Change-Id: I962b1ba6bff75854a301cad2969ac3a527b78131
2770e6a
Vladimir Vechkanov Fix problem with Range request
Change-Id: I637e93ce40b838f3198c916f83bb2744acf964c5
6b39baf
Vladimir Vechkanov Fix key_manager.
* Add Key-Id in metadata for POST request (fix for STF-113).
* Fix key_manager unittests wich testing setting up Key-Id header. Test
  test_call_with_object_put added into test_call_without_key_id_header.

Change-Id: I574c9fb945c9c9fbee61861d876a997434084de4
a902e22
Vladimir Vechkanov Skip functional tests depends on issue STF-121.
Skip tests (in file test/functional/tests.py):
    testCopy
    testCopyFromHeader
    testMetadataNumberLimit
    testMetadataOnPost
    testMetadataOnPut

Change-Id: Id326cc2ed280a24579bc76393e2369faa876c6d6
2f03077
Vladimir Vechkanov Skip functional test depends on issue STF-95.
Skip test testContainerInfo.

Change-Id: I1a445404905ab0b434bd07d036a0bd3685c28a8e
7970fbf
Vladimir Vechkanov Skip functional tests depends on issue STF-73.
Skip tests:
    testContainerInfo
    testSerialization

Change-Id: I37cfdee860c7d06790a2410020127a205e1582e3
1454e31
Roman Pystin Removed Meta mechanism from SQL key_manager
Replaced Meta mechanism with Session/Query one.
Updated unit tests.
dee20e1
Commits on Mar 22, 2013
@akscram akscram Fix PEP8
Change-Id: I9eb051126d7fde555c0de7bf7a8ca8e00cc33877
69c29c1
@skraynev skraynev Set new mirantis repositories .git-review
Change-Id: Ibfb5219af6c91e4a968f37a237c3e45464bd2e59
8cfc92c
Roman Pystin Removed reconnect_to_db from SQL driver
Updated unit tests.
c552058
Commits on Mar 26, 2013
Roman Pystin Removed wrong mock calls
Increased code readability.
b028065