Permalink
Change-Id: Ia8e31ed0d5aefe67f2f926dc92d9acd6c0c98007
swift (2.26.0, OpenStack Victoria) | |
* Extend concurrent reads to erasure coded policies. Previously, the | |
options `concurrent_gets` and `concurrency_timeout` only applied to | |
replicated policies. | |
* Add a new `concurrent_ec_extra_requests` option to allow the proxy to | |
make some extra backend requests immediately. The proxy will respond as | |
soon as there are enough responses available to reconstruct. | |
* The concurrent read options (`concurrent_gets`, `concurrency_timeout`, | |
and `concurrent_ec_extra_requests`) may now be configured per | |
storage-policy. | |
* Replication servers can now handle all request methods. This allows | |
ssync to work with a separate replication network. | |
* All background daemons now use the replication network. This allows | |
better isolation between external, client-facing traffic and internal, | |
background traffic. Note that during a rolling upgrade, replication | |
servers may respond with `405 Method Not Allowed`. To avoid this, | |
operators should remove the config option `replication_server = true` | |
from their replication servers; this will allow them to handle all | |
request methods before upgrading. | |
* S3 API improvements: | |
* Fixed some SignatureDoesNotMatch errors when using the AWS .NET SDK. | |
* Add basic read support for object tagging. This improves | |
compatibility with AWS CLI version 2. Write support is not | |
yet implemented, so the tag set will always be empty. | |
* CompleteMultipartUpload requests may now be safely retried. | |
* Improved quota-exceeded error messages. | |
* Improved logging and statsd metrics. Be aware that this will cause | |
an increase in the proxy-logging statsd metrics emited for S3 | |
responses. However, this should more accurately reflect the state | |
of the system. | |
* S3 requests are now less demanding on the container layer. | |
* Python 3 bug fixes: | |
* Fixed an error when reading encrypted data that was written while | |
running Python 2 for a path that includes non-ASCII characters. This | |
was caused by a difference in string types that resulted in | |
ambiguity when decrypting. To prevent the ambiguity for new data, set | |
`meta_version_to_write = 3` in your keymaster configuration after | |
upgrading all proxy servers. | |
If upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set | |
`meta_version_to_write = 1` in your keymaster configuration prior | |
to upgrading. | |
* Object expiration respects the `expiring_objects_container_divisor` | |
config option. | |
* `fallocate_reserve` may be specified as a percentage in more places. | |
* The ETag-quoting middleware no longer raises TypeErrors. | |
* Sharding improvements: | |
* Prevent object updates from auto-creating shard containers. This | |
ensures more consistent listings for sharded containers during | |
rebalances. | |
* Deleted shard containers are no longer considered root containers. | |
This prevents unnecessary sharding audit failures and allows the | |
deleted shard database to actually be unlinked. | |
* `swift-container-info` now summarizes shard range information. | |
Pass `-v`/`--verbose` if you want to see all of them. | |
* Improved container-sharder stat reporting to reduce load on root | |
container databases. | |
* Don't inject shard ranges when user quits. | |
* Servers now open one listen socket per worker, ensuring each worker | |
serves roughly the same number of concurrent connections. | |
* Server workers may now be gracefully terminated via `SIGHUP` or | |
`SIGUSR1`. The parent process will then spawn a fresh worker. | |
* During rebalances, clients should no longer get 404s for data that | |
exists but whose replicas are overloaded. | |
* Improved cache management for account and container responses. | |
* Allow proxy-logging middlewares to be configured more independently. | |
* Allow operators to pass either raw or URL-quoted paths to | |
swift-get-nodes. Notably, this allows swift-get-nodes to work with | |
the reserved namespace used for object versioning. | |
* Container read ACLs now work with object versioning. This only | |
allows access to the most-recent version via an unversioned URL. | |
* Improved how containers reclaim deleted rows to reduce locking and object | |
update throughput. | |
* Large object reads log fewer client disconnects. | |
* Allow ratelimit to be placed multiple times in a proxy pipeline, | |
such as both before s3api and auth (to handle swift requests without | |
needing to make an auth decision) and after (to limit S3 requests). | |
* Shuffle object-updater work. This somewhat reduces the impact a | |
single overloaded database has on other containers' listings. | |
* Fix a proxy-server error when retrieving erasure coded data when | |
there are durable fragments but not enough to reconstruct. | |
* Fix an error in the proxy server when finalizing data. | |
* Improve performance when increasing partition power. | |
* Various other minor bug fixes and improvements. | |
swift (2.25.0, OpenStack Ussuri) | |
* WSGI server processes can now notify systemd when they are ready. | |
* Added `ttfb` (Time to First Byte) and `pid` (Process ID) to the set | |
of available proxy-server log fields. For more information, see | |
https://docs.openstack.org/swift/latest/logs.html | |
* Improved proxy-server performance by reducing unnecessary locking, | |
memory copies, and eventlet scheduling. | |
* Reduced object-replicator and object-reconstructor CPU usage by only | |
checking that the device list is current when rings change. | |
* Improved performance of sharded container listings when performing | |
prefix listings. | |
* Improved container-sync performance when data has already been | |
deleted or overwritten. | |
* Account quotas are now enforced even on empty accounts. | |
* Getting an SLO manifest with `?format=raw` now responds with an ETag | |
that matches the MD5 of the generated body rather than the MD5 of | |
the manifest stored on disk. | |
* Provide useful status codes in logs for some versioning and symlink | |
subrequests that were previously logged as 499. | |
* Fixed 500 from cname_lookup middleware. Previously, if the looked-up | |
domain was used by domain_remap to update the request path, the | |
server would respond Internal Error. | |
* On Python 3, fixed an issue when reading or writing objects with a | |
content-type like `message/*`. Previously, Swift would fail to respond. | |
* On Python 3, fixed a RecursionError in swift-dispersion-report when | |
using TLS. | |
* Fixed a bug in the new object versioning API that would cause more | |
than `limit` results to be returned when listing. | |
* Various other minor bug fixes and improvements. | |
swift (2.24.0) | |
* Added a new object versioning mode, with APIs for querying and | |
accessing old versions. For more information, see the documentation | |
at https://docs.openstack.org/swift/latest/middleware.html#module-swift.common.middleware.versioned_writes.object_versioning | |
* Added support for S3 versioning using the above new mode. | |
* Added a new middleware to allow accounts and containers to opt-in to | |
RFC-compliant ETags. This may be useful when using Swift as an origin | |
for some content delivery networks. For more information, see the | |
documentation at https://docs.openstack.org/swift/latest/middleware.html#module-swift.common.middleware.etag_quoter | |
Clients should be aware of the fact that ETags may be quoted for RFC | |
compliance; this may become the default behavior in some future release. | |
* Proxy, account, container, and object servers now support "seamless | |
reloads" via `SIGUSR1`. This is similar to the existing graceful | |
restarts but keeps the server socket open the whole time, reducing | |
service downtime. | |
* New buckets created via the S3 API will now store multi-part upload | |
data in the same storage policy as other data rather than the | |
cluster's default storage policy. | |
* Device region and zone can now be changed via `swift-ring-builder`. | |
Note that this may cause a lot of data movement on the next rebalance | |
as the builder tries to reach full dispersion. | |
* Added support for Python 3.8. | |
* The container sharder can now handle containers with special | |
characters in their names. | |
* Internal client no longer logs object DELETEs as status 499. | |
* Objects with an `X-Delete-At` value in the far future no longer cause | |
backend server errors. | |
* The bulk extract middleware once again allows clients to specify metadata | |
(including expiration timestamps) for all objects in the archive. | |
* Container sync now synchronizes static symlinks in a way similar to | |
static large objects. | |
* `swift_source` is set for more sub-requests in the proxy-server. See | |
https://docs.openstack.org/swift/latest/logs.html#swift-source | |
* Errors encountered while validating static symlink targets no longer | |
cause BadResponseLength errors in the proxy-server. | |
* On Python 3, the KMS keymaster now works with secrets stored | |
in Barbican with a text/plain payload-content-type. | |
* On Python 3, the formpost middleware now works with unicode file names. | |
* Several utility scripts now work better on Python 3: | |
* swift-account-audit | |
* swift-dispersion-populate | |
* swift-drive-recon | |
* swift-recon | |
* On Python 3, certain S3 API headers are now lower case as they | |
would be coming from AWS. | |
* Per-service `auto_create_account_prefix` settings are now deprecated | |
and may be ignored in a future release; if you need to use this, please | |
set it in the `[swift-constraints]` section of /etc/swift/swift.conf. | |
* Various other minor bug fixes and improvements. | |
swift (2.23.1, train stable backports) | |
* On Python 3, the KMS keymaster now works with secrets stored | |
in Barbican with a text/plain payload-content-type. | |
* Several utility scripts now work better on Python 3: | |
* swift-account-audit | |
* swift-dispersion-populate | |
* swift-drive-recon | |
* swift-recon | |
swift (2.23.0, OpenStack Train) | |
* Python 3.6 and 3.7 are now fully supported. Several py3-related | |
fixes are included: | |
* Removed a request-smuggling vector when running a mixed | |
py2/py3 cluster. | |
* Allow fallocate_reserve to be specified as a percentage. | |
* Fixed listings for sharded containers. | |
* Fixed non-ASCII account metadata handling. | |
* Fixed rsync output parsing. | |
* Fixed some title-casing of headers. | |
If you've been testing Swift on Python 3, upgrade at your earliest | |
convenience. | |
* Added "static symlinks", which perform some validation as they | |
follow redirects and include more information about their target | |
in container listings. | |
* Multi-character strings may now be used as delimiters in account | |
and container listings. | |
* Sharding improvements | |
* Container metadata related to sharding are now removed when no | |
longer needed. | |
* Empty container databases (such as might be created on handoffs) | |
now shard much more quickly. | |
* The proxy-server now ignores 404 responses from handoffs that have | |
no data when deciding on the correct response for object requests, | |
similar to what it already does for account and container requests. | |
* Static Large Object sizes in listings for versioned containers are | |
now more accurate. | |
* When refetching Static Large Object manifests, non-manifest responses | |
are now handled better. | |
* S3 API now translates 503 Service Unavailable responses to a more | |
S3-like response instead of raising an error. | |
* Improved proxy-to-backend requests to be more RFC-compliant. | |
* Dependency update: eventlet must be at least 0.25.0. This also | |
dragged forward minimum-supported versions of dnspython (1.15.0), | |
greenlet (0.3.2), and six (1.10.0). | |
* Various other minor bug fixes and improvements. | |
swift (2.22.0) | |
* Experimental support for Python 3.6 and 3.7 is now available. | |
Note that this requires eventlet>=0.25.0. All unit tests pass, | |
and running functional tests under Python 2 will pass against | |
services running under Python 3. Expect full support in the | |
next minor release. | |
* Log formats are now more configurable and include support for | |
anonymization. See the log_msg_template option in proxy-server.conf | |
and https://docs.openstack.org/swift/latest/logs.html#proxy-logs | |
for more information. | |
* Added an operator tool, swift-container-deleter, to asynchronously | |
delete some or all objects in a container using the object expirers. | |
* Swift-all-in-one Docker images are now built and published to | |
https://hub.docker.com/r/openstackswift/saio. These are intended | |
for use as development targets, but will hopefully be useful as a | |
starting point for other work involving containerizing Swift. | |
* The object-expirer may now be configured in object-server.conf. | |
This is in anticipation of a future change to allow the | |
object-expirer to be deployed on all nodes that run object-servers. | |
* Correctness improvements | |
* The proxy-server now ignores 404 responses from handoffs without | |
databases when deciding on the correct response for account and | |
container requests. | |
* Object writes to a container whose existence cannot be verified | |
now 503 instead of 404. | |
* Sharding improvements | |
* The container-replicator now only attempts to fetch shard ranges if | |
the remote indicates that it has shard ranges. Further, it does so | |
with a timeout to prevent the process from hanging in certain cases. | |
* The proxy-server now caches 'updating' shards, improving write | |
performance for sharded containers. A new config option, | |
`recheck_updating_shard_ranges`, controls the cache time; set it to | |
0 to disable caching. | |
* The container-replicator now correctly enqueues container-reconciler | |
work for sharded containers. | |
* S3 API improvements | |
* Unsigned payloads work with v4 signatures once more. | |
* Multipart upload parts may now be copied from other multipart uploads. | |
* CompleteMultipartUpload requests with a Content-MD5 now work. | |
* Content-Type can now be updated when copying an object. | |
* Fixed v1 listings that end with a non-ASCII object name. | |
* Background corruption-detection improvements | |
* Detect and remove invalid entries from hashes.pkl | |
* When object path is not a directory, just quarantine it, | |
rather than the whole suffix. | |
* Dependency updates: we've increased our minimum supported version | |
of cryptography to 2.0.2 and netifaces to 0.8. This is largely due | |
to the difficulty of continuing to test with the old versions. | |
If running Swift under Python 3, eventlet must be at least 0.25.0. | |
* Various other minor bug fixes and improvements. | |
swift (2.21.1, stein stable backports) | |
* Sharding improvements | |
* The container-replicator now only attempts to fetch shard ranges if | |
the remote indicates that it has shard ranges. Further, it does so | |
with a timeout to prevent the process from hanging in certain cases. | |
* The container-replicator now correctly enqueues container-reconciler | |
work for sharded containers. | |
* Container metadata related to sharding are now removed when no | |
longer needed. | |
* S3 API improvements | |
* Unsigned payloads work with v4 signatures once more. | |
* Multipart upload parts may now be copied from other multipart uploads. | |
* CompleteMultipartUpload requests with a Content-MD5 now work. | |
* Content-Type can now be updated when copying an object. | |
* Fixed v1 listings that end with a non-ASCII object name. | |
* Background corruption-detection improvements | |
* Detect and remove invalid entries from hashes.pkl | |
* When object path is not a directory, just quarantine it, | |
rather than the whole suffix. | |
* Static Large Object sizes in listings for versioned containers are | |
now more accurate. | |
* When refetching Static Large Object manifests, non-manifest responses | |
are now handled better. | |
* Cross-account symlinks now store correct account information in | |
container listings. This was previously fixed in 2.22.0. | |
* Requesting multiple ranges from a Dynamic Large Object now returns the | |
entire object instead of incorrect data. This was previously fixed in | |
2.23.0. | |
* When making backend requests, the proxy-server now ensures query | |
parameters are always properly quoted. Previously, the proxy would | |
encounter an error on Python 2.7.17 if the client included non-ASCII | |
query parameters in object requests. This was previously fixed in | |
2.23.0. | |
swift (2.21.0, OpenStack Stein) | |
* Change the behavior of the EC reconstructor to perform a | |
fragment rebuild to a handoff node when a primary peer responds | |
with 507 to the REPLICATE request. This changes EC to match the | |
existing behavior of replication when drives fail. After a | |
rebalance of EC rings (potentially removing unmounted/failed | |
devices), it's most IO efficient to run in handoffs_only mode to | |
avoid unnecessary rebuilds. | |
* O_TMPFILE support is now detected by attempting to use it | |
instead of looking at the kernel version. This allows older | |
kernels with backported patches to take advantage of the | |
O_TMPFILE functionality. | |
* Add slo_manifest_hook callback to allow other middlewares to | |
impose additional constraints on or make edits to SLO manifests | |
before being written. For example, a middleware could enforce | |
minimum segment size or insert data segments. | |
* Fixed an issue with multi-region EC policies that caused the EC | |
reconstructor to constantly attempt cross-region rebuild | |
traffic. | |
* Fixed an issue where S3 API v4 signatures would not be validated | |
against the body of the request, allowing a replay attack if | |
request headers were captured by a malicious third party. | |
* Display crypto data/metadata details in swift-object-info. | |
* formpost can now accept a content-encoding parameter. | |
* Fixed an issue where multipart uploads with the S3 API would | |
sometimes report an error despite all segments being upload | |
successfully. | |
* Multipart object segments are now actually deleted when the | |
multipart object is deleted via the S3 API. | |
* Swift now returns a 503 (instead of a 500) when an account | |
auto-create fails. | |
* Fixed a bug where encryption would store the incorrect key | |
metadata if the object name starts with a slash. | |
* Fixed an issue where an object server failure during a client | |
download could leave an open socket between the proxy and | |
client. | |
* Fixed an issue where deleted EC objects didn't have their | |
on-disk directories cleaned up. This would cause extra resource | |
usage on the object servers. | |
* Fixed issue where bulk requests using xml and expect | |
100-continue would return a malformed HTTP response. | |
* Various other minor bug fixes and improvements. | |
swift (2.20.0) | |
* S3 API compatibility updates | |
* Swift can now cache the S3 secret from Keystone to use for | |
subsequent requests. This functionality is disabled by default but | |
can be enabled by setting the `secret_cache_duration` in the s3token | |
section of the proxy server config to a number greater than 0. | |
* s3api now mimics the AWS S3 behavior of periodically sending | |
whitespace characters on a Complete Multipart Upload request to keep | |
the connection from timing out. Note that since a request could fail | |
after the initial 200 OK response has been sent, it is important to | |
check the response body to determine if the request succeeded. | |
* s3api now properly handles x-amz-metadata-directive headers on | |
COPY operations. | |
* s3api now uses concurrency (default 2) to handle multi-delete | |
requests. This allows multi-delete requests to be processed much | |
more quickly. | |
* s3api now mimics some forms of AWS server-side encryption | |
based on whether Swift's at-rest encryption functionality is enabled. | |
Note that S3 API users are now able to know more about how the | |
cluster is configured than they were previously, ie knowledge of | |
encryption at-rest functionality being enabled or not. | |
* s3api responses now include a '-' in multipart ETags. | |
For new multipart-uploads via the S3 API, the ETag that is | |
stored will be calculated in the same way that AWS uses. This | |
ETag will be used in GET/HEAD responses, bucket listings, and | |
conditional requests via the S3 API. Accessing the same object | |
via the Swift API will use the SLO Etag; however, in JSON | |
container listings the multipart upload etag will be exposed | |
in a new "s3_etag" key. Previously, some S3 clients would complain | |
about download corruption when the ETag did not have a '-'. | |
* S3 ETag for SLOs now include a '-'. | |
Ordinary objects in S3 use the MD5 of the object as the ETag, | |
just like Swift. Multipart Uploads follow a different format, notably | |
including a dash followed by the number of segments. To that end | |
(and for S3 API requests *only*), SLO responses via the S3 API have a | |
literal '-N' added on the end of the ETag. | |
* The default location is now set to "us-east-1". This is more likely | |
to be the default region that a client will try when using v4 | |
signatures. | |
Deployers with clusters that relied on the old implicit default | |
location of "US" should explicitly set `location = US` in the | |
`[filter:s3api]` section of proxy-server.conf before upgrading. | |
* Add basic support for ?versions bucket listings. We still do not | |
have support for toggling S3 bucket versioning, but we can at least | |
support getting the latest versions of all objects. | |
* Fixed an issue with SSYNC requests to ensure that only one request | |
can be running on a partition at a time. | |
* Data encryption updates | |
* The kmip_keymaster middleware can now be configured directly in the | |
proxy-server config file. The existing behavior of using an external | |
config file is still supported. | |
* Multiple keymaster middlewares are now supported. This allows | |
migration from one key provider to another. | |
Note that secret_id values must remain unique across all keymasters | |
in a given pipeline. If they are not unique, the right-most keymaster | |
will take precedence. | |
When looking for the active root secret, only the right-most | |
keymaster is used. | |
* Prevent PyKMIP's kmip_protocol logger from logging at DEBUG. | |
Previously, some versions of PyKMIP would include all wire | |
data when the root logger was configured to log at DEBUG; this | |
could expose key material in logs. Only the kmip_keymaster was | |
affected. | |
* Fixed an issue where a failed drive could prevent the container sharder | |
from making progress. | |
* Storage policy definitions in swift.conf can now define the diskfile | |
to use to access objects. See the included swift.conf-sample file for | |
a description of usage. | |
* The EC reconstructor will now attempt to remove empty directories | |
immediately, while the inodes are still cached, rather than waiting | |
until the next run. | |
* Added a keep_idle config option to configure KEEPIDLE time for TCP | |
sockets. The default value is the old constant of 600. | |
* Add databases_per_second to the account-replicator, | |
container-replicator, and container-sharder. This prevents them from | |
using a full CPU core when they are not IO limited. | |
* Allow direct_client users to overwrite the X-Timestamp header. | |
* Various other minor bug fixes and improvements. | |
swift (2.19.2, rocky stable backports) | |
* Sharding improvements | |
* The container-replicator now only attempts to fetch shard ranges if | |
the remote indicates that it has shard ranges. Further, it does so | |
with a timeout to prevent the process from hanging in certain cases. | |
* The container-replicator now correctly enqueues container-reconciler | |
work for sharded containers. | |
* S3 API improvements | |
* Fixed an issue where v4 signatures would not be validated against | |
the body of the request, allowing a replay attack if request headers | |
were captured by a malicious third party. Note that unsigned payloads | |
still function normally. | |
* CompleteMultipartUpload requests with a Content-MD5 now work. | |
* Fixed v1 listings that end with a non-ASCII object name. | |
* Multipart object segments are now actually deleted when the | |
multipart object is deleted via the S3 API. | |
* Fixed an issue that caused Delete Multiple Objects requests with | |
large bodies to 400. This was previously fixed in 2.20.0. | |
* Fixed an issue where non-ASCII Keystone EC2 credentials would not get | |
mapped to the correct account. This was previously fixed in 2.20.0. | |
* Background corruption-detection improvements | |
* Detect and remove invalid entries from hashes.pkl | |
* When object path is not a directory, just quarantine it, | |
rather than the whole suffix. | |
* Fixed a bug where encryption would store the incorrect key | |
metadata if the object name starts with a slash. | |
* Fixed an issue where an object server failure during a client | |
download could leave an open socket between the proxy and | |
client. | |
* Static Large Object sizes in listings for versioned containers are | |
now more accurate. | |
* When refetching Static Large Object manifests, non-manifest responses | |
are now handled better. | |
* Cross-account symlinks now store correct account information in | |
container listings. This was previously fixed in 2.22.0. | |
* Requesting multiple ranges from a Dynamic Large Object now returns the | |
entire object instead of incorrect data. This was previously fixed in | |
2.23.0. | |
* When making backend requests, the proxy-server now ensures query | |
parameters are always properly quoted. Previously, the proxy would | |
encounter an error on Python 2.7.17 if the client included non-ASCII | |
query parameters in object requests. This was previously fixed in | |
2.23.0. | |
swift (2.19.1, rocky stable backports) | |
* Prevent PyKMIP's kmip_protocol logger from logging at DEBUG. | |
Previously, some versions of PyKMIP would include all wire | |
data when the root logger was configured to log at DEBUG; this | |
could expose key material in logs. Only the kmip_keymaster was | |
affected. | |
* Fixed an issue where a failed drive could prevent the container sharder | |
from making progress. | |
* Fixed a bug in how Swift uses eventlet that was exposed under high | |
concurrency. | |
swift (2.19.0, OpenStack Rocky) | |
* TempURLs now support IP range restrictions. Please see | |
https://docs.openstack.org/swift/latest/middleware.html#client-usage | |
for more information on how to use this additional restriction. | |
* Add support for multiple root encryption secrets for the trivial | |
and KMIP keymasters. This allows operators to rotate encryption | |
keys over time without needing to re-encrypt all existing data | |
in the cluster. Please see the included sample config files for | |
instructions on how to multiple encryption keys. | |
* The object updater now supports two configuration settings: | |
"concurrency" and "updater_workers". The latter controls how many | |
worker processes are spawned, while the former controls how many | |
concurrent container updates are performed by each worker | |
process. This should speed the processing of async_pendings. | |
On upgrade, a node configured with concurrency=N will still handle | |
async updates N-at-a-time, but will do so using only one process | |
instead of N. | |
If you have a config file like this: | |
[object-updater] | |
concurrency = <N> | |
and you want to take advantage of faster updates, then do this: | |
[object-updater] | |
concurrency = 8 # the default; you can omit this line | |
updater_workers = <N> | |
If you want updates to be processed exactly as before, do this: | |
[object-updater] | |
concurrency = 1 | |
updater_workers = <N> | |
* When listing objects in a container in json format, static large | |
objects (SLOs) will now include an additional new "slo_etag" key | |
that matches the etag returned when requesting the SLO. The | |
existing "hash" key remains unchanged as the MD5 of the SLO | |
manifest. Text and XML listings are unaffected by this change. | |
* Log deprecation warnings for `run_pause`. This setting was | |
deprecated in Swift 2.4.0 and is replaced by `interval`. | |
It may be removed in a future release. | |
* Object reconstructor logs are now prefixed with information | |
about the specific worker process logging the message. This | |
makes reading the logs and understanding the messages much simpler. | |
* Lower bounds of dependencies have been updated to reflect what | |
is actually tested. | |
* SSYNC replication mode now removes as much of the directory | |
structure as possible as soon at it observes that the directory | |
is empty. This reduces the work needed for subsequent replication | |
passes. | |
* The container-updater now reports zero objects and bytes used for | |
child DBs in sharded containers. This prevents double-counting in | |
utilization reports. | |
* Add fallocate_reserve to account and container servers. This | |
allows disks shared between account/container and object rings to | |
avoid getting 100% full. The default value of 1% matches the | |
existing default on object servers. | |
* Added an experimental `swift-ring-composer` CLI tool to build | |
composite rings. | |
* Added an optional `read_only` middleware to make an entire cluster | |
or individual accounts read only. | |
* Fixed a bug where zero-byte PUTs would not work properly | |
with "If-None-Match: *" conditional requests. | |
* ACLs now work with unicode in user/account names. | |
* COPY now works with unicode account names. | |
* Improved S3 API compatibility. | |
* Lock timeouts in the container updater are now logged at INFO | |
level, not ERROR. | |
* Various other minor bug fixes and improvements. | |
swift (2.18.0) | |
* Added container sharding, an operator controlled feature that | |
may be used to shard very large container databases into a | |
number of smaller shard containers. This mitigates the issues | |
with one large DB by distributing the data across multiple | |
smaller databases throughout the cluster. Please read the full | |
overview at | |
https://docs.openstack.org/swift/latest/overview_container_sharding.html | |
* Provide an S3 API compatibility layer. The external "swift3" | |
project has been imported into Swift's codebase as the "s3api" | |
middleware. | |
* Added "emergency mode" hooks in the account and container replicators. | |
These options may be used to prioritize moving handoff | |
partitions to primary locations more quickly. This helps when | |
adding capacity to a ring. | |
- Added `-d <devs>` and `-p <partitions>` command line options. | |
- Added a handoffs-only mode. | |
* Add a multiprocess mode to the object replicator. Setting the | |
"replicator_workers" setting to a positive value N will result | |
in the replicator using up to N worker processes to perform | |
replication tasks. At most one worker per disk will be spawned. | |
Worker process logs will have a bit of information prepended so | |
operators can tell which messages came from which worker. The | |
prefix is "[worker M/N pid=P] ", where M is the worker's index, | |
N is the total number of workers, and P is the process ID. Every | |
message from the replicator's logger will have the prefix | |
* The object reconstructor will now fork all available worker | |
processes when operating on a subset of local devices. | |
* Add support for PROXY protocol v1 to the proxy server. This | |
allows the Swift proxy server to log accurate client IP | |
addresses when there is a proxy or SSL-terminator between the | |
client and the Swift proxy server. Example servers supporting | |
this PROXY protocol include stunnel, haproxy, hitch, and | |
varnish. See the sample proxy server config file for the | |
appropriate config setting to enable or disable this | |
functionality. | |
* In the ratelimit middleware, account whitelist and blacklist | |
settings have been deprecated and may be removed in a future | |
release. When found, a deprecation message will be logged. | |
Instead of these config file values, set X-Account-Sysmeta- | |
Global-Write-Ratelimit:WHITELIST and X-Account-Sysmeta-Global- | |
Write-Ratelimit:BLACKLIST on the particular accounts that need | |
to be whitelisted or blacklisted. System metadata cannot be added | |
or modified by standard clients. Use the internal client to set sysmeta. | |
* Add a --drop-prefixes flag to swift-account-info, | |
swift-container-info, and swift-object-info. This makes the | |
output between the three more consistent. | |
* statsd error messages correspond to 5xx responses only. This | |
makes monitoring more useful because actual errors (5xx) will | |
not be hidden by common user requests (4xx). Previously, some 4xx | |
responses would be included in timing information in the statsd | |
error messages. | |
* Truncate error logs to prevent log handler from running out of buffer. | |
* Updated requirements.txt to match global exclusions and formatting. | |
* tempauth user names now support unicode characters. | |
* Various other minor bug fixes and improvements. | |
swift (2.17.1, queens stable backports) | |
* Fix SLO delete for accounts with non-ASCII names. | |
* Fixed an issue in COPY where concurrent requests may have copied the | |
wrong data. | |
* Fixed a bug in how Swift uses eventlet that was exposed under high | |
concurrency. | |
swift (2.17.0, OpenStack Queens) | |
* Added symlink objects support. | |
Symlink objects reference one other object. They are created by | |
creating an empty object with an X-Symlink-Target header. The value of | |
the header is of the format <container>/<object>, and the target does | |
not need to exist at the time of symlink creation. Cross-account | |
symlinks can be created by including the | |
X-Symlink-Target-Account header. | |
GET and HEAD requests to a symlink will operate on the | |
referenced object and require appropriate permission in the | |
target container. DELETE and PUT requests will operate on the | |
symlink object itself. POST requests are not forwarded to the | |
referenced object. POST requests sent to a symlink will result | |
in a 307 Temporary Redirect response. | |
* Added support for inline data segments in SLO manifests. | |
Upgrade impact: during a rolling upgrade, an updated proxy server | |
may write a manifest that an out-of-date proxy server will not be | |
able to read. This will resolve itself once the upgrade completes | |
on all nodes. | |
* The tempurl digest algorithm is now configurable, and Swift added | |
support for both SHA-256 and SHA-512. Supported tempurl digests | |
are exposed to clients in `/info`. Additionally, tempurl signatures | |
can now be base64 encoded. | |
* Object expiry improvements | |
- Disallow X-Delete-At header values equal to the X-Timestamp header. | |
- X-Delete-At computation now uses X-Timestamp instead of | |
system time. This prevents clock skew causing inconsistent | |
expiry data. | |
- Deleting an expiring object will now cause less work in the system. | |
The number of async pending files written has been reduced for all | |
objects and greatly reduced for erasure-coded objects. This | |
dramatically reduces the burden on container servers. | |
- Stopped logging tracebacks when receiving an unexpected response. | |
- Allow the expirer to gracefully move past updating stale work items. | |
* When the object auditor examines an object, it will now add any | |
missing metadata checksums. | |
* `swift-ring-builder` improvements | |
- Save the ring when dispersion improves, even if balance | |
doesn't improve. | |
- Improved the granularity of the ring dispersion metric so that | |
small improvements after a rebalance can show changes in the | |
dispersion number. Dispersion in existing and new rings can be | |
recalculated using the new '--recalculate' option to | |
`swift-ring-builder`. | |
- Display more info on empty rings. | |
* Fixed rare socket leak on range requests to erasure-coded objects. | |
* The number of container updates on object PUTs (ie to update listings) | |
has been recomputed to be far more efficient while maintaining | |
durability guarantees. Specifically, object PUTs to erasure-coded | |
policies will now normally result in far fewer container updates. | |
* Moved Zuul v3 tox jobs into the Swift code repo. | |
* Changed where liberasurecode-devel for CentOS 7 is referenced and | |
installed as a dependency. | |
* Added container/object listing with prefix to InternalClient. | |
* Added '--swift-versions' to `swift-recon` CLI to compare installed | |
versions in the cluster. | |
* Stop logging tracebacks in the `object-replicator` when it runs | |
out of handoff locations. | |
* Send ETag header in 206 Partial Content responses to SLO reads. | |
* Now `swift-recon-cron` works with conf.d configs. | |
* Improved `object-updater` stats logging. It now tells you all of | |
its stats (successes, failures, quarantines due to bad pickles, | |
unlinks, and errors), and it tells you incremental progress every | |
five minutes. The logging at the end of a pass remains and has | |
been expanded to also include all stats. | |
* If a proxy server is configured to autocreate accounts and the | |
account create fails, it will now return a server error (500) | |
instead of Not Found (404). | |
* Fractional replicas are no longer allowed for erasure code policies. | |
* Various other minor bug fixes and improvements. | |
swift (2.16.0) | |
* Add checksum to object extended attributes. | |
* Let clients request heartbeats during SLO PUTs by including | |
the query parameter `heartbeat=on`. | |
With heartbeating turned on, the proxy will start its response | |
immediately with 202 Accepted then send a single whitespace | |
character periodically until the request completes. At that | |
point, a final summary chunk will be sent which includes a | |
"Response Status" key indicating success or failure and (if | |
successful) an "Etag" key indicating the Etag of the resulting | |
SLO. | |
* Added support for retrieving the encryption root secret from an | |
external key management system. In practice, this is currently limited | |
to Barbican. | |
* Move listing formatting out to a new proxy middleware named | |
`listing_formats`. `listing_formats` should be just right of the | |
first proxy-logging middleware, and left of most other | |
middlewares. If it is not already present, it will be | |
automatically inserted for you. | |
Note: if you have a custom middleware that makes account or | |
container listings, it will only receive listings in JSON format. | |
* Log deprecation warning for `allow_versions` in the container | |
server config. Configure the `versioned_writes` middleware in | |
the proxy server instead. This option will be ignored in a | |
future release. | |
* Replaced `replication_one_per_device` by custom count defined by | |
`replication_concurrency_per_device`. The original config value | |
is deprecated, but continues to function for now. If both values | |
are defined, the old `replication_one_per_device` is ignored. | |
* Fixed a rare issue where multiple backend timeouts could result | |
in bad data being returned to the client. | |
* Cleaned up logged tracebacks when talking to memcached servers. | |
* Account and container replication stats logs now include | |
`remote_merges`, the number of times a whole database was sent | |
to another node. | |
* Respond 400 Bad Request when Accept headers fail to parse | |
instead of returning 406 Not Acceptable. | |
* The `domain_remap` middleware now supports the | |
`mangle_client_paths` option. Its default "false" value changes | |
`domain_remap` parsing to stop stripping the `path_root` value | |
from URL paths. If users depend on this path mangling, operators | |
should set `mangle_client_paths` to "True" before upgrading. | |
* Remove `swift-temp-url` script. The functionality has been in | |
swiftclient for a long time and this script has been deprecated | |
since 2.10.0. | |
* Removed all `post_as_copy` related code and configs. The option | |
has been deprecated since 2.13.0. | |
* Fixed XML responses (eg on bulk extractions and SLO upload | |
failures) to be more correct. The enclosing "delete" tag was | |
removed where it doesn't make sense and replaced with "extract" | |
or "upload" depending on the context. | |
* Static Large Object (SLO) manifest may now (again) have zero-byte | |
last segments. | |
* Fixed an issue where background consistency daemon child | |
processes would deadlock waiting on the same file descriptor. | |
* Removed a race condition where a POST to an SLO could modify the | |
X-Static-Large-Object metadata. | |
* Accept a trade off of dispersion for balance in the ring builder | |
that will result in getting to balanced rings much more quickly | |
in some cases. | |
* Fixed using `swift-ring-builder set_weight` with more than one | |
device. | |
* When requesting objects, return 404 if a tombstone is found and | |
is newer than any data found. Previous behavior was to return | |
stale data. | |
* Various other minor bug fixes and improvements. | |
swift (2.15.2, pike stable backports) | |
* Fixed a cache invalidation issue related to GET and PUT requests to | |
containers that would occasionally cause object PUTs to a container to | |
404 after the container had been successfully created. | |
* Removed a race condition where a POST to an SLO could modify the | |
X-Static-Large-Object metadata. | |
* Fixed rare socket leak on range requests to erasure-coded objects. | |
* Fix SLO delete for accounts with non-ASCII names. | |
* Fixed an issue in COPY where concurrent requests may have copied the | |
wrong data. | |
* Fixed time skew when using X-Delete-After. | |
* Send ETag header in 206 Partial Content responses to SLO reads. | |
swift (2.15.1, OpenStack Pike) | |
* Fixed a bug introduced in 2.15.0 where the object reconstructor | |
would exit with a traceback if no EC policy was configured. | |
* Fixed deadlock when logging from a tpool thread. | |
The object server runs certain IO-intensive methods outside the | |
main pthread for performance. Previously, if one of those methods | |
tried to log, this can cause a crash that eventually leads to an | |
object server with hundreds or thousands of greenthreads, all | |
deadlocked. The fix is to use a mutex that works across different | |
greenlets and different pthreads. | |
* The object reconstructor can now rebuild an EC fragment for an | |
expired object. | |
* Various other minor bug fixes and improvements. | |
swift (2.15.0) | |
* Add Composite Ring Functionality | |
A composite ring comprises two or more component rings that are | |
combined to form a single ring with a replica count equal to the | |
sum of the component rings. The component rings are built | |
independently, using distinct devices in distinct regions, which | |
means that the dispersion of replicas between the components can | |
be guaranteed. | |
Composite rings can be used for explicit replica placement and | |
"replicated EC" for global erasure codes policies. | |
Composite rings support 'cooperative' rebalance which means that | |
during rebalance all component rings will be consulted before a | |
partition is moved in any component ring. This avoids the same | |
partition being simultaneously moved in multiple components. | |
We do not yet have CLI tools for creating composite rings, but | |
the functionality has been enabled in the ring modules to | |
support this advanced functionality. CLI tools will be delivered | |
in a subsequent release. | |
For further information see the docs at | |
<https://docs.openstack.org/swift/latest/overview_ring.html#module-swift.common.ring.composite_builder> | |
* The EC reconstructor process has been dramatically improved by | |
adding support for multiple concurrent workers. Multiple | |
processes are required to get high concurrency, and this change | |
results in much faster rebalance times on servers with many | |
drives. | |
Currently the default is still only one process, and no workers. | |
Set `reconstructor_workers` in the `[object-reconstructor]` | |
section to some whole number <= the number of devices on a node | |
to get that many reconstructor workers. | |
* Add support to increase object ring partition power transparently | |
to end users and with no cluster downtime. Increasing the ring | |
part power allows for incremental adjustment to the upper bound | |
of the cluster size. Please review the full docs at | |
<https://docs.openstack.org/swift/latest/ring_partpower.html>. | |
* Added support for per-policy proxy config options. This allows | |
per-policy affinity options to be set for use with duplicated EC | |
policies and composite rings. Certain options found in per-policy | |
conf sections will override their equivalents that may be set | |
in the [app:proxy-server] section. Currently the options handled that | |
way are sorting_method, read_affinity, write_affinity, | |
write_affinity_node_count, and write_affinity_handoff_delete_count. | |
* Enabled versioned writes on Dynamic Large Objects (DLOs). | |
* Write-affinity aware object deletion | |
Previously, when deleting objects in multi-region swift | |
deployment with write affinity configured, users always get 404 | |
when deleting object before it's replicated to appropriate nodes. | |
Now Swift will use `write_affinity_handoff_delete_count` to | |
define how many local handoff nodes should swift send request to | |
get more candidates for the final response. The default value | |
"auto" means Swift will calculate the number automatically based | |
on the number of replicas and current cluster topology. | |
* Require that known-bad EC schemes be deprecated | |
Erasure-coded storage policies using isa_l_rs_vand and nparity | |
>= 5 must be configured as deprecated, preventing any new | |
containers from being created with such a policy. This | |
configuration is known to harm data durability. Any data in such | |
policies should be migrated to a new policy. See | |
https://bugs.launchpad.net/swift/+bug/1639691 for more | |
information | |
* Optimize the Erasure Code reconstructor protocol to reduce IO | |
load on servers. | |
* Fixed a bug where SSYNC would fail to replicate unexpired object. | |
* Fixed a bug in domain_remap when obj starts/ends with slash. | |
* Fixed a socket leak in copy middleware when a large object was copied. | |
* Fixed a few areas where the `swiftdir` option was not respected. | |
* `swift-recon` now respects storage policy aliases. | |
* cname_lookup middleware now accepts a `nameservers` config | |
variable that, if defined, will be used for DNS lookups instead of | |
the system default. | |
* Make mount_check option usable in containerized environments by | |
adding a check for an ".ismount" file at the root directory of | |
a device. | |
* Remove deprecated `vm_test_mode` option. | |
* The object and container server config option `slowdown` has been | |
deprecated in favor of the new `objects_per_second` and | |
`containers_per_second` options. | |
* The output of devices from `swift-ring-builder` has been reordered | |
by region, zone, ip, and device. | |
* Imported docs content from openstack-manuals project. | |
* Various other minor bug fixes and improvements. | |
swift (2.14.0) | |
* Fixed error where a container drive error resulted in double space | |
usage on rest drives. When drive with container or account database | |
is unmounted, the bug would create handoff replicas on all remaining | |
drives, increasing the drive space used and filling the cluster. | |
* Fixed UnicodeDecodeError in the object reconstructor that would | |
prevent objects with non-ascii names from being reconstructed and | |
caused the reconstructor process to hang. | |
* EC Fragment Duplication - Foundational Global EC Cluster Support. | |
* Fixed encoding issue in ssync where a mix of ascii and non-ascii | |
metadata values would cause an error. | |
* `name_check` and `cname_lookup` keys have been added to `/info`. | |
* Add Vary: headers for CORS responses. | |
* Always set Swift processes to use UTC. | |
* Prevent logged traceback in object-server on client disconnect for | |
chunked transfers to replicated policies. | |
* Removed per-device reconstruction stats. Now that the reconstructor | |
is shuffling parts before going through them, those stats no longer | |
make sense. | |
* Log correct status code for conditional requests. | |
* Drop support for auth-server from common/manager.py and `swift-init`. | |
* Include received fragment index in reconstructor log warnings. | |
* Fixed a race condition in updating hashes.pkl where a partition | |
suffix invalidation may have been skipped. | |
* `domain_remap` now accepts a list of domains in "storage_domain". | |
* Do not follow CNAME when host is in storage_domain. | |
* Enable cluster-wide CORS Expose-Headers setting via | |
"cors_expose_headers". | |
* Cache all answers from nameservers in cname_lookup. | |
* Log the correct request type of a subrequest downstream of copy. | |
* Various other minor bug fixes and improvements. | |
swift (2.13.0, OpenStack Ocata) | |
* Improvements in key parts of the consistency engine | |
- Improved performance by eliminating an unneeded directory | |
structure hash. | |
- Optimized the common case for hashing filesystem trees, thus | |
eliminating a lot of extraneous disk I/O. | |
- Updated the `hashes.pkl` file format to include timestamp information | |
for race detection. Also simplified hashing logic to prevent race | |
conditions and optimize for the common case. | |
- The erasure code reconstructor will now shuffle work jobs across all | |
disks instead of going disk-by-disk. This eliminates single-disk I/O | |
contention and allows continued scaling as concurrency is increased. | |
- Erasure code reconstruction handles moving data from handoff nodes | |
better. Instead of moving the data to another handoff, it waits | |
until it can be moved to a primary node. | |
Upgrade Impact: If you upgrade and roll back, you must delete all | |
`hashes.pkl` files. | |
* If using erasure coding with ISA-L in rs_vand mode and 5 or more parity | |
fragments, Swift will emit a warning. This is a configuration that is | |
known to harm data durability. In a future release, this warning will be | |
upgraded to an error unless the policy is marked as deprecated. All data | |
in an erasure code storage policy using isa_l_rs_vand with 5 or more | |
parity should be migrated as soon as possible. Please see | |
https://bugs.launchpad.net/swift/+bug/1639691 for more information. | |
* The erasure code reconstructor `handoffs_first` option has been | |
deprecated in favor of `handoffs_only`. `handoffs_only` is far more | |
useful, and just like `handoffs_first` mode in the replicator, it gives | |
the operator the option of forcing the consistency engine to focus | |
solely on revert (handoff) jobs, thus improving the speed of | |
rebalances. The `handoffs_only` behavior is somewhat consistent with | |
the replicator's `handoffs_first` option (any error on any handoff in | |
the replicator will make it essentially handoff only forever) but the | |
`handoff_only` option does what you want and is named correctly in the | |
reconstructor. | |
* The default for `object_post_as_copy` has been changed to False. The | |
option is now deprecated and will be removed in a future release. If | |
your cluster is still running with post-as-copy enabled, please update | |
it to use the "fast-post" method. Future versions of Swift will not | |
support post-as-copy, and future features will not be supported under | |
post-as-copy. ("Fast-post" is where `object_post_as_copy` is false). | |
* Temporary URLs now support one common form of ISO 8601 timestamps in | |
addition to Unix seconds-since-epoch timestamps. The ISO 8601 format | |
accepted is '%Y-%m-%dT%H:%M:%SZ'. This makes TempURLs more | |
user-friendly to produce and consume. | |
* Listing containers in accounts with json or xml now includes a | |
`last_modified` time. This does not change any on-disk data, but simply | |
exposes the value to offer consistency with the object listings on | |
containers. | |
* Fixed a bug where the ring builder would not allow removal of a device | |
when min_part_seconds_left was greater than zero. | |
* PUT subrequests generated from a client-side COPY will now properly log | |
the SSC (server-side copy) Swift source field. See | |
https://docs.openstack.org/swift/latest/logs.html#swift-source for | |
more information. | |
* Fixed a bug where an SLO download with a range request may have resulted | |
in a 5xx series response. | |
* SLO manifest PUT requests can now be properly validated by sending an | |
ETag header of the md5 sum of the concatenated md5 sums of the | |
referenced segments. | |
* Fixed the stats calculation in the erasure code reconstructor. | |
* Rings with min_part_hours set to zero will now only move one partition | |
replica per rebalance, thus matching behavior when min_part_hours is | |
greater than zero. | |
* I/O priority is now supported on AArch64 architecture. | |
* Various other minor bug fixes and improvements. | |
swift (2.12.0) | |
* Ring files now include byteorder information about the endian of | |
the machine used to generate the file, and the values are | |
appropriately byteswapped if deserialized on a machine with a | |
different endianness. | |
Newly created ring files will be byteorder agnostic, but | |
previously generated ring files will still fail on different | |
endian architectures. Regenerating older ring files will cause | |
them to become byteorder agnostic. The regeneration of the ring | |
files will not cause any new data movement. Newer ring files | |
will still be usable by older versions of Swift (on machines | |
with the same endianness--this maintains existing behavior). | |
* All 416 responses will now include a Content-Range header with | |
an unsatisfied-range value. This allows the caller to know the | |
valid range request value for an object. | |
* TempURLs now support a validation against a common prefix. A | |
prefix-based signature grants access to all objects which share the | |
same prefix. This avoids the creation of a large amount of signatures, | |
when a whole container or pseudofolder is shared. | |
* Correctly handle deleted files with if-none-match requests. | |
* Correctly send 412 Precondition Failed if a user sends an | |
invalid copy destination. Previously Swift would send a 500 | |
Internal Server Error. | |
* In SLO manifests, the `etag` and `size_bytes` keys are now fully | |
optional and not required. Previously, the keys needed to exist | |
but the values were optional. The only required key is `path`. | |
* Fixed a rare infinite loop in `swift-ring-builder` while placing parts. | |
* Ensure update of the container by object-updater, removing a rare | |
possibility that objects would never be added to a container listing. | |
* Fixed non-deterministic suffix updates in hashes.pkl where a partition | |
may be updated much less often than expected. | |
* Fixed regression in consolidate_hashes that occurred when a new | |
file was stored to new suffix to a non-empty partition. This bug | |
was introduced in 2.7.0 and could cause an increase in rsync | |
replication stats during and after upgrade, due to inconsistent | |
hashing of partition suffixes. | |
* Account and container databases will now be quarantined if the | |
database schema has been corrupted. | |
* Removed "in-process-" from func env tox name to work with | |
upstream CI. | |
* Respect server type for --md5 check in swift-recon. | |
* Remove empty db hash and suffix directories if a db gets quarantined. | |
* Various other minor bug fixes and improvements. | |
swift (2.11.0) | |
* We have made significant improvements and changes to the erasure | |
code implementation. | |
- Instead of using a separate .durable file to indicate the | |
durable status of an EC fragment archive, we rename the .data | |
to include a durable marker in the filename. This saves one | |
inode for every EC .data file. Existing .durable files will not | |
be removed, and they will continue to work just fine. | |
Note that after writing EC data with Swift 2.11.0 or later, that | |
data will not be accessible to earlier versions of Swift. | |
- Closed a bug where ssync may have written bad fragment data in | |
some circumstances. A check was added to ensure the correct number | |
of bytes is written for a fragment before finalizing the write. | |
Also, erasure coded fragment metadata will now be validated on read | |
requests and, if bad data is found, the fragment will be quarantined. | |
- The improvements to EC reads made in Swift 2.10.0 have also been | |
applied to the reconstructor. This allows fragments to be rebuilt | |
in more circumstances, resulting in faster recovery from failures. | |
- WARNING: If you are using the ISA-L library for erasure codes, | |
please upgrade to liberasurecode 1.3.1 (or later) as soon as | |
possible. If you are using isa_l_rs_vand with more than 4 parity, | |
please read https://bugs.launchpad.net/swift/+bug/1639691 and take | |
necessary action. | |
- Updated the PyECLib dependency to 1.3.1. | |
* Added a configurable URL base to staticweb. | |
* Support multi-range GETs for static large objects. | |
* TempURLs using the "inline" parameter can now also set the | |
"filename" parameter. Both are used in the Content-Disposition | |
response header. | |
* Mirror X-Trans-Id to X-Openstack-Request-Id. | |
* SLO will now concurrently HEAD segments, resulting in much faster | |
manifest validation and object creation. By default, two HEAD requests | |
will be done at a time, but this can be changed by the operator via | |
the new `concurrency` setting in the "[filter:slo]" section of | |
the proxy server config. | |
* Suppressed the KeyError message when auditor finds an expired object. | |
* Daemons using InternalClient can now be properly killed with SIGTERM. | |
* Added a "user" option to the drive-audit config file. Its value is | |
used to set the owner of the drive-audit recon cache. | |
* Throttle update_auditor_status calls so it updates no more than once | |
per minute. | |
* Suppress unexpected-file warnings for rsync temp files. | |
* Various other minor bug fixes and improvements. | |
swift (2.10.0, OpenStack Newton) | |
* Object versioning now supports a "history" mode in addition to | |
the older "stack" mode. The difference is in how DELETE requests | |
are handled. For full details, please read | |
https://docs.openstack.org/swift/latest/overview_object_versioning.html. | |
* New config variables to change the schedule priority and I/O | |
scheduling class. Servers and daemons now understand | |
`nice_priority`, `ionice_class`, and `ionice_priority` to | |
schedule their relative importance. Please read | |
https://docs.openstack.org/swift/latest/admin_guide.html | |
for full config details. | |
* On newer kernels (3.15+ when using xfs), Swift will use the O_TMPFILE | |
flag when opening a file instead of creating a temporary file | |
and renaming it on commit. This makes the data path simpler and | |
allows the filesystem to more efficiently optimize the files on | |
disk, resulting in better performance. | |
* Erasure code GET performance has been significantly | |
improved in clusters that are not completely healthy. | |
* Significant improvements to the api-ref doc available at | |
https://docs.openstack.org/api-ref/object-store/. | |
* A PUT or POST to a container will now update the container's | |
Last-Modified time, and that value will be included in a | |
GET/HEAD response. | |
* Include object sysmeta in POST responses. Sysmeta is still | |
stripped from the response before being sent to the client, but | |
this allows middleware to make use of the information. | |
* Fixed a bug where a container listing delimiter wouldn't work | |
with encryption. | |
* Fixed a bug where some headers weren't being copied correctly | |
in a COPY request. | |
* Container sync can now copy SLOs more efficiently by allowing | |
the manifest to be synced before all of the referenced segments. | |
This fixes a bug where container sync would not copy SLO manifests. | |
* Fixed a bug where some tombstone files might never be reclaimed. | |
* Update dnspython dependency to 1.14, removing the need to have | |
separate dnspython dependencies for Py2 and Py3. | |
* Deprecate swift-temp-url and call python-swiftclient's | |
implementation instead. This adds python-swiftclient as an | |
optional dependency of Swift. | |
* Moved other-requirements.txt to bindep.txt. bindep.txt lists | |
non-python dependencies of Swift. | |
* Various other minor bug fixes and improvements. | |
swift (2.9.0) | |
* Swift now supports at-rest encryption. This feature encrypts all | |
object data and user-set object metadata as it is sent to the cluster. | |
This feature is designed to prevent information leaks if a hard drive | |
leaves the cluster. The encryption is transparent to the end-user. | |
At-rest encryption in Swift is enabled on the proxy server by | |
adding two middlewares to the pipeline. The `keymaster` middleware | |
is responsible for managing the encryption keys and the `encryption` | |
middleware does the actual encryption and decryption. | |
Existing clusters will continue to work without enabling | |
encryption. Although enabling this feature on existing clusters | |
is supported, best practice is to enable this feature on new | |
clusters when the cluster is created. | |
For more information on the details of the at-rest encryption | |
feature, please see the docs at | |
https://docs.openstack.org/swift/latest/overview_encryption.html. | |
* `swift-recon` can now be called with more than one server type. | |
* Fixed a bug where non-ascii names could cause an error in logging | |
and cause a 5xx response to the client. | |
* The install guide and API reference have been moved into Swift's | |
source code repository. | |
* Various other minor bug fixes and improvements. | |
swift (2.8.0) | |
* Allow concurrent bulk deletes for server-side deletes of static | |
large objects. Previously this would be single-threaded and each | |
DELETE executed serially. The new `delete_concurrency` value | |
(default value is 2) in the `[filter:slo]` and `[filter:bulk]` | |
sections of the proxy server config controls the concurrency | |
used to perform the DELETE requests for referenced segments. The | |
default value is recommended, but setting the value to 1 | |
restores previous behavior. | |
* Refactor server-side copy as middleware | |
The COPY verb is now implemented in the `copy` middleware instead | |
of in the proxy server code. If not explicitly added, the server | |
side copy middleware is auto-inserted to the left of `dlo`, `slo` | |
and `versioned_writes` middlewares in the proxy server pipeline. | |
As a result, dlo and slo `copy_hooks` are no longer required. SLO | |
manifests are now validated when copied so when copying a | |
manifest to another account the referenced segments must be | |
readable in that account for the manifest copy to succeed | |
(previously this validation was not made, meaning the manifest | |
was copied but could be unusable if the segments were not | |
readable). | |
With this change, there should be no change in functionality or | |
existing behavior. | |
* `fallocate_reserve` can now be a percentage (a value ending in "%"), | |
and the default has been adjusted to "1%". | |
* Now properly require account/container metadata be valid UTF-8 | |
* TempURL responses now include an `Expires` header with the | |
expiration time embedded in the URL. | |
* Non-Python dependencies are now listed in other-requirements.txt. | |
* `swift-ring-builder` now supports a `--yes` option to assume a | |
yes response to all questions. This is useful for scripts. | |
* Write requests to a replicated storage policy with an even number | |
of replicas now have a quorum size of half the replica count | |
instead of half-plus-one. | |
* Container sync now logs per-container stat information so operators | |
can track progress. This is logged at INFO level. | |
* `swift-dispersion-*` now allows region to be specified when there | |
are multiple Swift regions served by the same Keystone instance | |
* Fix infinite recursion during logging when syslog is down. | |
* Fixed a bug where a backend failure during a read could result in | |
a missing byte in the response body. | |
* Stop `staticweb` revealing container existence to unauth'd requests. | |
* Reclaim isolated .meta files if they are older than the `reclaim_age`. | |
* Make `rsync` ignore its own temporary files instead of spreading | |
them around the cluster, wasting space. | |
* The object auditor now ignores files in the devices directory when | |
auditing objects. | |
* The deprecated `threads_per_disk` setting has been removed. Deployers | |
are encouraged to use `servers_per_port` instead. | |
* Fixed an issue where a single-replica configuration for account or | |
container DBs could result in the DB being inadvertently deleted if | |
it was placed on a handoff node. | |
* `disable_fallocate` now also correctly disables `fallocate_reserve`. | |
* Fixed a bug where the account-reaper did not delete all containers | |
in a reaped account. | |
* Correctly handle delimiter queries where results start with the | |
delimiter and no prefix is given. | |
* Changed the recommended ports for Swift services from ports | |
6000-6002 to unused ports 6200-6202 so they do not conflict with | |
X-Windows or other services. Since these config values must be | |
explicitly set in the config file, this doesn't impact existing | |
deployments. | |
* Fixed an instance where REPLICATE requests would not use | |
`replication_ip`. | |
* Various other minor bug fixes and improvements. | |
swift (2.7.0, OpenStack Mitaka) | |
* Bump PyECLib requirement to >= 1.2.0 | |
* Update container on fast-POST | |
"Fast-POST" is the mode where `object_post_as_copy` is set to | |
`False` in the proxy server config. This mode now allows for | |
fast, efficient updates of metadata without needing to fully | |
recopy the contents of the object. While the default still is | |
`object_post_as_copy` as True, the plan is to change the default | |
to False and then deprecate post-as-copy functionality in later | |
releases. Fast-POST now supports container-sync functionality. | |
* Add concurrent reads option to proxy. | |
This change adds 2 new parameters to enable and control concurrent | |
GETs in Swift, these are `concurrent_gets` and `concurrency_timeout`. | |
`concurrent_gets` allows you to turn on or off concurrent | |
GETs; when on, it will set the GET/HEAD concurrency to the | |
replica count. And in the case of EC HEADs it will set it to | |
ndata. The proxy will then serve only the first valid source to | |
respond. This applies to all account, container, and replicated | |
object GETs and HEADs. For EC only HEAD requests are affected. | |
The default for `concurrent_gets` is off. | |
`concurrency_timeout` is related to `concurrent_gets` and is | |
the amount of time to wait before firing the next thread. A | |
value of 0 will fire at the same time (fully concurrent), but | |
setting another value will stagger the firing allowing you the | |
ability to give a node a short chance to respond before firing | |
the next. This value is a float and should be somewhere between | |
0 and `node_timeout`. The default is `conn_timeout`, meaning by | |
default it will stagger the firing. | |
* Added an operational procedures guide to the docs. It can be | |
found at https://docs.openstack.org/swift/latest/ops_runbook/index.html and | |
includes information on detecting and handling day-to-day | |
operational issues in a Swift cluster. | |
* Make `handoffs_first` a more useful mode for the object replicator. | |
The `handoffs_first` replication mode is used during periods of | |
problematic cluster behavior (e.g. full disks) when replication | |
needs to quickly drain partitions from a handoff node and move | |
them to a primary node. | |
Previously, `handoffs_first` would sort that handoff work before | |
"normal" replication jobs, but the normal replication work could | |
take quite some time and result in handoffs not being drained | |
quickly enough. | |
In order to focus on getting handoff partitions off the node | |
`handoffs_first` mode will now abort the current replication | |
sweep before attempting any primary suffix syncing if any of the | |
handoff partitions were not removed for any reason - and start | |
over with replication of handoffs jobs as the highest priority. | |
Note that `handoffs_first` being enabled will emit a warning on | |
start up, even if no handoff jobs fail, because of the negative | |
impact it can have during normal operations by dog-piling on a | |
node that was temporarily unavailable. | |
* By default, inbound `X-Timestamp` headers are now disallowed | |
(except when in an authorized container-sync request). This | |
header is useful for allowing data migration from other storage | |
systems to Swift and keeping the original timestamp of the data. | |
If you have this migration use case (or any other requirement on | |
allowing the clients to set an object's timestamp), set the | |
`shunt_inbound_x_timestamp` config variable to False in the | |
gatekeeper middleware config section of the proxy server config. | |
* Requesting a SLO manifest file with the query parameters | |
"?multipart-manifest=get&format=raw" will return the contents of | |
the manifest in the format as was originally sent by the client. | |
The "format=raw" is new. | |
* Static web page listings can now be rendered with a custom | |
label. By default listings are rendered with a label of: | |
"Listing of /v1/<account>/<container>/<path>". This change adds | |
a new custom metadata key/value pair | |
`X-Container-Meta-Web-Listings-Label: My Label` that when set, | |
will cause the following: "Listing of My Label/<path>" to be | |
rendered instead. | |
* Previously, static large objects (SLOs) had a minimum segment | |
size (default to 1MiB). This limit has been removed, but small | |
segments will be ratelimited. The config parameter | |
`rate_limit_under_size` controls the definition of "small" | |
segments (1MiB by default), and `rate_limit_segments_per_sec` | |
controls how many segments per second can be served (default is 1). | |
With the default values, the effective behavior is identical to the | |
previous behavior when serving SLOs. | |
* Container sync has been improved to perform a HEAD on the remote | |
side of the sync for each object being synced. If the object | |
exists on the remote side, container-sync will no longer | |
transfer the object, thus significantly lowering the network | |
requirements to use the feature. | |
* The object auditor will now clean up any old, stale rsync temp | |
files that it finds. These rsync temp files are left if the | |
rsync process fails without completing a full transfer of an | |
object. Since these files can be large, the temp files may end | |
up filling a disk. The new auditor functionality will reap these | |
rsync temp files if they are old. The new object-auditor config | |
variable `rsync_tempfile_timeout` is the number of seconds old a | |
tempfile must be before it is reaped. By default, this variable | |
is set to "auto" or the rsync_timeout plus 900 seconds (falling | |
back to a value of 1 day). | |
* The Erasure Code reconstruction process has been made more | |
efficient by not syncing data files when only the durable commit | |
file is missing. | |
* Fixed a bug where 304 and 416 response may not have the right | |
Etag and Accept-Ranges headers when the object is stored in an | |
Erasure Coded policy. | |
* Versioned writes now correctly stores the date of previous versions | |
using GMT instead of local time. | |
* The deprecated Keystone middleware option is_admin has been removed. | |
* Fixed log format in object auditor. | |
* The zero-byte mode (ZBF) of the object auditor will now properly | |
observe the `--once` option. | |
* Swift keeps track, internally, of "dirty" parts of the partition | |
keyspace with a "hashes.pkl" file. Operations on this file no | |
longer require a read-modify-write cycle and use a new | |
"hashes.invalid" file to track dirty partitions. This change | |
will improve end-user performance for PUT and DELETE operations. | |
* The object replicator's succeeded and failed counts are now logged. | |
* `swift-recon` can now query hosts by storage policy. | |
* The log_statsd_host value can now be an IPv6 address or a hostname | |
which only resolves to an IPv6 address. | |
* Erasure coded fragments now properly call fallocate to reserve disk | |
space before being written. | |
* Various other minor bug fixes and improvements. | |
swift (2.6.0) | |
* Dependency changes | |
- Updated minimum version of eventlet to 0.17.4 to support IPv6. | |
- Updated the minimum version of PyECLib to 1.0.7. | |
* The ring rebalancing algorithm was updated to better handle edge cases | |
and to give better (more balanced) rings in the general case. New rings | |
will have better initial placement, capacity adjustments will move less | |
data for better balance, and existing rings that were imbalanced should | |
start to become better balanced as they go through rebalance cycles. | |
* Added container and account reverse listings. | |
A GET request to an account or container resource with a "reverse=true" | |
query parameter will return the listing in reverse order. When | |
iterating over pages of reverse listings, the relative order of marker | |
and end_marker are swapped. | |
* Storage policies now support having more than one name. | |
This allows operators to fix a typo without breaking existing clients, | |
or, alternatively, have "short names" for policies. This is implemented | |
with the "aliases" config key in the storage policy config in | |
swift.conf. The aliases value is a list of names that the storage | |
policy may also be identified by. The storage policy "name" is used to | |
report the policy to users (eg in container headers). The aliases have | |
the same naming restrictions as the policy's primary name. | |
* The object auditor learned the "interval" config value to control the | |
time between each audit pass. | |
* `swift-recon --all` now includes the config checksum check. | |
* `swift-init` learned the --kill-after-timeout option to force a service | |
to quit (SIGKILL) after a designated time. | |
* `swift-recon` now correctly shows timestamps in UTC instead of local | |
time. | |
* Fixed bug where `swift-ring-builder` couldn't select device id 0. | |
* Documented the previously undocumented | |
`swift-ring-builder pretend_min_part_hours_passed` command. | |
* The "node_timeout" config value now accepts decimal values. | |
* `swift-ring-builder` now properly removes devices with zero weight. | |
* `swift-init` return codes are updated via "--strict" and "--non-strict" | |
options. Please see the usage string for more information. | |
* `swift-ring-builder` now reports the min_part_hours lockout time | |
remaining | |
* Container sync has been improved to more quickly find and iterate over | |
the containers to be synced. This reduced server load and lowers the | |
time required to see data propagate between two clusters. Please see | |
https://docs.openstack.org/swift/latest/overview_container_sync.html for more details | |
about the new on-disk structure for tracking synchronized containers. | |
* A container POST will now update that container's put-timestamp value. | |
* TempURL header restrictions are now exposed in /info. | |
* Error messages on static large object manifest responses have been | |
greatly improved. | |
* Closed a bug where an unfinished read of a large object would leak a | |
socket file descriptor and a small amount of memory. (CVE-2016-0738) | |
* Fixed an issue where a zero-byte object PUT with an incorrect Etag | |
would return a 503. | |
* Fixed an error when a static large object manifest references the same | |
object more than once. | |
* Improved performance of finding handoff nodes if a zone is empty. | |
* Fixed duplication of headers in Access-Control-Expose-Headers on CORS | |
requests. | |
* Fixed handling of IPv6 connections to memcache pools. | |
* Continued work towards python 3 compatibility. | |
* Various other minor bug fixes and improvements. | |
swift (2.5.0, OpenStack Liberty) | |
* Added the ability to specify ranges for Static Large Object (SLO) | |
segments. | |
* Replicator configs now support an "rsync_module" value to allow | |
for per-device rsync modules. This setting gives operators the | |
ability to fine-tune replication traffic in a Swift cluster and | |
isolate replication disk IO to a particular device. Please see | |
the docs and sample config files for more information and | |
examples. | |
* Significant work has gone in to testing, fixing, and validating | |
Swift's erasure code support at different scales. | |
* Swift now emits StatsD metrics on a per-policy basis. | |
* Fixed an issue with Keystone integration where a COPY request to a | |
service account may have succeeded even if a service token was not | |
included in the request. | |
* Ring validation now warns if a placement partition gets assigned to the | |
same device multiple times. This happens when devices in the ring are | |
unbalanced (e.g. two servers where one server has significantly more | |
available capacity). | |
* Various other minor bug fixes and improvements. | |
swift (2.4.0) | |
* Dependency changes | |
- Added six requirement. This is part of an ongoing effort to add | |
support for Python 3. | |
- Dropped support for Python 2.6. | |
* Config changes | |
- Recent versions of Python restrict the number of headers allowed in a | |
request to 100. This number may be too low for custom middleware. The | |
new "extra_header_count" config value in swift.conf can be used to | |
increase the number of headers allowed. | |
- Renamed "run_pause" setting to "interval" (current configs with | |
run_pause still work). Future versions of Swift may remove the | |
run_pause setting. | |
* Versioned writes middleware | |
The versioned writes feature has been refactored and reimplemented as | |
middleware. You should explicitly add the versioned_writes middleware to | |
your proxy pipeline, but do not remove or disable the existing container | |
server config setting ("allow_versions"), if it is currently enabled. | |
The existing container server config setting enables existing | |
containers to continue being versioned. Please see | |
https://docs.openstack.org/swift/latest/middleware.html#how-to-enable-object-versioning-in-a-swift-cluster | |
for further upgrade notes. | |
* Allow 1+ object-servers-per-disk deployment | |
Enabled by a new > 0 integer config value, "servers_per_port" in the | |
[DEFAULT] config section for object-server and/or replication server | |
configs. The setting's integer value determines how many different | |
object-server workers handle requests for any single unique local port | |
in the ring. In this mode, the parent swift-object-server process | |
continues to run as the original user (i.e. root if low-port binding | |
is required), binds to all ports as defined in the ring, and forks off | |
the specified number of workers per listen socket. The child, per-port | |
servers drop privileges and behave pretty much how object-server workers | |
always have, except that because the ring has unique ports per disk, the | |
object-servers will only be handling requests for a single disk. The | |
parent process detects dead servers and restarts them (with the correct | |
listen socket), starts missing servers when an updated ring file is | |
found with a device on the server with a new port, and kills extraneous | |
servers when their port is found to no longer be in the ring. The ring | |
files are stat'ed at most every "ring_check_interval" seconds, as | |
configured in the object-server config (same default of 15s). | |
In testing, this deployment configuration (with a value of 3) lowers | |
request latency, improves requests per second, and isolates slow disk | |
IO as compared to the existing "workers" setting. To use this, each | |
device must be added to the ring using a different port. | |
* Do container listing updates in another (green)thread | |
The object server has learned the "container_update_timeout" setting | |
(with a default of 1 second). This value is the number of seconds that | |
the object server will wait for the container server to update the | |
listing before returning the status of the object PUT operation. | |
Previously, the object server would wait up to 3 seconds for the | |
container server response. The new behavior dramatically lowers object | |
PUT latency when container servers in the cluster are busy (e.g. when | |
the container is very large). Setting the value too low may result in a | |
client PUT'ing an object and not being able to immediately find it in | |
listings. Setting it too high will increase latency for clients when | |
container servers are busy. | |
* TempURL fixes (closes CVE-2015-5223) | |
Do not allow PUT tempurls to create pointers to other data. | |
Specifically, disallow the creation of DLO object manifests via a PUT | |
tempurl. This prevents discoverability attacks which can use any PUT | |
tempurl to probe for private data by creating a DLO object manifest and | |
then using the PUT tempurl to head the object. | |
* Ring changes | |
- Partition placement no longer uses the port number to place | |
partitions. This improves dispersion in small clusters running one | |
object server per drive, and it does not affect dispersion in | |
clusters running one object server per server. | |
- Added ring-builder-analyzer tool to more easily test and analyze a | |
series of ring management operations. | |
- Stop moving partitions unnecessarily when overload is on. | |
* Significant improvements and bug fixes have been made to erasure code | |
support. This feature is suitable for beta testing, but it is not yet | |
ready for broad production usage. | |
* Bulk upload now treats user xattrs on files in the given archive as | |
object metadata on the resulting created objects. | |
* Emit warning log in object replicator if "handoffs_first" or | |
"handoff_delete" is set. | |
* Enable object replicator's failure count in swift-recon. | |
* Added storage policy support to dispersion tools. | |
* Support keystone v3 domains in swift-dispersion. | |
* Added domain_remap information to the /info endpoint. | |
* Added support for a "default_reseller_prefix" in domain_remap | |
middleware config. | |
* Allow SLO PUTs to forgo per-segment integrity checks. Previously, each | |
segment referenced in the manifest also needed the correct etag and | |
bytes setting. These fields now allow the "null" value to skip those | |
particular checks on the given segment. | |
* Allow rsync to use compression via a "rsync_compress" config. If set to | |
true, compression is only enabled for an rsync to a device in a | |
different region. In some cases, this can speed up cross-region | |
replication data transfer. | |
* Added time synchronization check in swift-recon (the --time option). | |
* The account reaper now runs faster on large accounts. | |
* Various other minor bug fixes and improvements. | |
swift (2.3.0, OpenStack Kilo) | |
* Erasure Code support (beta) | |
Swift now supports an erasure-code (EC) storage policy type. This allows | |
deployers to achieve very high durability with less raw capacity as used | |
in replicated storage. However, EC requires more CPU and network | |
resources, so it is not good for every use case. EC is great for storing | |
large, infrequently accessed data in a single region. | |
Swift's implementation of erasure codes is meant to be transparent to | |
end users. There is no API difference between replicated storage and | |
EC storage. | |
To support erasure codes, Swift now depends on PyECLib and | |
liberasurecode. liberasurecode is a pluggable library that allows for | |
the actual EC algorithm to be implemented in a library of your choosing. | |
As a beta release, EC support is nearly fully feature complete, but it | |
is lacking support for some features (like multi-range reads) and has | |
not had a full performance characterization. This feature relies on | |
ssync for durability. Deployers are urged to do extensive testing and | |
not deploy production data using an erasure code storage policy. | |
Full docs are at https://docs.openstack.org/swift/latest/overview_erasure_code.html | |
* Add support for container TempURL Keys. | |
* Make more memcache options configurable. connection_timeout, | |
pool_timeout, tries, and io_timeout are all now configurable. | |
* Swift now supports composite tokens. This allows another service to | |
act on behalf of a user, but only with that user's consent. | |
See https://docs.openstack.org/swift/latest/overview_auth.html for more details. | |
* Multi-region replication was improved. When replicating data to a | |
different region, only one replica will be pushed per replication | |
cycle. This gives the remote region a chance to replicate the data | |
locally instead of pushing more data over the inter-region network. | |
* Internal requests from the ratelimit middleware now properly log a | |
swift_source. See https://docs.openstack.org/swift/latest/logs.html for details. | |
* Improved storage policy support for quarantine stats in swift-recon. | |
* The proxy log line now includes the request's storage policy index. | |
* Ring checker has been added to swift-recon to validate if rings are | |
built correctly. As part of this feature, storage servers have learned | |
the OPTIONS verb. | |
* Add support of x-remove- headers for container-sync. | |
* Rings now support hostnames instead of just IP addresses. | |
* Swift now enforces that the API version on a request is valid. Valid | |
versions are configured via the valid_api_versions setting in swift.conf | |
* Various other minor bug fixes and improvements. | |
swift (2.2.2) | |
* Data placement changes | |
This release has several major changes to data placement in Swift in | |
order to better handle different deployment patterns. First, with an | |
unbalance-able ring, less partitions will move if the movement doesn't | |
result in any better dispersion across failure domains. Also, empty | |
(partition weight of zero) devices will no longer keep partitions after | |
rebalancing when there is an unbalance-able ring. | |
Second, the notion of "overload" has been added to Swift's rings. This | |
allows devices to take some extra partitions (more than would normally | |
be allowed by the device weight) so that smaller and unbalanced clusters | |
will have less data movement between servers, zones, or regions if there | |
is a failure in the cluster. | |
Finally, rings have a new metric called "dispersion". This is the | |
percentage of partitions in the ring that have too many replicas in a | |
particular failure domain. For example, if you have three servers in a | |
cluster but two replicas for a partition get placed onto the same | |
server, that partition will count towards the dispersion metric. A | |
lower value is better, and the value can be used to find the proper | |
value for "overload". | |
The overload and dispersion metrics have been exposed in the | |
swift-ring-build CLI tools. | |
See https://docs.openstack.org/swift/latest/overview_ring.html | |
for more info on how data placement works now. | |
* Improve replication of large out-of-sync, out-of-date containers. | |
* Added console logging to swift-drive-audit with a new log_to_console | |
config option (default False). | |
* Optimize replication when a device and/or partition is specified. | |
* Fix dynamic large object manifests getting versioned. This was not | |
intended and did not work. Now it is properly prevented. | |
* Fix the GET's response code when there is a missing segment in a | |
large object manifest. | |
* Change black/white listing in ratelimit middleware to use sysmeta. | |
Instead of using the config option, operators can set | |
"X-Account-Sysmeta-Global-Write-Ratelimit: WHITELIST" or | |
"X-Account-Sysmeta-Global-Write-Ratelimit: BLACKLIST" on an account to | |
whitelist or blacklist it for ratelimiting. Note: the existing | |
config options continue to work. | |
* Use TCP_NODELAY on outgoing connections. | |
* Improve object-replicator startup time. | |
* Implement OPTIONS verb for storage nodes. | |
* Various other minor bug fixes and improvements. | |
swift (2.2.1) | |
* Swift now rejects object names with Unicode surrogates. | |
* Return 403 (instead of 413) on unauthorized upload when over account | |
quota. | |
* Fix a rare condition when a rebalance could cause swift-ring-builder | |
to crash. This would only happen on old ring files when "rebalance" | |
was the first command run. | |
* Storage node error limits now survive a ring reload. | |
* Speed up reading and writing xattrs for object metadata by using larger | |
xattr value sizes. The change is moving from 254 byte values to 64KiB | |
values. There is no migration issue with this. | |
* Deleted containers beyond the reclaim age are now properly reclaimed. | |
* Full Simplified Chinese translation (zh_CN locale) for errors and logs. | |
* Container quota is now properly enforced during cross-account COPY. | |
* ssync replication now properly uses the configured replication_ip. | |
* Fixed issue were ssync did not replicate custom object headers. | |
* swift-drive-audit now has the 'unmount_failed_device' config option | |
(default to True) that controls if the process will unmount failed | |
drives or not. | |
* swift-drive-audit will now dump drive error rates to a recon file. | |
The file location is controlled by the 'recon_cache_path' config value | |
and it includes each drive and its associated number of errors. | |
* When a filesystem does't support xattr, the object server now returns | |
a 507 Insufficient Storage error to the proxy server. | |
* Clean up empty account and container partitions directories if they | |
are empty. This keeps the system healthy and prevents a large number | |
of empty directories from slowing down the replication process. | |
* Show the sum of every policy's amount of async pendings in swift-recon. | |
* Various other minor bug fixes and improvements. | |
swift (2.2.0, OpenStack Juno) | |
* Added support for Keystone v3 auth. | |
Keystone v3 introduced the concept of "domains" and user names | |
are no longer unique across domains. Swift's Keystone integration | |
now requires that ACLs be set on IDs, which are unique across | |
domains, and further restricts setting new ACLs to only use IDs. | |
Please see https://docs.openstack.org/swift/latest/overview_auth.html for | |
more information on configuring Swift and Keystone together. | |
* Swift now supports server-side account-to-account copy. Server- | |
side copy in Swift requires the X-Copy-From header (on a PUT) | |
or the Destination header (on a COPY). To initiate an account-to- | |
account copy, the existing header value remains the same, but the | |
X-Copy-From-Account header (on a PUT) or the Destination-Account | |
(on a COPY) are used to indicate the proper account. | |
* Limit partition movement when adding a new placement tier. | |
When adding a new placement tier (server, zone, or region), Swift | |
previously attempted to move all placement partitions, regardless | |
of the space available on the new tier, to ensure the best possible | |
durability. Unfortunately, this could result in too many partitions | |
being moved all at once to a new tier. Swift's ring-builder now | |
ensures that only the correct number of placement partitions are | |
rebalanced, and thus makes adding capacity to the cluster more | |
efficient. | |
* Per storage policy container counts are now reported in an | |
account response headers. | |
* Swift will now reject, with a 4xx series response, GET requests | |
with more than 50 ranges, more than 3 overlapping ranges, or more | |
than 8 non-increasing ranges. | |
* The bind_port config setting is now required to be explicitly set. | |
* The object server can now use splice() for a zero-copy GET | |
response. This feature is enabled with the "splice" config variable | |
in the object server config and defaults to off. Also, this feature | |
only works on recent Linux kernels (AF_ALG sockets must be | |
supported). A zero-copy GET response can significantly reduce CPU | |
requirements for object servers. | |
* Added "--no-overlap" option to swift-dispersion populate so that | |
multiple runs of the tool can add coverage without overlapping | |
existing monitored partitions. | |
* swift-recon now supports filtering by region. | |
* Various other minor bug fixes and improvements. | |
swift (2.1.0) | |
* swift-ring-builder placement was improved to allow gradual addition | |
of new regions without causing a massive migration of data to the new | |
region. The change was to prefer device weight first, then look at | |
failure domains. | |
* Logging updates | |
- Eliminated "Handoff requested (N)" log spam. | |
- Added process pid to the end of storage node log lines. | |
- Container auditor now logs a warning if the devices path contains a | |
non-directory. | |
- Object daemons now send a user-agent string with their full name. | |
* 412 and 416 responses are no longer tracked as errors in the StatsD | |
messages from the backend servers. | |
* Parallel object auditor | |
The object auditor can now be controlled with a "concurrency" config | |
value that allows multiple auditor processes to run at once. Using | |
multiple parallel auditor processes can speed up the overall auditor | |
cycle time. | |
* The object updater will now concurrently update each necessary node | |
in a new greenthread. | |
* TempURL updates | |
- The default allowed methods have changed to also allow POST and | |
DELETE. The new default list is "GET HEAD PUT POST DELETE". | |
- TempURLs for POST now also allow HEAD, matching existing GET and PUT | |
functionality. | |
- Added filename*= support to TempURL Content-Disposition response | |
header. | |
* X-Delete-At/After can now be used with the FormPost middleware. | |
* Make swift-form-signature output a sample form. | |
* Add v2 API to list endpoints middleware | |
The new API adds better support for storage policies and changes the | |
response from a list of backend urls to a dictionary with the keys | |
"endpoints" and "headers". The endpoints key contains a list of the | |
backend urls, and the headers key is a dictionary of headers to send | |
along with the backend request. | |
* Added allow_account_management and account_autocreate values to /info | |
responses. | |
* Enable object system metadata on PUTs (Note: POST support is ongoing). | |
* Various other minor bug fixes and improvements. | |
swift (2.0.0) | |
* Storage policies | |
Storage policies allow deployers to configure multiple object rings | |
and expose them to end users on a per-container basis. Deployers | |
can create policies based on hardware performance, regions, or other | |
criteria and independently choose different replication factors on | |
them. A policy is set on a Swift container at container creation | |
time and cannot be changed. | |
Full docs are at https://docs.openstack.org/swift/latest/overview_policies.html | |
* Add profiling middleware in Swift | |
The profile middleware provides a tool to profile Swift | |
code on the fly and collects statistical data for performance | |
analysis. A native simple Web UI is also provided to help | |
query and visualize the data. | |
* Add --quoted option to swift-temp-url | |
* swift-recon now supports checking the md5sum of swift.conf, which | |
helps deployers verify configurations are consistent across a cluster. | |
* Users can now set the transaction id suffix by passing in | |
a value in the X-Trans-Id-Extra header. | |
* New log_max_line_length option caps the maximum length of a log line. | |
* Support If-[Un]Modified-Since for object HEAD | |
* Added missing constraints and ratelimit parameters to /info | |
* Add ability to remove subsections from /info | |
* Unify logging for account, container, and object server processes | |
to provide a consistent message format. This change reorders the | |
fields logged for the account server. | |
* Add targeted config loading to swift-init. This allows an easier | |
and more explicit way to tell swift-init to run specific server | |
process configurations. | |
* Properly quote www-authenticate (CVE-2014-3497) | |
* Fix logging issue when services stop on py26. | |
* Change the default logged length of the auth token to 16. | |
* Explicitly set permissions on generated ring files to 0644 | |
* Fix file uploads larger than 2GiB in the formpost feature | |
* Fixed issue where large objects would fail to download if the | |
auth token expired partway through the download | |
* Various other minor bug fixes and improvements | |
swift (1.13.1, OpenStack Icehouse) | |
* Change the behavior of CORS responses to better match the spec | |
A new proxy config variable (strict_cors_mode, default to True) | |
has been added. Setting it to False keeps the old behavior. For | |
an overview of old versus new behavior, please see | |
https://review.opendev.org/#/c/69419/ | |
* Invert the responsibility of the two instances of proxy-logging in | |
the proxy pipeline | |
The first proxy_logging middleware instance to receive a request | |
in the pipeline marks that request as handling it. So now, the | |
left most proxy_logging middleware handles logging for all | |
client requests, and the right most proxy_logging middleware | |
handles all other requests initiated from within the pipeline to | |
its left. This fixes logging related to large object | |
requests not properly recording bandwidth. | |
* Added swift-container-info and swift-account-info tools | |
* Allow specification of object devices for audit | |
* Dynamic large object COPY requests with ?multipart-manifest=get | |
now work as expected | |
* When a client is downloading a large object and one of the segment | |
reads gets bad data, Swift will now immediately abort the request. | |
* Fix ring-builder crash when a ring partition was assigned to a | |
deleted device, zero-weighted device, and normal device | |
* Make probetests work with conf.d configs | |
* Various other minor bug fixes and improvements. | |
swift (1.13.0) | |
* Account-level ACLs and ACL format v2 | |
Accounts now have a new privileged header to represent ACLs or | |
any other form of account-level access control. The value of | |
the header is a JSON dictionary string to be interpreted by the | |
auth system. A reference implementation is given in TempAuth. | |
Please see the full docs at | |
https://docs.openstack.org/swift/latest/overview_auth.html | |
* Added a WSGI environment flag to stop swob from always using | |
absolute location. This is useful if middleware needs to use | |
out-of-spec Location headers in a response. | |
* Container sync proxies now support simple load balancing | |
* Config option to lower the timeout for recoverable object GETs | |
* Add a way to ratelimit all writes to an account | |
* Allow multiple storage_domain values in cname_lookup middleware | |
* Moved all DLO functionality into middleware | |
The proxy will automatically insert the dlo middleware at an | |
appropriate place in the pipeline the same way it does with the | |
gatekeeper middleware. Clusters will still support DLOs after upgrade | |
even with an old config file that doesn't mention dlo at all. | |
* Remove python-swiftclient dependency | |
* Add secondary groups to process user during privilege escalation | |
* When logging request headers, it is now possible to specify | |
specifically which headers should be logged | |
* Added log_requests config parameter to account and container servers | |
to match the parameter in the object server. This allows a deployer | |
to turn off log messages for these processes. | |
* Ensure swift.source is set for DLO/SLO requests | |
* Fixed an issue where overwriting segments in a dynamic manifest | |
could cause issues on pipelined requests. | |
* Properly handle COPY verb in container quota middleware | |
* Improved StaticWeb 404 error message on web-listings and index | |
* Various other minor bug fixes and improvements. | |
swift (1.12.0) | |
* Several important pieces of information have been added to /info: | |
- Configured constraints are included and allow a client to discover | |
the limits on names and object sizes that the cluster supports. | |
- The supported tempurl methods are now included. | |
- Static large object constraints are now included. | |
* The Last-Modified header value returned will now be the object's | |
timestamp rounded up to the next second. This allows subsequent | |
requests with If-[un]modified-Since to use the Last-Modified | |
value as expected. | |
* Non-integer values for if-delete-at headers will now properly | |
report a 400 error instead of a 503. | |
* Fix object versioning with non-ASCII container names. | |
* Bulk delete with POST now works properly. | |
* Generic means for persisting system metadata | |
Swift now supports system-level metadata on accounts and | |
containers. System metadata provides a means to store internal | |
custom metadata with associated Swift resources in a safe and | |
secure fashion without actually having to plumb custom metadata | |
through the core swift servers. The new gatekeeper middleware | |
prevents this system metadata from leaking into the request or | |
being set by a client. | |
* catch_errors and gatekeeper middleware are now forced into the proxy | |
pipeline if not explicitly referenced. | |
* New container sync configuration option, separating the end user | |
from knowing the required end point and adding more secure | |
signed requests. See | |
https://docs.openstack.org/swift/latest/overview_container_sync.html | |
for full information. | |
* bulk middleware now can be configured to retry deleting containers. | |
* The default yield_frequency used to keep client connections alive | |
during slow bulk requests was reduced from 60 seconds to 10 seconds. | |
While this is a change to a default, it should not affect deployments | |
and there is no migration process needed. | |
* Swift processes will attempt to set RLIMIT_NPROC to 8192. | |
* Server processes will now exit with a non-zero error code on config | |
errors. | |
* Warn if read_affinity is configured but not enabled. | |
* Fix checkmount error parsing in swift-recon. | |
* Log at warn level when an object is quarantined. | |
* Fixed CVE-2014-0006 to avoid a potential timing attack with tempurl. | |
* Various other minor bug fixes and improvements. | |
swift (1.11.0) | |
* Added discoverable capabilities | |
A Swift proxy server now by default (although it can be turned off) | |
will respond to requests to /info. The response to these requests | |
include information about the cluster and can be used by clients to | |
determine which features are supported in the cluster. | |
* Object replication ssync (an rsync alternative) | |
A Swift storage node can now be configured to use Swift primitives | |
for replication transport instead of rsync. This is an experimental | |
feature that is not yet considered production ready. | |
* If a source times out on an object server read, try another one | |
of them with a modified range. | |
* The proxy now responds to many types of requests as soon as it | |
has a quorum. This can help speed up responses (without | |
changing the results), especially when one node is acting up. | |
There is a post_quorum_timeout config value that can tune how | |
long to wait for requests to finish after a quorum has been | |
established. | |
* Add accurate timestamps in proxy log lines for the start and | |
end of a request. These are added as new fields on the end of | |
the existing log lines, and therefore should not break | |
existing, well-behaved log processors. | |
* Add an "inline" query parameter to tempurl | |
By default, temporary URLs add a "Content-Disposition" header | |
that forces many clients to download the object. Now, temporary | |
URLs support an optional "inline" query parameter that will | |
force a "Content-Disposition: inline" header to be added to the | |
response, overriding the default. | |
* Use TCP_NODELAY for created sockets. This can dramatically | |
lower latency for small object workloads. | |
* DiskFile API, with reference implementation | |
The DiskFile abstraction for talking to data on disk has been | |
refactored to allow alternate implementations to be developed. | |
Included in the codebase is an in-memory reference | |
implementation. For full documentation, please see the developer | |
documentation. The DiskFile API is still a work in progress and | |
is not yet finalized. | |
* Removal of swift-bench | |
The included benchmarking tool swift-bench has been extracted | |
from the codebase and is now in its own repository at | |
https://github.com/openstack/swift-bench. New swift-bench | |
binaries and packages may be found on PyPI at | |
https://pypi.org/project/swift-bench | |
* Bulk delete now also supports the POST verb, in addition to DELETE | |
* Added functionality to the swift-ring-builder to support | |
limited recreation of ring builder files from the ring file itself. | |
* HEAD on account now returns 410 if account was deleted and | |
not yet reaped. The old behavior was to return a 404. | |
* Fixed a bug introduced since the 1.10.0 release that | |
prevented expired objects from being removed from the system. | |
This resulted in orphaned expired objects taking up space on | |
the system but inaccessible to the API. This regression and | |
fix are only important if you have deployed code since the | |
1.10.0 release. For a full discussion, including a script that | |
can be used to clean up orphaned objects, see | |
https://bugs.launchpad.net/swift/+bug/1257330 | |
* Tie socket write buffer size to server chunk size parameter. This | |
pairs the underlying network buffer size with the size of data | |
that Swift attempts to read from the connection, thereby | |
improving efficiency and throughput on connections. | |
* Fix 500 from account-quota middleware. If a user had set | |
X-Account-Meta-Quota-Bytes to something non-integer prior to | |
the installation of the account-quota middleware, then the | |
quota check would choke on it. Now a non-integer value is | |
treated as "no quota". | |
* Quarantine objects with busted metadata. Before, if you | |
encountered an object with corrupt or missing xattrs, the | |
object server would return a 500 on GET, and wouldn't quarantine | |
anything. Now the object server returns a 404 for that GET and | |
the corrupted file is quarantined, thus giving replication a | |
chance to fix it. | |
* Fix quarantine and error counts in audit logs | |
* Report transaction ID in failure exception logs | |
* Make pbr a build-time only dependency | |
* Worked around a bug in eventlet 0.9.16 where the size of the | |
memcache connection pools would grow unbounded. | |
* Tempurl keys are now properly stored as utf8 | |
* Fixed an issue where concurrent PUT requests to accounts or | |
containers may result in errors due to locked databases. | |
* Handle copy requests in account and container quota middleware | |
* Now ensure that a WWW-Authenticate header is on all 401 responses | |
* Various other bug fixes and improvements | |
swift (1.10.0, OpenStack Havana) | |
* Added support for pooling memcache connections | |
* Added support to replicating handoff partitions first in object | |
replication. Can also configure how many remote nodes a storage node | |
must talk to before removing a local handoff partition. | |
* Fixed bug where memcache entries would not expire | |
* Much faster calculation for choosing handoff nodes | |
* Added container listing ratelimiting | |
* Fixed issue where the proxy would continue to read from a storage | |
server even after a client had disconnected | |
* Added support for headers that are only visible to the owner of a Swift | |
account | |
* Fixed ranged GET with If-None-Match | |
* Fixed an issue where rings may not be balanced after initial creation | |
* Fixed internationalization support | |
* Return the correct etag for a static large object on the PUT response | |
* Allow users to extract archives to containers with ACLs set | |
* Fix support for range requests against static large objects | |
* Now logs x-copy-from header in a useful place | |
* Reverted back to old XML output of account and container listings to | |
ensure older clients do not break | |
* Account quotas now appropriately handle copy requests | |
* Fix issue with UTF-8 handling in versioned writes | |
* Various other bug fixes and improvements, including support for running | |
Swift under Pypy and continuing work to support storage policies | |
swift (1.9.1) | |
* Disallow PUT, POST, and DELETE requests from creating older tombstone | |
files, preventing the possibility of filling up the disk and removing | |
unnecessary container updates. | |
* Set default wsgi workers to cpu_count | |
Change the default value of wsgi workers from 1 to auto. The new | |
default value for workers in the proxy, container, account & object | |
wsgi servers will spawn as many workers per process as you have cpu | |
cores. This will not be ideal for some configurations, but it's much | |
more likely to produce a successful out of the box deployment. | |
* Added reveal_sensitive_prefix config setting to filter the auth token | |
logged by the proxy server. | |
* Ensure Keystone's reseller prefix ends with an underscore. Previously | |
this was a recommendation--now it is enforced. | |
* Added log_file_pattern config to swift-drive-audit for drive errors | |
* Add support for telling Swift to detect a content type on a request. | |
* Additional object stats are now logged in the object auditor | |
* Moved the DiskFile interface into its own module | |
* Ensure the SQLite cursors are closed when creating functions | |
* Better support for valid Accept headers | |
* In Keystone, don't allow users to delete their own account | |
* Return a UTC timezone designator in container listings | |
* Ensure that users can't remove their account quotas | |
* Allow floating point value for dispersion coverage | |
* Fix incorrect error page handling in staticweb | |
* Add utf-8 charset to multipart-manifest=get response. | |
* Allow dispersion tools to use keystone server with insecure certificate | |
* Ensure that files are always closed in tests | |
* Use OpenStack's "Hacking" guidelines for code formatting | |
* Various other minor bug fixes and improvements | |
swift (1.9.0) | |
* Global clusters support | |
The "region" concept introduced in Swift 1.8.0 has been augmented with | |
support for using a separate replication network and configuring read | |
and write affinity. These features combine to offer support for a single | |
Swift cluster spanning wide geographic area. | |
* Disk performance | |
The object server now can be configured to use threadpools to increase | |
performance and smooth out latency throughout the system. Also, many | |
disk operations were reordered to increase reliability and improve | |
performance. | |
* Added config file conf.d support | |
Allow Swift daemons and servers to optionally accept a directory as the | |
configuration parameter. This allows different parts of the config file | |
to be managed separately, eg each middleware could use a separate file | |
for its particular config settings. | |
* Allow two TempURL keys per account | |
By adding a second key, a user can safely rotate keys and prevent URLs | |
already in use from becoming invalid. TempURL middlware has also been | |
updated to allow a configuable set of allowed methods and to prevent a | |
bugrelated to content-disposition names. | |
* Added crossdomain.xml middleware. See | |
https://docs.openstack.org/swift/latest/crossdomain.html for details | |
* Added rsync bandwidth limit setting for object replicator | |
* Transaction ID updated to include the time and an optional suffix | |
* Added x-remove-versions-location header to disable versioned writes | |
* Improvements to support for Keystone ACLs | |
* Added parallelism to object expirer daemon | |
* Added support for ring hash prefix in addition to the existing suffix | |
* Allow all headers requested for CORS | |
* Stop getting useless bytes on manifest Range requests | |
* Improved container-sync resiliency | |
* Added example Apache config files. See | |
https://docs.openstack.org/swift/latest/apache_deployment_guide.html | |
for more info | |
* If an account is marked as deleted but hasn't been reaped and is still | |
on disk, responses will include an "X-Account-Status" header | |
* Fix 503 on account/container HEAD with invalid format | |
* Added extra safety on account-level DELETE when using bulk deletes | |
* Made colons quote-safe in logs (mainly for IPv6) | |
* Fixed bug with bulk delete max items | |
* Fixed static large object manifest range requests | |
* Prevent static large objects from containing other static large objects | |
* Fixed issue with use of delimiter in container queries where some | |
objects would not be listed | |
* Various other minor bug fixes and improvements | |
swift (1.8.0, OpenStack Grizzly) | |
* Make rings' replica count adjustable | |
* Added a region tier to the ring above zones | |
* Added timing-based sorting of object servers on read requests | |
* Added support for auto-extract archive uploads | |
* Added support for bulk delete requests | |
* Added support for large objects with static manifests | |
* Added list_endpoints middleware to provide an API for determining where | |
the ring places data | |
* proxy-logging middleware can now handle logging for other middleware | |
proxy-logging should be used twice in the proxy pipeline. The first | |
handles middleware logs for requests that never made it all the way | |
to the server. The last handles requests that do make it to the server. | |
This is a change that may require an update to your proxy server | |
config file or custom middleware that you may be using. See the full | |
docs at https://docs.openstack.org/swift/latest/misc.html. | |
* Changed the default sample rate for a few high-traffic requests. | |
Added log_statsd_sample_rate_factor to globally tune the StatsD | |
sample rate. This tunable can be used to reduce StatsD traffic | |
proportionally for all metrics and is intended to replace | |
log_statsd_default_sample_rate, which is left alone for | |
backward-compatibility, should anyone be using it. | |
* Added swift_hash_path_prefix option to swift.conf | |
New deployments are advised to set this value to a random secret | |
to protect against hash collisions | |
* Added user-managed container quotas | |
* Added support for account-level quotas managed by an auth reseller | |
* Added --run-dir option to swift-init | |
* Added more options to swift-bench | |
* Added support for CORS "actual requests" | |
* Added fallocate_reserve option to protect against full drives | |
* Allow ring rebalance to take a seed | |
* Ring serialization will now produce the same gzip file (Py2.7) | |
* Added support to swift-drive-audit for handling rotated logs | |
* Added first-byte latency timings for GET requests | |
* Added per disk PUT timing monitoring support | |
* Added speed limit options for DB auditor | |
* Force log entries to be one line | |
* Ensure that fsync is used and not just fdatasync | |
* Improved handoff node selection | |
* Deprecated keystone is_admin feature | |
* Fix large objects with unicode in the segment names | |
* Update Swift's MemcacheRing to provide API compatibility with | |
standard Python memcache libraries | |
* Various other minor bug fixes and improvements | |
swift (1.7.6) | |
* Better tempauth storage URL guessing | |
* Added --top option to swift-recon -d | |
* Allow optional, temporary healthcheck failure | |
* keystoneauth middleware now supports cross-tenant ACLs | |
* Add dispersion report flags to limit reports | |
* Add config option to turn eventlet debug on/off | |
* Added override option for swift-init's KILL_WAIT | |
* Added oldest and most recent replication pass to swift-recon | |
* Fixed 500 error response when GETing a many-segment manifest | |
* Memcached keys now use a delta timeout when possible | |
* Refactor DiskFile to hide temp file names and exts | |
* Remove IP-based container-sync ACLs from auth middlewares | |
* Fixed bug in deleting memcached account info data | |
* Fixed lazy-listing of object manifest segments | |
* Fixed bug where a ? in the object name caused an error | |
* Swift now returns 406 if it can't satisfy Accept | |
* Fix infinite recursion bug in object replicator | |
* Swift will now reject names with NULL characters | |
* Fixed object-auditor logging to use a minimum of unix sockets | |
* Various other minor bug fixes and improvements | |
swift (1.7.5) | |
* Support OPTIONS verb, including CORS preflight requests | |
* Added support for custom log handlers | |
* Range support is extended to support GET requests with multiple ranges. | |
Multi-range GETs are not yet supported against large-object manifests. | |
* Cluster constraints are now settable by config | |
* Replicators can now run against specific devices or partitions | |
* swift-bench now supports running on multiple cores and multiple servers | |
* Added partition option to swift-get-nodes | |
* Allow underscores in account and user in tempauth via base64 encodings | |
* New option to the dispersion report to output the missing partitions | |
* Changed storage server StatsD metrics to report timings instead of | |
counts for errors. See the admin guide for the updated metric names. | |
* Removed a dependency on WebOb and replaced it with an internal module | |
* Fixed config parsing in swift-bench -x | |
* Fixed sample_rate in StatsD logging | |
* Track unlinks of async_pendings with StatsD | |
* Remove double GET on range requests | |
* Allow unsetting of X-Container-Sync-To and ACL headers | |
* DB reclamation now removes empty suffix directories | |
* Fix non-standard 100-continue behavior | |
* Allow object-expirer to delete the last copy of a versioned object | |
* Only set TCP_KEEPIDLE on systems where it is supported | |
* Fix stdin flush and fdatasync issues on BSD platforms | |
* Allow object-expirer to delete the last version of an object | |
* Various other minor bug fixes and improvements | |
swift (1.7.4, OpenStack Folsom) | |
* Fix issue where early client disconnects may have caused a memory leak | |
swift (1.7.2) | |
* Fix issue where memcache serialization was not properly loading | |
the config value | |
swift (1.7.0) | |
* Use custom encoding for ring data instead of pickle | |
Serialize RingData in a versioned, custom format which is a combination | |
of a JSON-encoded header and .tostring() dumps of the | |
replica2part2dev_id arrays. This format deserializes hundreds of times | |
faster than rings serialized with Python 2.7's pickle (a significant | |
performance regression for ring loading between Python 2.6 and Python | |
2.7). Fixes bug 1031954. | |
The new implementation is backward-compatible; if a ring | |
does not begin with a new-style magic string, it is assumed to be an | |
old-style pickle-dumped ring and is handled as before. So new Swift | |
code can read old rings, but old Swift code will not be able to read | |
newly-serialized rings. | |
* Do not use pickle for serialization in memcache, but JSON | |
To avoid issues on upgrades (unability to read pickled values, and cache | |
poisoning for old servers not understanding JSON), we add a | |
memcache_serialization_support configuration option, with the following | |
values: | |
0 = older, insecure pickle serialization | |
1 = json serialization but pickles can still be read (still insecure) | |
2 = json serialization only (secure and the default) | |
To avoid an instant full cache flush, existing installations should | |
upgrade with 0, then set to 1 and reload, then after some time (24 | |
hours) set to 2 and reload. Support for 0 and 1 will be removed in | |
future versions. | |
* Update proxy-server StatsD logging. This is a significant change to the | |
existing StatsD intigration. Docs for this feature can be found in | |
doc/source/admin_guide.rst. | |
* Improved swift-bench to allow random object sizes and better usability | |
* Updated probe tests | |
* Replicator removal metrics are now generated on a per-device basis | |
* Made object replicator locking more optimistic | |
* Split proxy-server code into separate modules | |
* Fixed bug where swift-recon would not report all unmounted drives | |
* Fixed issue where a LockTimeout may have caused a file descriptor to | |
not be closed properly | |
* Fixed a bug where an error may have caused the proxy to stop returning | |
data to a client | |
* Fixed bug where expirer would get confused by odd deletion times | |
* Fixed a bug where auto-creating accounts would return an error if they | |
were recreated after being deleted | |
* Fix when rate_limit_after_segment kicks in | |
* fallocate() failures properly return HTTPInsufficientStorage from | |
object-server before reading from wsgi.input, allowing the proxy | |
server to quickly error_limit that node | |
* Fixed error with large object manifests and x-newest headers on GET | |
* Various other minor bug fixes and improvements | |
swift (1.6.0) | |
* Removed bin/swift and swift/common/client.py from the swift repo. These | |
tools are now managed in the python-swiftclient project. The | |
python-swiftclient project is a second deliverable of the openstack | |
swift project. | |
* Moved swift_auth (openstack keystone) middleware from keystone project | |
into swift project | |
* Made dispersion report work with any replica count other than 3. This | |
substantially affects the JSON output of the dispersion report, and any | |
tools written to consume this output will need to be updated. | |
* Added Solaris (Illumos) compatibility | |
* Added -a option to swift-get-nodes to show all handoffs | |
* Add UDP protocol support for logger | |
* Added config options for rate limiting of large object downloads. | |
* Added config option `log_handoffs` (defaults to True) to proxy server | |
to log and update statsd with information about when a handoff node is | |
used. This is helpful to track the health of the cluster. | |
* swift-bench can now use auth 2.0 | |
* Support forbidding substrings based on a regexp in name_filter | |
middleware | |
* Hardened internal server processes so only authorized methods can be | |
called. | |
* Made ranged requests on large objects work correctly when size of | |
manifest file is not 0 byte | |
* Added option to dispersion report to print 404s to stdout | |
* Fix object replication on older rsync versions when using ipv4 | |
* Fixed bug with container reclaim/report race | |
* Make object server's caching more configurable. | |
* Check disk failure before syncing for each partition | |
* Allow special characters to be referenced by manifest objects | |
* Validate devices and partitions to avoid directory traversals | |
* Support WebOb 1.2 | |
* Ensure that accessing the ring devs reloads the ring if necessary. | |
Specifically, this allows replication to work when it has been started | |
with an empty ring. | |
* Various other minor bug fixes and improvements | |
swift (1.5.0) | |
* New option to toggle SQLite database preallocation with account | |
and container servers. | |
IMPORTANT: | |
The default for database preallocation is now off when before | |
it was always on. This will affect performance on clusters that | |
use standard drives with shared account, container, object | |
servers. Such deployments will need to update their | |
configurations to turn database preallocation back on (see | |
account-server.conf-sample and container-server.conf.sample | |
files). | |
If you are using dedicated account and container servers with | |
SSDs, you should defragment your file systems after upgrade and | |
should notice dramatically less disk usage. | |
* swift3 middleware removed and moved to http://github.com/fujita/swift3. | |
This will require a config change in the proxy server and adds a new | |
dependency for deployers using this middleware. | |
* Moved proxy server logging to middleware. This requires a config change | |
in the proxy server. | |
* Added object versioning feature. (See docs for full description) | |
* Add statsd logging throughout the system (beta, some event names may | |
change) | |
* Expanded swift-recon middleware support | |
* The ring builder now supports as-unique-as-possible partition | |
placement, unified balancing methods, and can work on more than one | |
device at a time. | |
* Numerous bug fixes to StaticWeb (previously unusable at scale). | |
* Bug fixes to all middleware to allow passthrough requests under various | |
conditions and to share pre-authed request code (which previously had | |
differing behaviors and interaction bugs). | |
* Bug fix to object expirer that could cause infinite looping. | |
* Added optional delay to account reaping. | |
* Async-pending write optimization. | |
* Dispersion tools now support multiple auth versions | |
* Updated man pages | |
* Proxy server can now deny requests to particular hostnames | |
* Updated docs for domain remap middleware | |
* Updated docs for cname lookup middleware | |
* Made swift CLI binary easier to wrap | |
* Proxy will now also return X-Timestamp header | |
* Added associated projects doc as a place to track ecosystem projects | |
* end_marker made consistent across both object and container listings | |
* Various other minor bug fixes and improvements | |
swift (1.4.8, OpenStack Essex) | |
* Added optional max_containers_per_account restriction | |
* Added alternate metadata header removal method | |
* Added optional name_check middleware filter | |
* Added support for venv-based test runs with tox | |
* StaticWeb behavior change with X-Web-Mode: true and | |
non-StaticWeb-enabled containers (immediately 404s instead of passing | |
the request on down the WSGI pipeline). | |
* Fixed typo in swift-dispersion-report JSON output. | |
* Swift-Recon-related fix to create temporary files on the same disk as | |
their final destinations. | |
* Updated return codes in swift3 middleware | |
* Fixed swift3 middleware to allow Content-Range header in response | |
* Updated swift.common.client and swift CLI tool with auth 2.0 changes | |
* Swift CLI tool now supports common openstack auth args | |
* Body of HTTP responses now included in error messages of swift CLI tool | |
* Refactored some ring building functions for clarity and simplicity | |
swift (1.4.7) | |
* Improvements to account and container replication. | |
* Fix for account servers allowing .pending to exist before .db. | |
* Fixed possible key-guessing exploit in formpost. | |
* Fixed bug in ring builder when removing a large percentage of devices. | |
* Swift CLI tool now supports openstack-standard CLI flags. | |
* New JSON output option for swift-dispersion-report. | |
* Removed old stats tools. | |
* Other bug fixes and documentation updates. | |
swift (1.4.6) | |
* TempURL and FormPost middleware added | |
* Added memcache.conf option | |
* Dropped eval-based json parser fallback | |
* Properly lose all groups when dropping privileges | |
* Fix permissions when creating files | |
* Fixed bug regarding negative Content-Length in requests | |
* Consistent formatting on Last-Modified response header | |
* Added timeout option to swift-recon | |
* Allow arguments to be passed to nosetest | |
* Removed tools/rfc.sh | |
* Other minor bug fixes | |
swift (1.4.5) | |
* New swift-orphans and swift-oldies command line tools to detect | |
orphaned Swift processes and long running processes. | |
* Command line tool "swift" now supports marker queries. | |
* StaticWeb middleware improved to save an extra request when | |
possible. | |
* Updated swift-init to support swift-object-expirer. | |
* Fixed object replicator timeout handling [bug 814263]. | |
* Fixed accept header 503 vs. 400 [bug 891247]. | |
* More exception handling for auditors. | |
* Doc updates for PPA [bug 905608]. | |
* Doc updates to explain replication more clearly [bug 906976]. | |
* Updated SAIO instructions to no longer mention ~/swift/trunk. | |
* Fixed docstrings in the ring code. | |
* PEP8 Updates. | |
swift (1.4.4) | |
* Fixes to prevent socket hoarding (memory leak) | |
* Add sockstat info to recon. | |
* Fixed leak from SegmentedIterable. | |
* Fixed bufferedhttp to deref socks and fps. | |
* Add support for OS Auth API version 2. | |
* Make Eventlet's WSGI server log differently. | |
* Updated TimeoutError and except Exception refs. | |
* Fixed time-sensitive tests. | |
* Fixed object manifest etags. | |
* Fixes for swift-recon disk usage distribution graph. | |
* Adding new manpages for configuration files. | |
* Change bzr to swift in getting_started doc. | |
* Fixes the HTTPConflict import. | |
* Expiring Objects Support. | |
* Fixing bug with x-trans-id. | |
* Requote the source when doing a COPY. | |
* Add documentation for Swift Recon. | |
* Make drive audit regexes detect 4-letter drives. | |
* Adding what acc/cont/obj into the ratelimit error messages. | |
* Query only specific zone via swift-recon. | |
swift (1.4.3, OpenStack Diablo) | |
* Additional quarantine catching code. | |
* Added client_ip to all proxy log lines not otherwise containing it. | |
* Content-Type is now application/xml for "GET services/bucket" swift3 | |
middleware requests. | |
* Alpha release of the Swift Recon Experiment | |
* Fix last modified date for swift3 middleware. | |
* Fix to clear account/container metadata on account/container deletion. | |
* Fix for corner case regarding X-Newest. | |
* Fix for object auditor running out of file descriptors. | |
* Fix to return all proper headers for manifest objects. | |
* Fix to the swift tool to strip any leading slashes on file names when | |
uploading. | |
swift (1.4.2) | |
* Removed stats/logging code from Swift [now in separate slogging project]. | |
* Container Synchronization Feature - First Edition | |
* Fix swift3 authentication bug about the Date and X-Amz-Date handling. | |
* Changing ratelimiting so that it only limits PUTs/DELETEs. | |
* Object POSTs are implemented as COPYs now by default (you can revert to | |
previous implementation with conf object_post_as_copy = false) | |
* You can specify X-Newest: true on GETs and HEADs to indicate you want | |
Swift to query all backend copies and return the newest version | |
retrieved. | |
* Object COPY requests now always copy the newest object they can find. | |
* Account and container GETs and HEADs now shuffle the nodes they use to | |
balance load. | |
* Fixed the infinite charset: utf-8 bug | |
* This fixes the bug that drop_buffer_cache() doesn't work on systems where | |
off_t isn't 64 bits. | |
swift (1.4.1) | |
* st renamed to swift | |
* swauth was separated froms swift. It is now its own project and can be | |
found at https://github.com/gholt/swauth. | |
* tempauth middleware added as an extremely limited auth system for dev | |
work. | |
* Account and container listings now properly labeled UTF-8 (previously the | |
label was "utf8"). | |
* Accounts are auto-created if an auth token is valid when the | |
account_autocreate proxy config parameter is set to true. | |
swift (1.4.0) | |
* swift-bench now cleans up containers it creates. | |
* WSGI servers now load WSGI filters and applications after forking for | |
better plugin support. | |
* swauth-cleanup-tokens now handles 404s on token containers and tokens | |
better. | |
* Proxy logs the remote IP address as the client IP in the absence of | |
X-Forwarded-For and X-Cluster-Client-IP headers instead of - like it did | |
before. | |
* Swift3 WSGI middleware added support for param-signed URLs. | |
* swauth- scripts now exit with proper exit codes. | |
* Fixed a bug where allowed_headers weren't honored for HEAD requests. | |
* Double quarantining of corrupted sqlite3 databases now works. | |
* Fix for Object replicator breaking when running object replicator with no | |
objects on the server. | |
* Added the Accept-Ranges header to GET and HEAD requests. | |
* When a single object has multiple async pending updates on a single | |
device, only latest async pending is now sent. | |
* Fixed issue of Swift3 WSGI middleware not working correctly with '/' in | |
object names. | |
* Renamed swift-stats-* to swift-dispersion-* to avoid confusion with log | |
stats stuff. | |
* Added X-Trans-Id transaction id header to every response. | |
* Fixed a Python 2.7 compatibility problem. | |
* Now using bracketed notation for ip literals in rsync calls, so | |
compressed ipv6 literals work. | |
* Added a container stats collector and refactoring some of the stats code. | |
* Changed subdir nodes in XML formatted object listings to align with | |
object nodes. Now: <subdir name="foo"><name>foo</name></subdir> Before: | |
<subdir name="foo" />. | |
* Fixed bug in Swauth to support for multiple swauth instances. | |
* swift-ring-builder: Added list_parts command which shows common | |
partitions for a given list of devices. | |
* Object auditor now shows better statistics updates in the logs. | |
* Stats uploaders now allow overrides for source_filename_pattern and | |
new_log_cutoff values. | |
---- | |
Changelog entries for previous versions are incomplete | |
swift (1.3.0, OpenStack Cactus) | |
swift (1.2.0, OpenStack Bexar) | |
swift (1.1.0, OpenStack Austin) | |
swift (1.0.0, Initial Release) |