Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update aws sdk to 1.10.69 and add throttle_retries repository setting #18048

Merged
merged 4 commits into from
Jun 16, 2016
Merged

Update aws sdk to 1.10.69 and add throttle_retries repository setting #18048

merged 4 commits into from
Jun 16, 2016

Conversation

dadoonet
Copy link
Member

@dadoonet dadoonet commented Apr 28, 2016 •
edited
Loading

This PR brings 2 changes:

  • Upgrade to AWS SDK 1.10.69
  • Add repositories.s3.use_throttle_retries setting

Upgrade to AWS SDK 1.10.69

Release notes highlights from 1.10.33 for the SDK

  • Moving from JSON.org to Jackson for request marshallers.
  • The Java SDK now supports retry throttling to limit the rate of retries during periods of reduced availability. This throttling behavior can be enabled via ClientConfiguration or via the system property "-Dcom.amazonaws.sdk.enableThrottledRetry".
  • Fixed String case conversion issues when running with non English locales.
  • AWS SDK for Java introduces a new dynamic endpoint system that can compute endpoints for services in new regions.
  • Introducing a new AWS region, ap-northeast-2.
  • Added a new metric, HttpSocketReadTime, that records socket read latency. You can enable this metric by adding enableHttpSocketReadMetric to the system property com.amazonaws.sdk.enableDefaultMetrics. For more information, see Enabling Metrics with the AWS SDK for Java.
  • New Client Execution timeout feature to set a limit spent across retries, backoffs, ummarshalling, etc. This new timeout can be specified at the client level or per request.
    Also included in this release is the ability to specify the existing HTTP Request timeout per request rather than just per client.

Release notes highlights from 1.10.33 for S3

  • Added support for RequesterPays for all operations.
  • Ignore the 'Connection' header when generating S3 responses.
  • Allow users to generate an AmazonS3URI from a string without using URL encoding.
  • Fixed issue that prevented creating buckets when using a client configured for the s3-external-1 endpoint.
  • Amazon S3 bucket lifecycle configuration supports two new features: the removal of expired object delete markers and an action to abort incomplete multipart uploads.
  • Allow TransferManagerConfiguration to accept integer values for multipart upload threshold.
  • Copy the list of ETags before sorting Copy the list of ETags before sorting aws/aws-sdk-java#589.
  • Option to disable chunked encoding Option to disable chunked encoding aws/aws-sdk-java#586.
  • Adding retry on InternalErrors in CompleteMultipartUpload operation. Add Default Retry on InternalError during CompleteMultipartUpload aws/aws-sdk-java#538
  • Deprecated two APIs : AmazonS3#changeObjectStorageClass and AmazonS3#setObjectRedirectLocation.
  • Added support for the aws-exec-read canned ACL. Owner gets FULL_CONTROL. Amazon EC2 gets READ access to GET an Amazon Machine Image (AMI) bundle from Amazon S3.

Release notes highlights from 1.10.33 for EC2

  • Added support for referencing security groups in peered Virtual Private Clouds (VPCs). For more information see the service announcement at https://aws.amazon.com/about-aws/whats-new/2016/03/announcing-support-for-security-group-references-in-a-peered-vpc/ .
  • Fixed a bug in AWS SDK for Java - Amazon EC2 module that returns NPE for dry run requests.
  • Regenerated client with new implementation of code generator.
  • This feature enables support for DNS resolution of public hostnames to private IP addresses when queried over ClassicLink. Additionally, you can now access private hosted zones associated with your VPC from a linked EC2-Classic instance. ClassicLink DNS support makes it easier for EC2-Classic instances to communicate with VPC resources using public DNS hostnames.
  • You can now use Network Address Translation (NAT) Gateway, a highly available AWS managed service that makes it easy to connect to the Internet from instances within a private subnet in an AWS Virtual Private Cloud (VPC). Previously, you needed to launch a NAT instance to enable NAT for instances in a private subnet. Amazon VPC NAT Gateway is available in the US East (N. Virginia), US West (Oregon), US West (N. California), EU (Ireland), Asia Pacific (Tokyo), Asia Pacific (Singapore), and Asia Pacific (Sydney) regions. To learn more about Amazon VPC NAT, see New - Managed NAT (Network Address Translation) Gateway for AWS
  • A default read timeout is now applied when querying data from EC2 metadata service.

Add repositories.s3.use_throttle_retries setting

Defaults to true.
If anyone is having trouble with this option, you could disable it with repositories.s3.use_throttle_retries: false in elasticsearch.yml file.

Backport of #17784 in 2.x branch

@dadoonet
Update aws sdk to 1.10.69 and add cloud.aws.s3.throttle_retries setting
feb9c6a 
This PR brings 2 changes:

* Upgrade to AWS SDK 1.10.69
* Add cloud.aws.s3.throttle_retries setting

# Upgrade to AWS SDK 1.10.69

## Release notes highlights from 1.10.33 for the SDK

* Moving from JSON.org to Jackson for request marshallers.
* The Java SDK now supports retry throttling to limit the rate of retries during periods of reduced availability. This throttling behavior can be enabled via ClientConfiguration or via the system property "-Dcom.amazonaws.sdk.enableThrottledRetry".
* Fixed String case conversion issues when running with non English locales.
* AWS SDK for Java introduces a new dynamic endpoint system that can compute endpoints for services in new regions.
* Introducing a new AWS region, ap-northeast-2.
* Added a new metric, HttpSocketReadTime, that records socket read latency. You can enable this metric by adding enableHttpSocketReadMetric to the system property com.amazonaws.sdk.enableDefaultMetrics. For more information, see [Enabling Metrics with the AWS SDK for Java](https://java.awsblog.com/post/Tx3C0RV4NRRBKTG/Enabling-Metrics-with-the-AWS-SDK-for-Java).
* New Client Execution timeout feature to set a limit spent across retries, backoffs, ummarshalling, etc. This new timeout can be specified at the client level or per request.
  Also included in this release is the ability to specify the existing HTTP Request timeout per request rather than just per client.

## Release notes highlights from 1.10.33 for S3

* Added support for RequesterPays for all operations.
* Ignore the 'Connection' header when generating S3 responses.
* Allow users to generate an AmazonS3URI from a string without using URL encoding.
* Fixed issue that prevented creating buckets when using a client configured for the s3-external-1 endpoint.
* Amazon S3 bucket lifecycle configuration supports two new features: the removal of expired object delete markers and an action to abort incomplete multipart uploads.
* Allow TransferManagerConfiguration to accept integer values for multipart upload threshold.
* Copy the list of ETags before sorting aws/aws-sdk-java#589.
* Option to disable chunked encoding aws/aws-sdk-java#586.
* Adding retry on InternalErrors in CompleteMultipartUpload operation. aws/aws-sdk-java#538
* Deprecated two APIs : AmazonS3#changeObjectStorageClass and AmazonS3#setObjectRedirectLocation.
* Added support for the aws-exec-read canned ACL. Owner gets FULL_CONTROL. Amazon EC2 gets READ access to GET an Amazon Machine Image (AMI) bundle from Amazon S3.

## Release notes highlights from 1.10.33 for EC2

* Added support for referencing security groups in peered Virtual Private Clouds (VPCs). For more information see the service announcement at https://aws.amazon.com/about-aws/whats-new/2016/03/announcing-support-for-security-group-references-in-a-peered-vpc/ .
* Fixed a bug in AWS SDK for Java - Amazon EC2 module that returns NPE for dry run requests.
* Regenerated client with new implementation of code generator.
* This feature enables support for DNS resolution of public hostnames to private IP addresses when queried over ClassicLink. Additionally, you can now access private hosted zones associated with your VPC from a linked EC2-Classic instance. ClassicLink DNS support makes it easier for EC2-Classic instances to communicate with VPC resources using public DNS hostnames.
* You can now use Network Address Translation (NAT) Gateway, a highly available AWS managed service that makes it easy to connect to the Internet from instances within a private subnet in an AWS Virtual Private Cloud (VPC). Previously, you needed to launch a NAT instance to enable NAT for instances in a private subnet. Amazon VPC NAT Gateway is available in the US East (N. Virginia), US West (Oregon), US West (N. California), EU (Ireland), Asia Pacific (Tokyo), Asia Pacific (Singapore), and Asia Pacific (Sydney) regions. To learn more about Amazon VPC NAT, see [New - Managed NAT (Network Address Translation) Gateway for AWS](https://aws.amazon.com/blogs/aws/new-managed-nat-network-address-translation-gateway-for-aws/)
* A default read timeout is now applied when querying data from EC2 metadata service.

# Add `cloud.aws.s3.throttle_retries` setting

Defaults to `true`.
If anyone is having trouble with this option, you could disable it with `cloud.aws.s3.throttle_retries: false` in `elasticsearch.yml` file.

Backport of elastic#17784 in 2.x branch
tlrx
tlrx reviewed May 6, 2016
View reviewed changes
plugins/cloud-aws/src/main/java/org/elasticsearch/cloud/aws/InternalAwsS3Service.java
@@ -130,6 +130,7 @@ private synchronized AmazonS3 getClient(String endpoint, String protocol, String
if (maxRetries != null) {
// If not explicitly set, default to 3 with exponential backoff policy
clientConfiguration.setMaxErrorRetry(maxRetries);
clientConfiguration.setUseThrottleRetries(settings.getAsBoolean(AwsS3Service.CLOUD_S3.THROTTLE_RETRIES, true));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as #17784 (comment)

@dadoonet
Copy link
Member Author

@tlrx also updated the code here

@dadoonet dadoonet mentioned this pull request May 30, 2016
Closed
@dadoonet
Copy link
Member Author

ping @tlrx

@dadoonet
Copy link
Member Author

@tlrx I guess you also agreed with this one as it's similar to the one in master and the one for ES 1.7?

@dadoonet dadoonet changed the title Update aws sdk to 1.10.69 and add cloud.aws.s3.throttle_retries setting Update aws sdk to 1.10.69 and add throttle_retries repository setting Jun 15, 2016
@tlrx
Copy link
Member

tlrx commented Jun 15, 2016

Yep

@dadoonet dadoonet merged commit 898d468 into elastic:2.x Jun 16, 2016
@dadoonet dadoonet removed the review label Jun 16, 2016
@dadoonet dadoonet deleted the pr/17784-update-aws-sdk branch June 16, 2016 05:05
@clintongormley clintongormley added :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Plugin Cloud AWS labels Feb 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@tlrx tlrx

Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >upgrade v2.4.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dadoonet @tlrx @clintongormley