Plugin does not handle 302 redirect correctly

[smjgithub]

AWS Cloud plugin does not seem to correctly handle a 302 Redirect when encountered on the S3 end-point

Not sure if this is a bug or that perhaps we should access the end-point via a proxy of some sort?

Below is the error from the ES log when attempting to create a new S3 repo...

The S3 repo service is actually provided by IBM CleverSafe and should be fully S3 compliant.

Accessing the service directly via IP address works fine where the redirect does not happen but when using via a generic DNS address which redirects then problem occurs...

[2016-10-25 12:05:16,324][WARN ][rest.suppressed ] path: /_snapshot/s4-es-mon-prod, params: {repository=s4-es-mon-prod}
RepositoryException[[s4-es-mon-prod] failed to create repository]; nested: CreationException[Guice creation errors:

1. Error injecting constructor, com.amazonaws.services.s3.model.AmazonS3Exception: Found (Service: Amazon S3; Status Code: 302; Error Code: 302 Found; Request ID: null), S3 Extended Request ID: null
   at org.elasticsearch.repositories.s3.S3Repository.(Unknown Source)
   while locating org.elasticsearch.repositories.s3.S3Repository
   while locating org.elasticsearch.repositories.Repository

[dadoonet]

Can't you define cloud.aws.s3.endpoint in such a case?
See https://github.com/elastic/elasticsearch-cloud-aws/tree/v2.7.1/#using-other-s3-endpoint

[smjgithub]

Yes, I am defining the S3 end-point but that is an internal DNS address which responds with a redirect.

To allow the owner of that end-point to point somewhere else in the event of system outages I guess only way is to implement some sort of proxy between the AWS Cloud plugin and the S3 end-point so handle the redirect?

If I use the IP of the S3 service directly then it works but they'd prefer me not to hard-code it in this way. The proxy is just my idea to work around this issue?

Ideally it would be good if the AWS Cloud plugin could respond to the 302 but it seems to just treat this as an error?

[dadoonet]

Ideally it would be good if the AWS Cloud plugin could respond to the 302 but it seems to just treat this as an error?

Yes. Actually we are using AWS SDK here which apparently does not support that.
Unless there is a magical knob I did not see :)

Thinking out loud: may be this is something which can be reported in AWS SDK project instead?

[smjgithub]

Yes, that seems to make sense! Is that something that you could raise there citing this ticket as a 'use case' calling for it?

[smjgithub]

I'm not familiar with the AWS SDK and only use the plugin so perhaps this additional feature request would be best explained by a developer?

[dadoonet]

BTW do you have in logs the full stack trace?

[smjgithub]

Is this what you're after?

1. Error injecting constructor, com.amazonaws.services.s3.model.AmazonS3Exception: Found (Service: Amazon S3; Status Code: 302; Error Code: 302 Found; Request ID: null), S3 Extended Request ID: null
   at org.elasticsearch.repositories.s3.S3Repository.(Unknown Source)
   while locating org.elasticsearch.repositories.s3.S3Repository
   while locating org.elasticsearch.repositories.Repository

1 error
at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:360)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
... 11 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Found (Service: Amazon S3; Status Code: 302; Error Code: 302 Found; Request ID: null), S3 Extended Request ID: null
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1389)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:902)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3654)
at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1020)
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:978)
at org.elasticsearch.cloud.aws.blobstore.S3BlobStore.(S3BlobStore.java:84)
at org.elasticsearch.repositories.s3.S3Repository.(S3Repository.java:135)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:201)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:879)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
... 15 more
^C

[dadoonet]

I opened aws/aws-sdk-java#903. Will see what is the outcome of this.
For now, you need to define the right endpoint.

Please note that this repo is now deprecated and will only get bug fixes.
New features will only go to elasticsearch repo.

[smjgithub]

OK, great! thanks for following up on this...

I spoke to the internal provider of our S3 repo and he said that others are using the AWS SDK and they have worked around this 302 redirect problem so I will see if I can find any more info and pass it on if I do...

[dadoonet]

A guess is that we check manually this URL and if we get a redirect we change the configuration you defined with the new URL.
But IMO this is something that should be supported by the SDK.

[dadoonet]

So AWS team rejected the feature request for the reasons they exposed there: aws/aws-sdk-java#903

I'm not sure if we want to implement such a thing on our side and I believe we don't want to make the code more complex.

@tlrx WDYT?

[smjgithub]

Fair enough, I've got the end-point hard-coded to the S3 service IP at the moment and working fine.

I guess the proxy approach is still an option if we choose to go down that route.

Thanks for your persistence in following this up :)

[tlrx]

If I understand correctly, the IBM CleverSafe S3 implementation sends redirections to the client but the AWS SDK team indicates that it is not part of the service contract in AWS to send redirect responses?

If that's correct then I don't think we should implement such a thing on plugin's side.

[dadoonet]

Thanks Tanguy. We are on the same page.
Closing

[smjgithub]

Thanks (Tan)guys! Best to stick to AWS contract...