Modify CloudSearch Search to use POST instead of GET

[joel-cass]

Any search request using AmazonCloudSearchDomain.SearchAsync(searchRequest) or AmazonCloudSearchDomain.Search(searchRequest) that has a large search string will end up with a 413 Full Head message being returned from the AWS service.

This exception is not handled gracefully by the Marshaller and will return a Null Pointer Exception that obfuscates the actual cause of the issue. I don't care much for this; it would be better that the actual issue above is addressed.

Expected Behavior

Searching for a long string should return results (create a search longer than 4000 characters for example).

The same search can be run in the console with no problems as it is using post

Current Behavior

Instead it returns
``[NullReferenceException: Object reference not set to an instance of an object.]
Amazon.Runtime.Internal.Transform.JsonErrorResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context) +874
Amazon.CloudSearchDomain.Model.Internal.MarshallTransformations.SearchResponseUnmarshaller.UnmarshallException(JsonUnmarshallerContext context, Exception innerException, HttpStatusCode statusCode) +40
Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.UnmarshallException(UnmarshallerContext input, Exception innerException, HttpStatusCode statusCode) +61
Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) +282

[AmazonUnmarshallingException: Error unmarshalling response back from AWS. ]
Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) +1011
Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception) +275``

And using Fiddler we see the response from the server:
HTTP/1.1 413 FULL head Content-Length: 0 Connection: keep-alive

Possible Solution

SearchRequestMarshaller has hard-coded the method of GET for requests.

aws-sdk-net/sdk/src/Services/CloudSearchDomain/Generated/Model/Internal/MarshallTransformations/SearchRequestMarshaller.cs

Line 58 in 1a5187e

request.HttpMethod = "GET";

Change it to POST, and the problem will probably go away.

Steps to Reproduce (for bugs)

Create a search index with the field 'type'
Run test:

[Test] public void Test_AWSCloudSearch_GivenReallyLongSearchString() { var search = new AmazonCloudSearchDomainClient("<your-search-domain-here>"); var result = search.Search(new SearchRequest { QueryParser = QueryParser.Structured, Query = "(and type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' )", Start = 1, Size = 15, Sort = "_score desc,objectid asc", Return = "_score,_all_fields" }); Assert.AreNotEqual(result, null); }

Context

We are trying to do a search where words in any field are matched using an or statement. Because there is no way to group terms by field, or an ability to restrict fields conditionally in simple search, we have had to use structured search in this manner, when a long list of keywords are used across a range of fields then the string can become too long for the service to handle.

This is just the first search where we have hit this limit, I am concerned that even if we are able to work around the issue, it exposes a limitation in the ASP.Net library, which could impact whether we choose to use this service in production

Your Environment

• AWSSDK.Core version used: 3.3.21.16
• Service assembly and version used: AWSSDK.CloudSearch 3.3.0.25
• Operating System and version: Windows 10
• Visual Studio version: 2015
• Targeted .NET platform: 4.5

[sstevenkang]

Interesting... thanks for the detailed write up. It looks like CloudSearchDoomain is recommending using GET(the code snipped you included is generated code based on model provided by CSD team). Let me follow up with the service team if it's safe to default to POST.

Also, it looks like we have some gaps where we should re-throw a meaningful exception.

[EyeDreamt]

I also have the same issue. I see this was adjusted to use POST in the Java SDK. Are there plans to update the .Net SDK soon?

[tpund]

This has become a production issue for me as well. Workarounds are failing at this point. Please look into t his.

[franzcars]

This has become an issue for me as well. Please look into this.

[CerebralUnit]

This has become an issue for our team. At least 50% of our queries are failing because we're requesting all of our fields to be returned and it's too much for a GET request. Have you guys made progress on this since March?

[joel-cass]

I gave up on AWS helping me out on this a while ago, checked out their code and created a modified version that uses POST instead of GET. It's very easy to swap in if you're using an IOC framework such as Windsor. This should help if you're in a pinch.

aws-sdk-net-issue-900.zip

[klaytaybai]

@joel-cass, @EyeDreamt, @tpund, @franzcars, @CerebralUnit, and others, sorry we let this issue get lost in the backlog. It looks like this was an issue faced by almost all of our SDKs because we depend on the service models to generate a lot of our code. They specify the HTTP method, which is defaulted to GET for this operation, but the service models don't provide a mechanism for alternative HTTP methods. We should have escalated this as an issue that affects all SDKs when first discovered, but I don't think we did. I'll raise this as a discussion point and see if we can get a fix out for .NET relatively soon.

[phillip-haydon]

@sstevenkang @klaytaybai

aws/aws-sdk-java#660

This was fixed in the Java SDK in 2016

https://aws.amazon.com/releasenotes/release-aws-sdk-for-java-1-11-32/

It's really disappointing to see this isn't resolved, we've been running with our own implementation for like 4 years now and I don't want to maintain code that the AWS SDK should be doing for me.

[brandongregoryscott]

Would also love to see this resolved for the .NET SDK ☝️ We just started running into this with a longer filter query, expecting the underlying request to go out as a POST - two years to the day when it was escalated as a global issue, or supposedly.

[github-actions]

We have noticed this issue has not received attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.

[Kralizek]

@ashishdhingra

This bot has a really unhealthy setup trying to close issues that haven't been solved just because the AWS team forgot about them. Could you please check with your team on how to change its settings so that issues like this one are not closed by mistake?

[phillip-haydon]

The fact that we can implement our own class to resolve this ourselves makes me absolutely baffled why AWS doesn’t want to fix this issue.

[phillip-haydon]

4 years and unresolved…

[joel-cass]

Well AWS isnt exactly kicking goals with this one.

Ive created a PR in an attempt to expedite the process, am not holding my breath in anticipation.