"cloudsearchdomain search" generates invalid signature, 'upload-documents' works

[TiVoMaker]

Latest awscli (1.5.4). I first use "upload-documents" to my search domain, and it works:

['aws', '--region', 'us-west-2', 'cloudsearchdomain', '--endpoint-url', u'http://doc-context-labs-2-2yzynxoc77jbuf4qrnwxbc5ssu.us-west-2.cloudsearch.amazonaws.com', 'upload-documents', '--content-type', 'application/json', '--documents', '/var/folders/r9/78dky0cj71v1dt8g71qft7wm0000gn/T/tmplT0ZDR']

(endpoints obscured on purpose)

Debug output:
logtime="2014-10-28 14:31:16,302" module="boto" level="DEBUG" Using access key found in environment variable.
logtime="2014-10-28 14:31:16,302" module="boto" level="DEBUG" Using secret key found in environment variable.
logtime="2014-10-28 14:31:16,302" module="boto" level="DEBUG" Method: POST
logtime="2014-10-28 14:31:16,303" module="boto" level="DEBUG" Path: /
logtime="2014-10-28 14:31:16,303" module="boto" level="DEBUG" Data:
logtime="2014-10-28 14:31:16,303" module="boto" level="DEBUG" Headers: {}
logtime="2014-10-28 14:31:16,303" module="boto" level="DEBUG" Host: cloudsearch.us-west-2.amazonaws.com
logtime="2014-10-28 14:31:16,303" module="boto" level="DEBUG" Port: 443
logtime="2014-10-28 14:31:16,303" module="boto" level="DEBUG" Params: {'Action': 'DescribeDomains', 'Version': '2013-01-01', 'ContentType': 'JSON', 'DomainNames.member.1': 'context-labs-2'}
logtime="2014-10-28 14:31:16,303" module="boto" level="DEBUG" establishing HTTPS connection: host=cloudsearch.us-west-2.amazonaws.com, kwargs={'port': 443, 'timeout': 70}
logtime="2014-10-28 14:31:16,303" module="boto" level="DEBUG" Token: None
logtime="2014-10-28 14:31:16,303" module="boto" level="DEBUG" CanonicalRequest:
POST
/

host:cloudsearch.us-west-2.amazonaws.com
x-amz-date:20141028T213116Z

host;x-amz-date
da0c116c28d4e60a54b9796c032e16262242fddf4f846719f84250bf7432222e
logtime="2014-10-28 14:31:16,303" module="boto" level="DEBUG" StringToSign:
AWS4-HMAC-SHA256
20141028T213116Z
20141028/us-west-2/cloudsearch/aws4_request
fe958485e5203e203ab3447cbb47060d57156b4e593e3e0d3382c18698742670
logtime="2014-10-28 14:31:16,304" module="boto" level="DEBUG" Signature:
0993b20ab5e7bdb5f6f7ef36c58602f3daea0833f50f268a4c0a13de1d023045
logtime="2014-10-28 14:31:16,304" module="boto" level="DEBUG" Final headers: {'Content-Length': '94', 'User-Agent': 'Boto/2.33.0 Python/2.7.6 Darwin/14.0.0', 'Host': u'cloudsearch.us-west-2.amazonaws.com', 'X-Amz-Date': '20141028T213116Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Authorization': u'AWS4-HMAC-SHA256 Credential=AKIAJXFDHKBFBTCGDBGA/20141028/us-west-2/cloudsearch/aws4_request,SignedHeaders=host;x-amz-date,Signature=0993b20ab5e7bdb5f6f7ef36c58602f3daea0833f50f268a4c0a13de1d023045'}
logtime="2014-10-28 14:31:16,447" module="boto" level="DEBUG" wrapping ssl socket; CA certificate file=/Users/jmb/depots/stuff0/.venv/lib/python2.7/site-packages/boto/cacerts/cacerts.txt
logtime="2014-10-28 14:31:16,534" module="boto" level="DEBUG" validating server certificate: hostname=cloudsearch.us-west-2.amazonaws.com, certificate hosts=['cloudsearch.us-west-2.amazonaws.com']
logtime="2014-10-28 14:31:16,712" module="boto" level="DEBUG" Response headers: [('x-amzn-requestid', 'bfc46e82-5ee9-11e4-a8af-8f18f1271d33'), ('date', 'Tue, 28 Oct 2014 21:31:16 GMT'), ('content-length', '755'), ('content-type', 'application/json')]
logtime="2014-10-28 14:31:16,712" module="boto" level="DEBUG" {"DescribeDomainsResponse":{"DescribeDomainsResult":{"DomainStatusList":[{"ARN":"arn:aws:cloudsearch:us-west-2:351790146037:domain/context-labs-2","Created":true,"Deleted":false,"DocService":{"Endpoint":"doc-context-labs-2-2yzynxoc77jbuf4qrnwxbc5ssu.us-west-2.cloudsearch.amazonaws.com"},"DomainId":"351790146037/context-labs-2","DomainName":"context-labs-2","Limits":{"MaximumPartitionCount":10,"MaximumReplicationCount":5},"Processing":false,"RequiresIndexDocuments":false,"SearchInstanceCount":1,"SearchInstanceType":"search.m1.small","SearchPartitionCount":1,"SearchService":{"Endpoint":"search-context-labs-2-2yzynxoc77jbuf4qrnwxbc5ssu.us-west-2.cloudsearch.amazonaws.com"}}]},"ResponseMetadata":{"RequestId":"bfc46e82-5ee9-11e4-a8af-8f18f1271d33"}}}

Same keys, same terminal window, 'search' fails:

['aws', '--region', 'us-west-2', 'cloudsearchdomain', '--endpoint-url', u'https://search-context-labs-2-2yzynxoc77jbuf4qrnwxbc5ssu.us-west-2.cloudsearch.amazonaws.com', '--output', 'json', 'search', '--query-parser', 'structured', '--filter-query', u"(and user_id:'user_id:1')", '--size', '20', '--return', '_all_fields,_score', '--sort', '_score desc', '--search-query', u"(and 'john')"]

Debug output:
logtime="2014-10-28 15:17:04,417" module="boto" level="DEBUG" Using access key found in environment variable.
logtime="2014-10-28 15:17:04,417" module="boto" level="DEBUG" Using secret key found in environment variable.
logtime="2014-10-28 15:17:04,417" module="boto" level="DEBUG" Method: POST
logtime="2014-10-28 15:17:04,417" module="boto" level="DEBUG" Path: /
logtime="2014-10-28 15:17:04,417" module="boto" level="DEBUG" Data:
logtime="2014-10-28 15:17:04,417" module="boto" level="DEBUG" Headers: {}
logtime="2014-10-28 15:17:04,417" module="boto" level="DEBUG" Host: cloudsearch.us-west-2.amazonaws.com
logtime="2014-10-28 15:17:04,417" module="boto" level="DEBUG" Port: 443
logtime="2014-10-28 15:17:04,417" module="boto" level="DEBUG" Params: {'Action': 'DescribeDomains', 'Version': '2013-01-01', 'ContentType': 'JSON', 'DomainNames.member.1': 'context-labs-2'}
logtime="2014-10-28 15:17:04,417" module="boto" level="DEBUG" establishing HTTPS connection: host=cloudsearch.us-west-2.amazonaws.com, kwargs={'port': 443, 'timeout': 70}
logtime="2014-10-28 15:17:04,417" module="boto" level="DEBUG" Token: None
logtime="2014-10-28 15:17:04,418" module="boto" level="DEBUG" CanonicalRequest:
POST
/

host:cloudsearch.us-west-2.amazonaws.com
x-amz-date:20141028T221704Z

host;x-amz-date
da0c116c28d4e60a54b9796c032e16262242fddf4f846719f84250bf7432222e
logtime="2014-10-28 15:17:04,418" module="boto" level="DEBUG" StringToSign:
AWS4-HMAC-SHA256
20141028T221704Z
20141028/us-west-2/cloudsearch/aws4_request
85beb198ba5836119cbfbb60e95ad440c343fd545e3bd565a5be896e293534e4
logtime="2014-10-28 15:17:04,418" module="boto" level="DEBUG" Signature:
a891e0bfa5bb2e6f388e7b912a86b2e7a6edf22915aebd58799d964796348eee
logtime="2014-10-28 15:17:04,418" module="boto" level="DEBUG" Final headers: {'Content-Length': '94', 'User-Agent': 'Boto/2.33.0 Python/2.7.6 Darwin/14.0.0', 'Host': u'cloudsearch.us-west-2.amazonaws.com', 'X-Amz-Date': '20141028T221704Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Authorization': u'AWS4-HMAC-SHA256 Credential=AKIAJXFDHKBFBTCGDBGA/20141028/us-west-2/cloudsearch/aws4_request,SignedHeaders=host;x-amz-date,Signature=a891e0bfa5bb2e6f388e7b912a86b2e7a6edf22915aebd58799d964796348eee'}
logtime="2014-10-28 15:17:04,504" module="boto" level="DEBUG" wrapping ssl socket; CA certificate file=/Users/jmb/depots/stuff0/.venv/lib/python2.7/site-packages/boto/cacerts/cacerts.txt
logtime="2014-10-28 15:17:04,589" module="boto" level="DEBUG" validating server certificate: hostname=cloudsearch.us-west-2.amazonaws.com, certificate hosts=['cloudsearch.us-west-2.amazonaws.com']
logtime="2014-10-28 15:17:04,740" module="boto" level="DEBUG" Response headers: [('x-amzn-requestid', '25bc0bac-5ef0-11e4-8697-cb6a4f3fdb68'), ('date', 'Tue, 28 Oct 2014 22:17:04 GMT'), ('content-length', '754'), ('content-type', 'application/json')]
logtime="2014-10-28 15:17:04,740" module="boto" level="DEBUG" {"DescribeDomainsResponse":{"DescribeDomainsResult":{"DomainStatusList":[{"ARN":"arn:aws:cloudsearch:us-west-2:351790146037:domain/context-labs-2","Created":true,"Deleted":false,"DocService":{"Endpoint":"doc-context-labs-2-2yzynxoc77jbuf4qrnwxbc5ssu.us-west-2.cloudsearch.amazonaws.com"},"DomainId":"351790146037/context-labs-2","DomainName":"context-labs-2","Limits":{"MaximumPartitionCount":10,"MaximumReplicationCount":5},"Processing":true,"RequiresIndexDocuments":false,"SearchInstanceCount":1,"SearchInstanceType":"search.m1.small","SearchPartitionCount":1,"SearchService":{"Endpoint":"search-context-labs-2-2yzynxoc77jbuf4qrnwxbc5ssu.us-west-2.cloudsearch.amazonaws.com"}}]},"ResponseMetadata":{"RequestId":"25bc0bac-5ef0-11e4-8697-cb6a4f3fdb68"}}}
logtime="2014-10-28 15:17:04,740" module="search.cloudsearch_util" level="INFO" calling out to awscli: ['aws', '--region', 'us-west-2', 'cloudsearchdomain', '--endpoint-url', u'https://search-context-labs-2-2yzynxoc77jbuf4qrnwxbc5ssu.us-west-2.cloudsearch.amazonaws.com', '--output', 'json', 'search', '--query-parser', 'structured', '--filter-query', u"(and user_id:'user_id:1')", '--size', '20', '--return', '_all_fields,_score', '--sort', '_score desc', '--search-query', u"(and 'john')"]

A client error (Unknown) occurred when calling the Search operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'GET
/2013-01-01/search
format=sdk&fq=%28and%20user_id%3A%27user_id%3A1%27%29&pretty=true&q=%28and%20%27john%27%29&q.parser=structured&return=_all_fields%2C_score&size=20&sort=_score%20desc
host:search-context-labs-2-2yzynxoc77jbuf4qrnwxbc5ssu.us-west-2.cloudsearch.amazonaws.com
user-agent:aws-cli/1.5.3 Python/2.7.6 Darwin/14.0.0
x-amz-date:20141028T221705Z

host;user-agent;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20141028T221705Z
20141028/us-west-2/cloudsearch/aws4_request

1f09ab269873c255c30f44f474c44c4397c6b75fcc802abfa8127763fbf30f99'

In case it helps any, I get the exact same failure mode when running on an EC2 instance with an instance profile set to allow full domain access. From the debug output, it finds the access token just fine, upload-documents works, search fails identically to above.

[catufunwa]

Any update to this? I notice it happens if you use the --query-parser option it breaks the auth check.

[jamesls]

Taking a look.

[jamesls]

Confirmed, working on fix.