New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

endOfInput error when calling ListObjects against localstack/moto mock of S3 service #432

Open
rcook opened this Issue Dec 18, 2017 · 0 comments

Comments

Projects
None yet
1 participant
@rcook
Contributor

rcook commented Dec 18, 2017

I don't know if this is a bug in amazonka or a bug in the emulation of S3 by localstack/moto, so I apologize in advance if this bug is going to the wrong place.

Anyway, I've been using localstack for local development of my AWS client apps. This platform's S3 implementation is, I believe, built on moto. It's quite a common platform for local/dev testing of AWS apps and I've run into this problem trying to use the ListObjects API against it.

Repro program

See https://gist.github.com/rcook/21ec658bce70cc3841de2e591de5b79f#file-main-hs. This full project can be cloned and built locally as follows:

git clone https://gist.github.com/553ae2f596bbad3852f5f6d2edd3a257.git amazonka-localstack-repro
cd amazonka-localstack-repro
stack build
stack exec s3-app

Repro steps

  1. Repro steps work on Ubuntu and Linux Mint
  2. Follow instructions at https://github.com/localstack/localstack to install localstack
  3. Start localstack with localstack
  4. Once localstack is running, running repro program.
  5. Run repro program with stack exec s3-app.

Here's the output:

CreateBucket
[Client Request] {
  host      = localhost:4572
  secure    = False
  method    = PUT
  target    = Nothing
  timeout   = ResponseTimeoutMicro 70000000
  redirects = 0
  path      = /mybucket
  query     = 
  headers   = host: localhost; x-amz-date: 20171218T183702Z; x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855; authorization: AWS4-HMAC-SHA256 Credential=AKIAJXT7OAEJFXCG7QOA/20171218/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=983e522d9b1abc00bed3aa8f2c03714ac69fcfde58be022b2d8ea7eec4dd914a
  body      = 
}
[Client Response] {
  status  = 200 OK
  headers = server: BaseHTTP/0.3 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; content-type: text/html; charset=utf-8; content-length: 162; server: Werkzeug/0.13 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; access-control-allow-origin: *; access-control-allow-methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH; access-control-allow-headers: authorization,content-type,content-md5,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
}
[Client Request] {
  host      = localhost:4572
  secure    = False
  method    = HEAD
  target    = Nothing
  timeout   = ResponseTimeoutMicro 70000000
  redirects = 0
  path      = /mybucket
  query     = 
  headers   = host: localhost; x-amz-date: 20171218T183702Z; x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855; authorization: AWS4-HMAC-SHA256 Credential=AKIAJXT7OAEJFXCG7QOA/20171218/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2abc727f586521b7ed33c67b5e982b688a80692a6899eb242d2a76200a394255
  body      = 
}
[Client Response] {
  status  = 200 OK
  headers = server: BaseHTTP/0.3 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; content-type: text/html; charset=utf-8; content-length: 0; server: Werkzeug/0.13 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; access-control-allow-origin: *; access-control-allow-methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH; access-control-allow-headers: authorization,content-type,content-md5,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
}
[Await BucketExists] Success after 1 attempts.
ListObjects
[Client Request] {
  host      = localhost:4572
  secure    = False
  method    = GET
  target    = Nothing
  timeout   = ResponseTimeoutMicro 70000000
  redirects = 0
  path      = /mybucket
  query     = ?list-type%3D2=
  headers   = host: localhost; x-amz-date: 20171218T183702Z; x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855; authorization: AWS4-HMAC-SHA256 Credential=AKIAJXT7OAEJFXCG7QOA/20171218/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=11ffdb07eb5ea708ec382159188fcdbb693fa881986ccd15e8c4ad25b5e2215e
  body      = 
}
[Client Response] {
  status  = 200 OK
  headers = server: BaseHTTP/0.3 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; content-type: text/html; charset=utf-8; content-length: 248; server: Werkzeug/0.13 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; access-control-allow-origin: *; access-control-allow-methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH; access-control-allow-headers: authorization,content-type,content-md5,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
}
[Raw Response Body] {
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>mybucket</Name><Prefix>None</Prefix><MaxKeys>1000</MaxKeys><Delimiter>None</Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>
}
[SerializeError] {
  service = S3
  status  = 200 OK
  message = endOfInput
  body    = Just <?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>mybucket</Name><Prefix>None</Prefix><MaxKeys>1000</MaxKeys><Delimiter>None</Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>
}
s3-app: SerializeError (SerializeError' {_serializeAbbrev = Abbrev "S3", _serializeStatus = Status {statusCode = 200, statusMessage = "OK"}, _serializeBody = Just "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<ListBucketResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Name>mybucket</Name><Prefix>None</Prefix><MaxKeys>1000</MaxKeys><Delimiter>None</Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>", _serializeMessage = "endOfInput"})

You'll see that this fails with an endOfInput error. This example works just fine against a real AWS S3 instance.

@rcook rcook changed the title from endOfInput error when calling listObjects against localstack/moto mock of S3 service to endOfInput error when calling ListObjects against localstack/moto mock of S3 service Dec 18, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment