Potential bug in handling of rate-limit errors

[cyclotron3k]

I've been playing with this a bit and it generally works very well, but I get intermittent errors, and they seem to occur when I'm pushing it quite hard and (presumably) causing the AWS API gateway to throw rate limit errors.

I see errors about unexpected EOF.

Is this something you have seen too? Are you accepting PRs?

[saluzafa]

Hello @cyclotron3k !
I didn't experienced that, but I'm not using it in the context of a high volume of requests.
Feel free to open a pull request, it will be received with pleasure 😄 .

Cheers!

[cyclotron3k]

I've been digging into this a bit recently. My notes so far:

• I can reliably recreate the issue by sending hundreds of "describe stack" requests to the cloud formation API.

• Rate limit exceptions are being thrown and handled correctly, but eventually I see the following

  4m    error: Async::Task [oid=0x4ff24] [ec=0x45f10] [pid=7] [2021-11-22 01:25:04 +0000]
           |   EOFError: Could not read line!
           |   → /usr/local/bundle/gems/async-http-0.56.5/lib/async/http/protocol/http1/connection.rb:55 in `read_line'
           |     /usr/local/bundle/gems/protocol-http1-0.14.2/lib/protocol/http1/connection.rb:201 in `read_response'
           |     /usr/local/bundle/gems/async-http-0.56.5/lib/async/http/protocol/http1/response.rb:31 in `read'
           |     /usr/local/bundle/gems/async-http-0.56.5/lib/async/http/protocol/http1/client.rb:78 in `call'
           |     /usr/local/bundle/gems/protocol-http-0.22.5/lib/protocol/http/request.rb:53 in `call'
           |     /usr/local/bundle/gems/async-http-0.56.5/lib/async/http/client.rb:143 in `make_response'
           |     /usr/local/bundle/gems/async-http-0.56.5/lib/async/http/client.rb:106 in `call'
           |     /usr/local/bundle/gems/async-aws-1.0.2/lib/async/aws/http_handler.rb:41 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/content_length.rb:24 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/request_callback.rb:85 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/xml/error_handler.rb:10 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/signature_v4.rb:58 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/transfer_encoding.rb:26 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:349 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:383 in `retry_request'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:371 in `retry_if_possible'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:360 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:383 in `retry_request'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:371 in `retry_if_possible'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:360 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:383 in `retry_request'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:371 in `retry_if_possible'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:360 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:383 in `retry_request'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:371 in `retry_if_possible'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:360 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/http_checksum.rb:18 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/query/handler.rb:30 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/user_agent.rb:13 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/endpoint_discovery.rb:80 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/endpoint.rb:47 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/param_validator.rb:26 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/logging.rb:41 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/raise_response_errors.rb:16 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/param_converter.rb:26 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/request_callback.rb:71 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/response_paging.rb:12 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/response_target.rb:24 in `call'
           |     /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/request.rb:72 in `send_request'
           |     /usr/local/bundle/gems/aws-sdk-cloudformation-1.61.0/lib/aws-sdk-cloudformation/client.rb:3051 in `describe_stacks'
           |     stress_test.rb:106 in `_get_stack_parameters'
           |     stress_test.rb:62 in `get_stack_parameters'
           |     stress_test.rb:42 in `block (4 levels) in run'
           |     /usr/local/bundle/gems/async-1.30.1/lib/async/task.rb:260 in `block in make_fiber'
  

[cyclotron3k]

I've noticed that the issue usually happens 4-5 minutes into a stress test. It doesn't seem to be correlated with how many connections I'm allowing, and what the backoff parameters are - which is unusual. This could be a red herring though.