Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Socket error due timeout kills the pipeline instead to go in backoff retry #55

Closed
andsel opened this issue Oct 9, 2020 · 0 comments · Fixed by #57
Closed

Socket error due timeout kills the pipeline instead to go in backoff retry #55

andsel opened this issue Oct 9, 2020 · 0 comments · Fixed by #57
Labels
bug int-shortlist

Comments

@andsel
Copy link
Contributor

andsel commented Oct 9, 2020

  • Version: <= 3.1.2

Sometimes is the SQS plugin is in run and encounter an error from the HTTP client library, the plugin doesn't retry to reconnect but kills the pipeline.

stack trace 
[2020-09-26T00:17:14,174][ERROR][logstash.javapipeline    ][main] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::SQS threads=>4, 
id=>"1e95d5586024e71da3c6c432de1c06d5ed6f9b05af26dfe45bfe945c182278f0", 
region=>"eu-west-1", queue=>"MonSysCloudwatchMetricsQueue", enable_metric=>true, 
codec=><LogStash::Codecs::JSON 
     id=>"json_6f7b71ca-94a4-4dcc-8b24-debafa66959d", 
    enable_metric=>true, 
    charset=>"UTF-8">, 
role_session_name=>"logstash", 
polling_frequency=>20
>
  Error: Net::OpenTimeout
  Exception: Seahorse::Client::NetworkingError
  Stack: uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:41:in `ssl_socket_connect'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:985:in `connect'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in `do_start'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:919:in `start'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83:in `method_missing'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/net_http/connection_pool.rb:285:in `start_session'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/net_http/connection_pool.rb:92:in `session_for'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/net_http/handler.rb:119:in `session'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/net_http/handler.rb:71:in `transmit'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/net_http/handler.rb:45:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/plugins/content_length.rb:12:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/request_signer.rb:88:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:108:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/sqs_queue_urls.rb:13:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/query/handler.rb:27:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/endpoint_pattern.rb:27:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/endpoint_discovery.rb:67:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/plugins/endpoint.rb:41:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/plugins/response_target.rb:21:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/request.rb:70:in `send_request'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/base.rb:207:in `block in define_operation_methods'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.429/lib/aws-sdk-resources/services/sqs/queue_poller.rb:390:in `send_request'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.429/lib/aws-sdk-resources/services/sqs/queue_poller.rb:383:in `get_messages'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.429/lib/aws-sdk-resources/services/sqs/queue_poller.rb:332:in `block in poll'
org/jruby/RubyKernel.java:1446:in `loop'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.429/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331:in `block in poll'
org/jruby/RubyKernel.java:1193:in `catch'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.429/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330:in `poll'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-sqs-3.1.2/lib/logstash/inputs/sqs.rb:143:in `block in run'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-sqs-3.1.2/lib/logstash/inputs/sqs.rb:164:in `run_with_backoff'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-sqs-3.1.2/lib/logstash/inputs/sqs.rb:142:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:328:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:320:in `block in start_input'

the problem is that the client throws Seahorse::Client::NetworkingError in case of Net::OpenTimeout but the rescue code

logstash-input-sqs/lib/logstash/inputs/sqs.rb

Line 166 in f21a643

rescue Aws::SQS::Errors::ServiceError => e
handle only subclasses of Aws::SQS::Errors::ServiceError
@andsel andsel added the bug label Oct 9, 2020
This was referenced Nov 16, 2020
Closed
Merged
@kares kares closed this as completed in #57 Nov 18, 2020
kares added a commit that referenced this issue Nov 18, 2020
@kares
Fix: retry networking errors (with backoff) (#57)
6ea3485 
in case of networking errors such as a timeout or a (temporary) failure to resolve DNS names the plugin should retry

resolves #56
resolves #55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug int-shortlist
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix: retry networking errors (with backoff) kares/logstash-input-sqs
2 participants
@andsel @roaksoax