New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Background worker - Clair request timeouts crashes background worker #1751

Closed
owenhaynes opened this Issue Apr 2, 2018 · 5 comments

Comments

Projects
2 participants
@owenhaynes

owenhaynes commented Apr 2, 2018

The background worker crashes when a request timeout in Clair happens, this seems to happen when processing layers which are big e.g 600mb+.

The Clair API request time out is set to the default of 900 seconds. The background worker timeout before this time has elapsed. I assume Portus is using a smaller client timeout.

I would expect the background worker not to crash because of a timeout as well.

Portus version 2.3.1

Running under Kubernetes 1.9.4

Error:

/usr/lib64/ruby/2.5.0/net/protocol.rb:181:in rbuf_fill': Net::ReadTimeout (Net::ReadTimeout) from /usr/lib64/ruby/2.5.0/net/protocol.rb:157:in readuntil'
from /usr/lib64/ruby/2.5.0/net/protocol.rb:167:in readline' from /usr/lib64/ruby/2.5.0/net/http/response.rb:40:in read_status_line'
from /usr/lib64/ruby/2.5.0/net/http/response.rb:29:in read_new' from /usr/lib64/ruby/2.5.0/net/http.rb:1494:in block in transport_request'
from /usr/lib64/ruby/2.5.0/net/http.rb:1491:in catch' from /usr/lib64/ruby/2.5.0/net/http.rb:1491:in transport_request'
from /usr/lib64/ruby/2.5.0/net/http.rb:1464:in request' from /srv/Portus/lib/portus/http_helpers.rb:160:in block in get_response_token'
from /usr/lib64/ruby/2.5.0/net/http.rb:910:in start' from /usr/lib64/ruby/2.5.0/net/http.rb:609:in start'
from /srv/Portus/lib/portus/http_helpers.rb:159:in get_response_token' from /srv/Portus/lib/portus/security_backends/clair.rb:97:in post_layer'
from /srv/Portus/lib/portus/security_backends/clair.rb:28:in block in vulnerabilities' from /srv/Portus/lib/portus/security_backends/clair.rb:28:in each_index'
from /srv/Portus/lib/portus/security_backends/clair.rb:28:in vulnerabilities' from /srv/Portus/lib/portus/security.rb:57:in block in vulnerabilities'
from /srv/Portus/lib/portus/security.rb:55:in each' from /srv/Portus/lib/portus/security.rb:55:in vulnerabilities'
from /srv/Portus/lib/portus/background/security_scanning.rb:47:in block in execute!' from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in block (2 levels) in find_each'
from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in each' from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in block in find_each'
from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in find_in_batches' from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in find_each'
from /srv/Portus/lib/portus/background/security_scanning.rb:35:in execute!' from /srv/Portus/bin/background.rb:56:in block (2 levels) in <top (required)>'
from /srv/Portus/bin/background.rb:54:in each' from /srv/Portus/bin/background.rb:54:in each_with_index'
from /srv/Portus/bin/background.rb:54:in block in <top (required)>' from /srv/Portus/bin/background.rb:53:in loop'
from /srv/Portus/bin/background.rb:53:in <top (required)>' from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/runner.rb:60:in load'
from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/runner.rb:60:in <top (required)>' from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:123:in require'
from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:123:in require_command!' from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:90:in runner'
from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:39:in run_command!' from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands.rb:17:in <top (required)>'
from bin/rails:12:in require' from bin/rails:12:in

'
exit status 1

@mssola mssola added the bug label Apr 5, 2018

@mssola

This comment has been minimized.

Contributor

mssola commented Apr 5, 2018

You are totally right. Thanks 👍

mssola added a commit to mssola/Portus that referenced this issue Apr 5, 2018

security: don't crash on clair timeouts
Moreover, I've also added a configuration option so the timeouts for the
registry and clair can be configured separately. This has been done
because the Clair timeout by default is 900 seconds, which might be a
bit too high for registries.

Fixes SUSE#1751

Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>

mssola added a commit to mssola/Portus that referenced this issue Apr 5, 2018

security: don't crash on clair timeouts
Moreover, I've also added a configuration option so the timeouts for the
registry and clair can be configured separately. This has been done
because the Clair timeout by default is 900 seconds, which might be a
bit too high for registries.

Fixes SUSE#1751

Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>
@mssola

This comment has been minimized.

Contributor

mssola commented Apr 5, 2018

PR #1762 should fix this. When merged, we will cherry pick it into the v2.3 branch so it's available in the 2.3 Docker image too.

@mssola mssola added this to In progress in 2.4 Apr 5, 2018

mssola added a commit to mssola/Portus that referenced this issue Apr 5, 2018

security: don't crash on clair timeouts
Moreover, I've also added a configuration option so the timeouts for the
registry and clair can be configured separately. This has been done
because the Clair timeout by default is 900 seconds, which might be a
bit too high for registries.

Fixes SUSE#1751

Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>

mssola added a commit to mssola/Portus that referenced this issue Apr 5, 2018

security: don't crash on clair timeouts
Moreover, I've also added a configuration option so the timeouts for the
registry and clair can be configured separately. This has been done
because the Clair timeout by default is 900 seconds, which might be a
bit too high for registries.

Fixes SUSE#1751

Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>

@mssola mssola closed this in #1762 Apr 6, 2018

mssola added a commit that referenced this issue Apr 6, 2018

security: don't crash on clair timeouts
Moreover, I've also added a configuration option so the timeouts for the
registry and clair can be configured separately. This has been done
because the Clair timeout by default is 900 seconds, which might be a
bit too high for registries.

Fixes #1751

Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>
@mssola

This comment has been minimized.

Contributor

mssola commented Apr 6, 2018

@owenhaynes the 2.3 image has already been updated. Could you pull the image again and tell me if you can still reproduce this issue ?

@mssola

This comment has been minimized.

Contributor

mssola commented Apr 6, 2018

After some time we will release a proper 2.3.4 release with this fix included 👍

@owenhaynes

This comment has been minimized.

owenhaynes commented Apr 7, 2018

@mssola mssola moved this from In progress to Done in 2.4 Apr 9, 2018

vitoravelino added a commit to vitoravelino/Portus that referenced this issue Apr 12, 2018

security: don't crash on clair timeouts
Moreover, I've also added a configuration option so the timeouts for the
registry and clair can be configured separately. This has been done
because the Clair timeout by default is 900 seconds, which might be a
bit too high for registries.

Fixes SUSE#1751

Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment