*** closed stream Error comes randomly at deployment

[kumarrai9]

Hi All,

Sometimes i am getting the mentioned Closed Stream error in between the deployment executions. Specified the versions of Ruby capistrano & net-ssh that i am using. Please help me on this.

Ruby : ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
Capistrano :v2.5.9
net-ssh : 2.6.7

servers: ["10.208.67.46", "10.208.67.47", "10.208.67.50", "10.208.67.51", "10.208.68.210", "10.208.64.204", "10.208.65.191"]
[10.208.65.191:2525] executing command
*** closed stream
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/net-ssh-2.6.7/lib/net/ssh/ruby_compat.rb:41:in select' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/net-ssh-2.6.7/lib/net/ssh/ruby_compat.rb:41:inio_select'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/net-ssh-2.6.7/lib/net/ssh/ruby_compat.rb:40:in synchronize' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/net-ssh-2.6.7/lib/net/ssh/ruby_compat.rb:40:inio_select'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/net-ssh-2.6.7/lib/net/ssh/transport/packet_stream.rb:73:in available_for_read?' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/net-ssh-2.6.7/lib/net/ssh/transport/packet_stream.rb:85:innext_packet'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/net-ssh-2.6.7/lib/net/ssh/transport/session.rb:172:in poll_message' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/net-ssh-2.6.7/lib/net/ssh/transport/session.rb:167:inloop'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/net-ssh-2.6.7/lib/net/ssh/transport/session.rb:167:in poll_message' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:454:indispatch_incoming_packets'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:216:in preprocess' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/processable.rb:17:inprocess_iteration'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/processable.rb:43:in ensure_each_session' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/processable.rb:41:ineach'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/processable.rb:41:in ensure_each_session' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/processable.rb:17:inprocess_iteration'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/command.rb:165:in process!' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/command.rb:164:inloop'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/command.rb:164:in process!' *** /usr/lib/ruby/1.8/benchmark.rb:308:inrealtime'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/command.rb:163:in process!' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/command.rb:134:inprocess'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/actions/invocation.rb:175:in run_tree' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/connections.rb:192:inexecute_on_servers'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/activerecord-2.3.11/lib/active_record/base.rb:2236:in each_slice' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/connections.rb:180:ineach'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/connections.rb:180:in each_slice' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/connections.rb:180:inexecute_on_servers'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/actions/invocation.rb:173:in run_tree' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/actions/invocation.rb:145:inrun'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/namespaces.rb:186:in send' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/namespaces.rb:186:inmethod_missing'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/namespaces.rb:186:in send' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/namespaces.rb:186:inmethod_missing'
*** :87:in load' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/execution.rb:139:ininstance_eval'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/execution.rb:139:in invoke_task_directly_without_callbacks' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/callbacks.rb:27:ininvoke_task_directly'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/execution.rb:89:in execute_task' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/namespaces.rb:186:insend'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/namespaces.rb:186:in method_missing' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/namespaces.rb:186:insend'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/namespaces.rb:186:in method_missing' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/namespaces.rb:104:insymlink'
*** :9:in load' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/execution.rb:139:ininstance_eval'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/execution.rb:139:in invoke_task_directly_without_callbacks' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/callbacks.rb:27:ininvoke_task_directly'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/execution.rb:89:in execute_task' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/configuration/execution.rb:101:infind_and_execute_task'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/cli/execute.rb:46:in execute_requested_actions' *** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/cli/execute.rb:45:ineach'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/capistrano-2.6.0/lib/capistrano/cli/execute.rb:45:in execute_requested_actions' *** /usr/local/webistrano-master/lib/webistrano/deployer.rb:72:inexecute!'
*** /usr/local/webistrano-master/lib/webistrano/deployer.rb:62:in catch' *** /usr/local/webistrano-master/lib/webistrano/deployer.rb:62:inexecute!'
*** /usr/local/webistrano-master/lib/webistrano/deployer.rb:46:in invoke_task!' *** (eval):1 *** script/runner:3:ineval'
*** /usr/local/webistrano-master/vendor/bundler/ruby/1.8/gems/rails-2.3.11/lib/commands/runner.rb:48
*** script/runner:3:in `require'
*** script/runner:3

[kumarrai9]

I am using Webistrano for deployment

[delano]

It's difficult to say without more information but it sounds like it could be related to the network.

It would be helpful if you try again with different versions of ruby, net-ssh, and capistrano. Also check your deployment script to see if you have any sections that hang for say 30 seconds or more (that's about the minimum I've seen for networks dropping inactive connections).

[kumarrai9]

Thanks Delano.
What kind of info that you want to dig further, i can try to provide such info..

I have setup more than 100 projects in Webistrano but issues generally comes in 2 projects sometimes (rest of the others projects never faced such issues closed stream error failed) and these 2 projects recipes are big compare to other projects and some of the commands in the recipes could be taking more than 30 sec to execute. Is there any way to increase commands executions timeout in such kind of cases.

Is there any way to trace network issues from Webistrano logs or ? (We use IPSec tunnel and ISP traffic handle by Sonic Firewall Appliance), Without trace & logs, it's very difficult to give proof to Network team to fix this issues.

[delano]

I've never used Webistrano so I can't provide any guidance there, but I'd suggest trying to run it with Capistrano directly (if that's possible) or at least with 1 different version of Ruby and/or net-ssh and/or capistrano.

My hunch is still that there is a long running command. Check with your network folks about the idle timeout at that level and also check your sshd config:

http://blog.hwarf.com/2009/06/howto-set-sshd-idle-timeout.html