Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chef-15: Bootstrapping failed due to SSH connection timeout in AIX 7 #8764

Open
vsingh-msys opened this issue Jul 23, 2019 · 1 comment

Comments

@vsingh-msys
Copy link
Contributor

commented Jul 23, 2019

Description

Bootstrapping of AIX 7 node raises the SSH command failed (Timeout, server IP_ADDRESS not responding.) (Train::Transports::SSHFailed)

Chef Version

15.1.36

Platform Version

AIX Version 7.1

Replication Case

knife bootstrap IP_ADDRESS --connection-user username -N aix-7 -r "recipe[starter]" --connection-password password --ssh-verify-host-key 'never' -VV

Stacktrace

DEBUG: Checking if we need to accept Chef license to bootstrap node
DEBUG: Reading products and relationships...
DEBUG: Successfully read products and relationships
DEBUG: License acceptance required for chef version: 15
DEBUG: Searching for the following licenses: ["infra-client", "inspec"]
DEBUG: Found license chef_infra_client at /.chef/accepted_licenses/chef_infra_client
DEBUG: Found license inspec at /.chef/accepted_licenses/inspec
DEBUG: Missing licenses remaining: []
DEBUG: All licenses present
Connecting to IP_ADDRESS
WARN: [SSH] PTY requested: stderr will be merged into stdout
DEBUG: [SSH] username@IP_ADDRESS<{:user_known_hosts_file=>"/dev/null", :port=>22, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>60, :auth_methods=>["none", "password", "keyboard-interactive"], :keys_only=>nil, :keys=>[], :password=>"<hidden>", :forward_agent=>nil, :non_interactive=>true, :verify_host_key=>:never}> (cmd.exe /c ver)
DEBUG: [SSH] opening connection to username@IP_ADDRESS<{:user_known_hosts_file=>"/dev/null", :port=>22, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>60, :auth_methods=>["none", "password", "keyboard-interactive"], :keys_only=>nil, :keys=>[], :password=>"<hidden>", :forward_agent=>nil, :non_interactive=>true, :verify_host_key=>:never}>
DEBUG: [SSH] username@IP_ADDRESS<{:user_known_hosts_file=>"/dev/null", :port=>22, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>60, :auth_methods=>["none", "password", "keyboard-interactive"], :keys_only=>nil, :keys=>[], :password=>"<hidden>", :forward_agent=>nil, :non_interactive=>true, :verify_host_key=>:never}> (uname -s)
DEBUG: [SSH] username@IP_ADDRESS<{:user_known_hosts_file=>"/dev/null", :port=>22, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>60, :auth_methods=>["none", "password", "keyboard-interactive"], :keys_only=>nil, :keys=>[], :password=>"<hidden>", :forward_agent=>nil, :non_interactive=>true, :verify_host_key=>:never}> (uname -m)

Traceback (most recent call last):
  40: from /opt/chef/embedded/bin/knife:23:in `<main>'
  39: from /opt/chef/embedded/bin/knife:23:in `load'
  38: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/bin/knife:24:in `<top (required)>'
  37: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/application/knife.rb:162:in `run'
  36: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife.rb:222:in `run'
  35: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife.rb:474:in `run_with_pretty_exceptions'
  34: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/local_mode.rb:42:in `with_server_connectivity'
  33: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife.rb:475:in `block in run_with_pretty_exceptions'
  32: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife/bootstrap.rb:568:in `run'
  31: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife/bootstrap.rb:619:in `connect!'
  30: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife/bootstrap.rb:672:in `do_connect'
  29: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife/bootstrap/train_connector.rb:69:in `connect!'
  28: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife/bootstrap/train_connector.rb:56:in `connection'
  27: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh.rb:82:in `connection'
  26: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh.rb:238:in `create_new_connection'
  25: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh.rb:238:in `new'
  24: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh_connection.rb:53:in `initialize'
  23: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/extras/command_wrapper.rb:165:in `load'
  22: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/plugins/base_connection.rb:110:in `platform'
  21: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect.rb:9:in `scan'
  20: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:27:in `scan'
  19: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:27:in `each'
  18: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:33:in `block in scan'
  17: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:45:in `scan_children'
  16: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:45:in `each'
  15: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:46:in `block in scan_children'
  14: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:46:in `instance_eval'
  13: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/specifications/os.rb:47:in `block in load'
  12: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/helpers/os_common.rb:56:in `unix_uname_m'
  11: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/helpers/os_common.rb:33:in `command_output'
  10: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/plugins/base_connection.rb:122:in `run_command'
   9: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh_connection.rb:217:in `run_command_via_connection'
   8: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh_connection.rb:309:in `execute_on_channel'
   7: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh/connection/session.rb:181:in `loop'
   6: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh/connection/session.rb:181:in `loop'
   5: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh/connection/session.rb:181:in `block in loop'
   4: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh/connection/session.rb:228:in `process'
   3: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh/connection/event_loop.rb:31:in `process'
   2: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh/connection/event_loop.rb:113:in `ev_select_and_postprocess'
   1: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh/connection/session.rb:290:in `ev_do_postprocess'
/opt/chef/embedded/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh/connection/keepalive.rb:52:in `send_as_needed': Timeout, server IP_ADDRESS not responding. (Net::SSH::Timeout)
  32: from /opt/chef/embedded/bin/knife:23:in `<main>'
  31: from /opt/chef/embedded/bin/knife:23:in `load'
  30: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/bin/knife:24:in `<top (required)>'
  29: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/application/knife.rb:162:in `run'
  28: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife.rb:222:in `run'
  27: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife.rb:474:in `run_with_pretty_exceptions'
  26: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/local_mode.rb:42:in `with_server_connectivity'
  25: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife.rb:475:in `block in run_with_pretty_exceptions'
  24: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife/bootstrap.rb:568:in `run'
  23: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife/bootstrap.rb:619:in `connect!'
  22: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife/bootstrap.rb:672:in `do_connect'
  21: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife/bootstrap/train_connector.rb:69:in `connect!'
  20: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.1.36/lib/chef/knife/bootstrap/train_connector.rb:56:in `connection'
  19: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh.rb:82:in `connection'
  18: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh.rb:238:in `create_new_connection'
  17: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh.rb:238:in `new'
  16: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh_connection.rb:53:in `initialize'
  15: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/extras/command_wrapper.rb:165:in `load'
  14: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/plugins/base_connection.rb:110:in `platform'
  13: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect.rb:9:in `scan'
  12: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:27:in `scan'
  11: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:27:in `each'
  10: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:33:in `block in scan'
   9: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:45:in `scan_children'
   8: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:45:in `each'
   7: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:46:in `block in scan_children'
   6: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/scanner.rb:46:in `instance_eval'
   5: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/specifications/os.rb:47:in `block in load'
   4: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/helpers/os_common.rb:56:in `unix_uname_m'
   3: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/platforms/detect/helpers/os_common.rb:33:in `command_output'
   2: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/plugins/base_connection.rb:122:in `run_command'
   1: from /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh_connection.rb:212:in `run_command_via_connection'
/opt/chef/embedded/lib/ruby/gems/2.6.0/gems/train-core-2.1.13/lib/train/transports/ssh_connection.rb:224:in `rescue in run_command_via_connection': SSH command failed (Timeout, server IP_ADDRESS not responding.) (Train::Transports::SSHFailed)
@vsingh-msys

This comment has been minimized.

Copy link
Contributor Author

commented Aug 2, 2019

Train::Transports::SSHFailed issue in AIX node due to setting pty: true for AIX(Unix).

In order to fixed need to set pty: false for AIX node as well but we can't determine at the time of SSH connection from chef side.

Another issue found while creating temp dir

MKTEMP_NIX_COMMAND ||= "bash -c 'd=$(mktemp -d ${TMPDIR:-/tmp}/chef_XXXXXX); echo $d'".freeze

ERROR: bash: mktemp: command not found

In order to make it work, we have to update the command for creating temp dir

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.