Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

knife bootstrap Net::SSH::ConnectionTimeout #8721

Open
johnreed00 opened this issue Nov 29, 2018 · 3 comments

Comments

Projects
None yet
4 participants
@johnreed00
Copy link

commented Nov 29, 2018

Description

Boostrapping a node runs into issue:

$ knife bootstrap 192.169.1.1 -N domain.local -x chefadmin --sudo
ERROR: Net::SSH::ConnectionTimeout: Net::SSH::ConnectionTimeout

Briefly describe the issue

The issue is similar to #1174 - the same command and error.
First was created chef-server with chefdk and workstation on it.
Running on virtual machines in Asure.

ChefDK Version

Tell us which version of the ChefDK you are running. Run chef --version to display the version.

$ chef --version
Chef Development Kit Version: 3.5.13
chef-client version: 14.7.17
delivery version: master (6862f27aba89109a9630f0b6c6798efec56b4efe)
berks version: 7.0.6
kitchen version: 1.23.2
inspec version: 3.0.52

Platform Version

Tell us which operating system distribution and version ChefDK is running on.

server/workstation/nodes:
CentOS Linux 7 (Core) 

Name        : chefdk
Version     : 3.5.13
Release     : 1.el7
Architecture: x86_64

Replication Case

Tell us what steps to take to replicate your problem. See How to create a Minimal, Complete, and Verifiable example for information on how to create a good replication case.

Create Chef Server
Create & Configure workstation with Starter pak
Confirm above
Create Node1
Configure SSH access
Attempt to bootstrap (both password and key tried, both user and root tried)

Stacktrace

username: 'chefadmin@DOMAIN.LOCAL'
node: node1.domain.local

Please include the stacktrace.out output or link to a gist of it, if there is one.

$ knife bootstrap 192.169.1.9 -N node1.domain.local -x chefadmin --sudo
or
$ knife bootstrap node1.domain.local --ssh-user 'chefadmin@DOMAIN.LOCAL' --ssh-password 'qwerty' --node-name node1
or
$ knife bootstrap 192.169.1.9 -x 'chefadmin@DOMAIN.LOCAL' -i key.rsa --sudo -N node1.domain.local -V --no-fips

Creating new client for node1.domain.local 
Creating new node for node1.domain.local 
Connecting to 192.169.1.9
#<Thread:0x0000000007065408@/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
        8: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
        7: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/net-ssh-multi-1.2.1/lib/net/ssh/multi/session_actions.rb:36:in `block (2 levels) in sessions'
        6: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/net-ssh-multi-1.2.1/lib/net/ssh/multi/server.rb:138:in `session'
        5: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/net-ssh-multi-1.2.1/lib/net/ssh/multi/session.rb:488:in `next_session'
        4: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/net-ssh-multi-1.2.1/lib/net/ssh/multi/server.rb:186:in `new_session'
        3: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/net-ssh-4.2.0/lib/net/ssh.rb:237:in `start'
        2: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/net-ssh-4.2.0/lib/net/ssh.rb:237:in `new'
        1: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/net-ssh-4.2.0/lib/net/ssh/transport/session.rb:56:in `initialize'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/net-ssh-4.2.0/lib/net/ssh/transport/session.rb:90:in `rescue in initialize': Net::SSH::ConnectionTimeout (Net::SSH::ConnectionTimeout)
ERROR: Net::SSH::ConnectionTimeout: Net::SSH::ConnectionTimeout

sshd DEBUG3 get no logs.
tcpdump get SYN, SYN ACK and RESET packets only (!) on both sides. May be it's threading issue(?).
script mentioned in #1174 works too.

NOTE: CHEFDK BUGS ONLY

This issue tracker is for the code contained within this repo -- chefdk.

@krudskoy

This comment has been minimized.

Copy link

commented Nov 30, 2018

Solution: Set the SSH Client parameter "ConnectTimeout" to more than 0.(We put 30)

Debug:

As we can see the problem in the session file inside the initialize function.

/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/net-ssh-4.2.0/lib/net/ssh/transport/session.rb:90:in `rescue in initialize': Net::SSH::ConnectionTimeout (Net::SSH::ConnectionTimeout)
ERROR: Net::SSH::ConnectionTimeout: Net::SSH::ConnectionTimeout

Here it is:

def initialize(host, options={})
      self.logger = options[:logger]

      @host = host
      @port = options[:port] || DEFAULT_PORT
      @bind_address = options[:bind_address] || nil
      @options = options

      @socket =
        if (factory = options[:proxy])
          debug { "establishing connection to #{@host}:#{@port} through proxy" }
          factory.open(@host, @port, options)
        else
          debug { "establishing connection to #{@host}:#{@port}" }
          Socket.tcp(@host, @port, @bind_address, nil,
                     connect_timeout: options[:timeout])
        end
      @socket.extend(PacketStream)
      @socket.logger = @logger

      debug { "connection established" }

      @queue = []

      @host_key_verifier = select_host_key_verifier(options[:verify_host_key])


      @server_version = ServerVersion.new(socket, logger, options[:timeout])

      @algorithms = Algorithms.new(self, options)
      @algorithms.start
      wait { algorithms.initialized? }
    rescue Errno::ETIMEDOUT
      raise Net::SSH::ConnectionTimeout
    end

This function return error Net::SSH::ConnectionTimeout for any error within the function. That's what we saw above.

Also we found out that ssh connection always reset by the server node. It says that negotiation process have some problems.

17:43:38.426910 IP 192.169.4.5.58772 > 192.169.1.9.ssh: Flags [S], seq 2375017601, win 29200, options [mss 1460,sackOK,TS val 2432855330 ecr 0,nop,wscale 7], length 0
17:43:38.428774 IP 192.169.1.9.ssh > 192.169.4.5.58772: Flags [S.], seq 3772914432, ack 2375017602, win 28960, options [mss 1418,sackOK,TS val 2432871334 ecr 2432855330,nop,wscale 7], length 0
17:43:38.428804 IP 192.169.4.5.58772 > 192.169.1.9.ssh: Flags [R], seq 2375017602, win 0, length 

We checked the initialize function, specifically Socket initializing. Cause this is the place where error happens.

 @socket =
        if (factory = options[:proxy])
          debug { "establishing connection to #{@host}:#{@port} through proxy" }
          factory.open(@host, @port, options)
        else
          debug { "establishing connection to #{@host}:#{@port}" }
          Socket.tcp(@host, @port, @bind_address, nil,
                     connect_timeout: options[:timeout])

It passes host, port, bind_address and timeout variables. We checked it.
Host,port and bind_address was right. Timeout was equal to 0. So we decided to change it. And voila. It worked.

So finally we changed "/etc/ssh/ssh_config" parameter ConnectTimeout from 0 to 30.

@mounishrddrdd

This comment has been minimized.

Copy link

commented Apr 12, 2019

did any one found solution for this finally ?

@tas50

This comment has been minimized.

Copy link
Member

commented Jul 8, 2019

@johnreed00 any chance you can try this again in the latest DK or Workstation. We entirely rewrote the knife bootstrap feature in Chef 15 to use our train transport to handle all the connections. It's quite likely this was magically resolved with that.

@tas50 tas50 transferred this issue from chef/chef-dk Jul 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.