GCE Plugin infinite loop

[brauliobo]

How to disable it with knife solo?
chef/chef#1756

/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:54:in `http_client'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:59:in `fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:69:in `block in fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `each'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:69:in `block in fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `each'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:69:in `block in fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `each'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:69:in `block in fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `each'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:69:in `block in fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `each'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:69:in `block in fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `each'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:69:in `block in fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `each'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:69:in `block in fetch_metadata'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `each'
/usr/lib/ruby/gems/1.9.1/gems/ohai-7.2.0/lib/ohai/mixin/gce_metadata.rb:68:in `fetch_metadata'

[jvanbaarsen]

@brauliobo Did you by any change found a work around for this?

👍 For a fix for this

[brauliobo]

@jvanbaarsen Yeah, disable it! Ohai::Config[:disabled_plugins] = [:GCE] on solo.rb or client.rb

[lamont-granquist]

Does this still replicate on ohai-8.0.x and chef-12?

Also, we don't support non-omnibus installs of chef that I can see that you're doing in /usr/lib/ruby and we're dropping support for 1.9.x in Chef-12.1.0 (the gem will no longer install on that ruby).

[jvanbaarsen]

@lamont-granquist I finally installed a vagrant box with chef succesfully yesterday, but im not sure if its because I switched to VMWare provider or an update in chef / ohai. I'll look at that later this week.

[alexdo]

Same problem here. Some of my nodes have a NATed network, all of them are encountering this error.
It seems to come down to the GCE Metatdata Mixin which successfully establishes a socket to a non-resolvable GCE Metadata Endpoint. (as the aforementioned nodes are not hosted at GCE)

I was also able to reproduce this issue via chef's embedded irb:

$ /opt/chef/embedded/bin/irb
irb(main):005:0> require 'ohai/mixin/gce_metadata'
=> true
irb(main):007:0> Ohai::Mixin::GCEMetadata::GCE_METADATA_ADDR
=> "metadata.google.internal"
irb(main):008:0> include Ohai::Mixin::GCEMetadata
=> Object
irb(main):010:0> can_metadata_connect?(Ohai::Mixin::GCEMetadata::GCE_METADATA_ADDR,80)
=> true
irb(main):011:0> require 'socket'
=> false
irb(main):012:0> t = Socket.new(Socket::Constants::AF_INET, Socket::Constants::SOCK_STREAM, 0)
=> #<Socket:fd 7>
irb(main):013:0> saddr = Socket.pack_sockaddr_in(80, Ohai::Mixin::GCEMetadata::GCE_METADATA_ADDR)
=> "\x02\x00\x00P\x90L+\x1C\x00\x00\x00\x00\x00\x00\x00\x00"
irb(main):015:0> t.connect_nonblock(saddr)
Errno::EINPROGRESS: Operation now in progress - connect(2) would block
    from (irb):15:in `connect_nonblock'
    from (irb):15
    from /opt/chef/embedded/bin/irb:12:in `<main>'

Supplying an invalid port returns the result which is expected by the GCE Mixin (-> failing):

irb(main):016:0> saddr = Socket.pack_sockaddr_in(-400, Ohai::Mixin::GCEMetadata::GCE_METADATA_ADDR)
SocketError: getaddrinfo: Servname not supported for ai_socktype
    from (irb):16:in `pack_sockaddr_in'
    from (irb):16
    from /opt/chef/embedded/bin/irb:12:in `<main>'
irb(main):017:0> t.connect_nonblock(saddr)
=> 0

However, I'm quite sure this has something to do with my NATed setup, as a connection on TCP 80 to any URL via Ruby socket succeeds:

irb(main):018:0> t = Socket.new(Socket::Constants::AF_INET, Socket::Constants::SOCK_STREAM, 0)
=> #<Socket:fd 8>
irb(main):019:0> saddr = Socket.pack_sockaddr_in(80, 'some-crazy-invalid-domain-name')
=> "\x02\x00\x00P\x90L+\x1C\x00\x00\x00\x00\x00\x00\x00\x00"
irb(main):021:0> t.connect_nonblock(saddr)
Errno::EINPROGRESS: Operation now in progress - connect(2) would block
    from (irb):21:in `connect_nonblock'
    from (irb):21
    from /opt/chef/embedded/bin/irb:12:in `<main>'
irb(main):022:0> 

---

EDIT
After some further investigation, I discovered that my ISPs DNS redirects unknown domain names to some generic page. As a result, the socket to metadata.google.internal is established (obviously to the wrong target), but the metadata API endpoint won't yield any results.

$ nslookup metadata.google.internal
Server:     192.168.173.1
Address:    192.168.173.1#53

Non-authoritative answer:
Name:   metadata.google.internal
Address: my.providers.generic.page

$ curl metadata.google.internal
=> generic page html

[bgehman]

We hit the same problem, and it is easy to reproduce once you understand the root-cause. It all boils down to if your VM is in the "smoketest.com" domain, then the GCE plugin is able to resolve the metadata.google.internal hostname, as in:

nslookup metadata.google.internal.smoketest.com

Server: 10.0.80.11
Address: 10.0.80.11#53

Non-authoritative answer:
Name: metadata.google.internal.smoketest.com
Address: 69.172.201.208

This tricks the Ohai/GCE plugin into thinking that the VM is running in the GCE cloud, and hangs chef-server / chef-client startup (since 69.172.201.208 is not a real GCE metadata server).

[thommay]

#736 should make this better

[tas50]

This is resolved with #736 via a short timeout period.