New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Never-ending getaddress() call when using compressed IPv6 nameservers in /etc/resolv.conf #3663

Closed
nbarrientos opened this Issue Feb 13, 2016 · 21 comments

Comments

Projects
None yet
3 participants
@nbarrientos
Contributor

nbarrientos commented Feb 13, 2016

Hi,

I managed to reproduce the following bug in a CentOS 7.2 box with only IPv6 nameservers and dual stack, jRuby 1.7.20.1 and java-1.7.0-openjdk-1.7.0.95-2.6.4.0.el7_2. The same code works fine on MRI 2.0.0. Perhaps this has already been fixed in newer versions, as to me the bug looks too obvious to be still alive. Anyway, here we go:

# cat /etc/resolv.conf
# generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1000::5
nameserver 2001:1458:201:1100::5

The following program and the resolv.conf above makes the interpreter hang for a long time and return a failure:

require 'resolv'
puts Resolv.getaddress 'web.cern.ch'
# time java -cp /usr/share/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.puppetserver.cli.ruby --config /etc/puppetserver/conf.d -- -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
Resolv::ResolvError: no address for web.cern.ch
  getaddress at /usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/resolv.rb:98
  getaddress at /usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/resolv.rb:48
      (root) at -e:1
      invoke at jruby_puppet_core.clj:232
      invoke at jruby_puppet_core.clj:226
      invoke at subcommand.clj:38
    doInvoke at ruby.clj:7
      invoke at core.clj:624
      invoke at main.clj:315
    doInvoke at main.clj:420

real    2m51.910s
user    0m21.020s
sys 0m0.654s

These are the last lines in a debugging session before the first exception is raised. The program blocks in the select() call until it times out and ResolvTimeout is raised.

#0:/usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/resolv.rb:676:Resolv::DNS::Requester:-:           if s = @senders[[from,msg.id]]
#0:/usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/resolv.rb:654:Resolv::DNS::Requester:-:           now = Time.now
#0:/usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/resolv.rb:655:Resolv::DNS::Requester:-:           timeout = timelimit - now
#0:/usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/resolv.rb:656:Resolv::DNS::Requester:-:           if timeout <= 0
#0:/usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/resolv.rb:659:Resolv::DNS::Requester:-:           select_result = IO.select(@socks, nil, nil, timeout)
#0:/usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/resolv.rb:660:Resolv::DNS::Requester:-:           if !select_result
#0:/usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/resolv.rb:661:Resolv::DNS::Requester:-:             raise ResolvTimeout
#0:/usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/resolv.rb:661:Resolv::DNS::Requester:^:             raise ResolvTimeout
/usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/resolv.rb:661: `Resolv::ResolvTimeout' (Resolv::ResolvTimeout)

I think that this is happening because during the previous iteration this statement was evaluated as false because

@senders[[from,msg.id]]

was nil. Why was it nil? Because the search keys didn't match, as 'from' contains the uncompressed flavor of the IPv6 address of the DNS server, whereas 'senders' has the compacted form (presumably coming from /etc/resolv.conf):

(rdb:1) p from
["2001:1458:201:1000:0:0:0:5", 53]
(rdb:1) p msg.id
12941
(rdb:1) p @senders
{[["2001:1458:201:1000::5", 53], 12941]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x4605f6fa @host="2001:1458:201:1000::5", @msg="2\x8D\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 28>, @port=53>}

This situation leads to the outer loop not stopping (see "unexpected DNS message ignored"), therefore the program executes IO.select again but there's nothing to read anymore so ResolvTimeout is raised. This exception is probably caught by the caller further up and at some point .request is called again, leading to another loop. This situation repeats many times after a few minutes the call returns a ResolvError all the way up back to the user.

Handcrafting resolv.conf so all addreses are expanded there makes resolv.rb happy:

# generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1000:0:0:0:5
nameserver 2001:1458:201:1100:0:0:0:5

This way the program quickly exits without hanging:

# time java -cp /usr/share/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.puppetserver.cli.ruby --config /etc/puppetserver/conf.d -- -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
188.184.9.235

real    0m10.945s
user    0m19.857s
sys 0m0.553s

Configuring a Resolv::DNS object by hand with a compressed IPv6 address seems to work.

# cat /etc/resolv.conf
# generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1000::5
nameserver 2001:1458:201:1100::5
....
irb(main):008:0* require 'resolv'
=> true
irb(main):009:0> Resolv::DNS.new(:nameserver => '2001:1458:201:1000::5').getaddress "web.cern.ch"
=> #<Resolv::IPv4 188.184.9.235>
irb(main):010:0> Resolv::DNS.new().getaddress "web.cern.ch"
# Hangs...
@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Feb 13, 2016

Contributor

I can reproduce the problem using the latest jRuby 1.x:

# cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1000::5
nameserver 2001:1458:201:1100::5
# bin/jruby -v
jruby 1.7.24 (1.9.3p551) 2016-01-20 bd68d85 on OpenJDK 64-Bit Server VM 1.7.0_95-mockbuild_2016_01_21_15_30-b00 +jit [linux-amd64]
# bin/jruby -r debug -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
Debug.rb
Emacs support available.

/root/jruby-1.7.24/lib/ruby/1.9/debug.rb:910 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag
/root/jruby-1.7.24/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:39:
(rdb:1) tr on
Trace on.
(rdb:1) c
...
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676:Resolv::DNS::Requester:-:           if s = @senders[[from,msg.id]]
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:654:Resolv::DNS::Requester:-:           now = Time.now
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:655:Resolv::DNS::Requester:-:           timeout = timelimit - now
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:656:Resolv::DNS::Requester:-:           if timeout <= 0
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:659:Resolv::DNS::Requester:-:           select_result = IO.select(@socks, nil, nil, timeout)
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:660:Resolv::DNS::Requester:-:           if !select_result
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:Resolv::DNS::Requester:-:             raise ResolvTimeout
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:Resolv::DNS::Requester:^:             raise ResolvTimeout
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661: `Resolv::ResolvTimeout' (Resolv::ResolvTimeout)
    from -e:1:in `(root)'
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:            raise ResolvTimeout
(rdb:1) p from
["2001:1458:201:1000:0:0:0:5", 53]
(rdb:1) p msg.id
35327
(rdb:1) p @senders
{[["2001:1458:201:1000::5", 53], 35327]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x1386c56d @host="2001:1458:201:1000::5", @msg="\x89\xFF\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 20>, @port=53>}
(rdb:1)

On the other hand:

# cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1000:0:0:0:5
nameserver 2001:1458:201:1100:0:0:0:5
# bin/jruby -r debug -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
Debug.rb
Emacs support available.

/root/jruby-1.7.24/lib/ruby/1.9/debug.rb:910 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag
/root/jruby-1.7.24/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:39:
(rdb:1) tr on
Trace on.
(rdb:1) c
...
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:626:#<Class:Resolv::DNS>:-:         if h = RequestID[key]
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:627:#<Class:Resolv::DNS>:-:           h.delete id
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:628:#<Class:Resolv::DNS>:-:           if h.empty?
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:629:#<Class:Resolv::DNS>:-:             RequestID.delete key
188.184.9.235
#

The program exits normally.

Same as above but stopping to inspect:

(rdb:1) b /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
Set breakpoint 1 at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
(rdb:1) c
Breakpoint 1, request at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676:          if s = @senders[[from,msg.id]]
(rdb:1) p from
["2001:1458:201:1000:0:0:0:5", 53]
(rdb:1) p msg.id
25133
(rdb:1) p @senders
{[["2001:1458:201:1000:0:0:0:5", 53], 25133]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x4dbc377 @host="2001:1458:201:1000:0:0:0:5", @msg="b-\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 15>, @port=53>}
(rdb:1) c
188.184.9.235
#
Contributor

nbarrientos commented Feb 13, 2016

I can reproduce the problem using the latest jRuby 1.x:

# cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1000::5
nameserver 2001:1458:201:1100::5
# bin/jruby -v
jruby 1.7.24 (1.9.3p551) 2016-01-20 bd68d85 on OpenJDK 64-Bit Server VM 1.7.0_95-mockbuild_2016_01_21_15_30-b00 +jit [linux-amd64]
# bin/jruby -r debug -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
Debug.rb
Emacs support available.

/root/jruby-1.7.24/lib/ruby/1.9/debug.rb:910 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag
/root/jruby-1.7.24/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:39:
(rdb:1) tr on
Trace on.
(rdb:1) c
...
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676:Resolv::DNS::Requester:-:           if s = @senders[[from,msg.id]]
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:654:Resolv::DNS::Requester:-:           now = Time.now
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:655:Resolv::DNS::Requester:-:           timeout = timelimit - now
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:656:Resolv::DNS::Requester:-:           if timeout <= 0
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:659:Resolv::DNS::Requester:-:           select_result = IO.select(@socks, nil, nil, timeout)
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:660:Resolv::DNS::Requester:-:           if !select_result
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:Resolv::DNS::Requester:-:             raise ResolvTimeout
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:Resolv::DNS::Requester:^:             raise ResolvTimeout
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661: `Resolv::ResolvTimeout' (Resolv::ResolvTimeout)
    from -e:1:in `(root)'
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:            raise ResolvTimeout
(rdb:1) p from
["2001:1458:201:1000:0:0:0:5", 53]
(rdb:1) p msg.id
35327
(rdb:1) p @senders
{[["2001:1458:201:1000::5", 53], 35327]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x1386c56d @host="2001:1458:201:1000::5", @msg="\x89\xFF\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 20>, @port=53>}
(rdb:1)

On the other hand:

# cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1000:0:0:0:5
nameserver 2001:1458:201:1100:0:0:0:5
# bin/jruby -r debug -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
Debug.rb
Emacs support available.

/root/jruby-1.7.24/lib/ruby/1.9/debug.rb:910 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag
/root/jruby-1.7.24/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:39:
(rdb:1) tr on
Trace on.
(rdb:1) c
...
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:626:#<Class:Resolv::DNS>:-:         if h = RequestID[key]
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:627:#<Class:Resolv::DNS>:-:           h.delete id
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:628:#<Class:Resolv::DNS>:-:           if h.empty?
#0:/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:629:#<Class:Resolv::DNS>:-:             RequestID.delete key
188.184.9.235
#

The program exits normally.

Same as above but stopping to inspect:

(rdb:1) b /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
Set breakpoint 1 at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
(rdb:1) c
Breakpoint 1, request at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676:          if s = @senders[[from,msg.id]]
(rdb:1) p from
["2001:1458:201:1000:0:0:0:5", 53]
(rdb:1) p msg.id
25133
(rdb:1) p @senders
{[["2001:1458:201:1000:0:0:0:5", 53], 25133]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x4dbc377 @host="2001:1458:201:1000:0:0:0:5", @msg="b-\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 15>, @port=53>}
(rdb:1) c
188.184.9.235
#
@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Feb 13, 2016

Contributor

I see the same behaviour when running 9.0.5.0.

Contributor

nbarrientos commented Feb 13, 2016

I see the same behaviour when running 9.0.5.0.

@nbarrientos nbarrientos changed the title from Never ending getaddress() call when using compressed IPv6 nameservers in /etc/resolv.conf to Never-ending getaddress() call when using compressed IPv6 nameservers in /etc/resolv.conf Feb 13, 2016

@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Feb 13, 2016

Contributor

And the same debugging process but using MRI (which works with compacted addresses):

# cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1000::5
nameserver 2001:1458:201:1100::5
# ruby -v
ruby 2.0.0p598 (2014-11-13) [x86_64-linux]
# ruby -r debug -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
Debug.rb
Emacs support available.

/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:57:
(rdb:1) b /usr/share/ruby/resolv.rb:691
Set breakpoint 1 at /usr/share/ruby/resolv.rb:691
(rdb:1) c
Breakpoint 1, request at /usr/share/ruby/resolv.rb:691
/usr/share/ruby/resolv.rb:691:          if s = @senders[[from,msg.id]]
(rdb:1) p from
["2001:1458:201:1000::5", 53]
(rdb:1) p msg.id
50581
(rdb:1) p @senders
{[["2001:1458:201:1000::5", 53], 50581]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x00000002c996e8 @msg="\xC5\x95\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 7>, @host="2001:1458:201:1000::5", @port=53>}
(rdb:1) c
188.184.9.235
#
Contributor

nbarrientos commented Feb 13, 2016

And the same debugging process but using MRI (which works with compacted addresses):

# cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1000::5
nameserver 2001:1458:201:1100::5
# ruby -v
ruby 2.0.0p598 (2014-11-13) [x86_64-linux]
# ruby -r debug -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
Debug.rb
Emacs support available.

/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:57:
(rdb:1) b /usr/share/ruby/resolv.rb:691
Set breakpoint 1 at /usr/share/ruby/resolv.rb:691
(rdb:1) c
Breakpoint 1, request at /usr/share/ruby/resolv.rb:691
/usr/share/ruby/resolv.rb:691:          if s = @senders[[from,msg.id]]
(rdb:1) p from
["2001:1458:201:1000::5", 53]
(rdb:1) p msg.id
50581
(rdb:1) p @senders
{[["2001:1458:201:1000::5", 53], 50581]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x00000002c996e8 @msg="\xC5\x95\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 7>, @host="2001:1458:201:1000::5", @port=53>}
(rdb:1) c
188.184.9.235
#
@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Feb 14, 2016

Contributor

Hi again,

(It's getting quite amusing to speak to myself :))

It's also interesting to observe that the problem is only triggered when more than one (compressed) IPv6 nameservers are available and therefore UnconnectedUDP is used. I realized about this after trying to reproduce the problem by creating Resolv::DNS objects with manual configuration aiming to take resolv.conf out of the equation.

So first test case, two IPv6 nameservers with compressed addresses. As shown in previous comments this should trigger the bug and therefore fail.

# bin/jruby -r debug -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:1458:201:1000::5', '2001:1458:201:1100::5']).getaddress 'web.cern.ch'"
(rdb:1) c
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661: `Resolv::ResolvTimeout' (Resolv::ResolvTimeout)
        from -e:1:in `(root)'
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:            raise ResolvTimeout
(rdb:1) p from
["2001:1458:201:1000:0:0:0:5", 53]
(rdb:1) p @senders
{[["2001:1458:201:1000::5", 53], 22726]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x3dcf9961 @host="2001:1458:201:1000::5", @msg="X\xC6\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 13>, @port=53>}
(rdb:1) c
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661: `Resolv::ResolvTimeout' (Resolv::ResolvTimeout)
        from -e:1:in `(root)'
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:            raise ResolvTimeout
(rdb:1) c
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661: `Resolv::ResolvTimeout' (Resolv::ResolvTimeout)
        from -e:1:in `(root)'
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:            raise ResolvTimeout
.... and on and on ....
(rdb:1) q
Really quit? (y/n) y
# FAIL

Now only one compressed nameserver, note that Resolv::DNS::Requester::ConnectedUDP is used instead:

# bin/jruby -r debug -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => '2001:1458:201:1000::5').getaddress 'web.cern.ch'"Debug.rb
Emacs support available.

/root/jruby-1.7.24/lib/ruby/1.9/debug.rb:910 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag
/root/jruby-1.7.24/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:39:
(rdb:1) b /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
Set breakpoint 1 at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
(rdb:1) c
Breakpoint 1, request at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676:          if s = @senders[[from,msg.id]]
(rdb:1) p @senders
{[nil, 39392]=>#<Resolv::DNS::Requester::ConnectedUDP::Sender:0x36a70903 @msg="\x99\xE0\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @sock=#<UDPSocket:fd 13>, @data=#<Resolv::DNS::Name: web.cern.ch.>>}
(rdb:1) p from
nil
(rdb:1) c
188.184.9.235
# SUCCESS

It works!

Finally, the last test case for completeness' sake: more than one IPv6 nameserver without compressed addresses. This should also work as seen before (when I expanded the addresses in resolv.conf by hand).

# bin/jruby -r debug -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:1458:201:1000:0:0:0:5', '2001:1458:201:1100:0:0:0:5']).getaddress 'web.cern.ch'"
Debug.rb
Emacs support available.

/root/jruby-1.7.24/lib/ruby/1.9/debug.rb:910 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag
/root/jruby-1.7.24/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:39:
(rdb:1)  b /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
Set breakpoint 1 at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
(rdb:1) c
Breakpoint 1, request at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676:          if s = @senders[[from,msg.id]]
(rdb:1) p from
["2001:1458:201:1000:0:0:0:5", 53]
(rdb:1) p @senders
{[["2001:1458:201:1000:0:0:0:5", 53], 54764]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x238048a @host="2001:1458:201:1000:0:0:0:5", @msg="\xD5\xEC\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 13>, @port=53>}
(rdb:1) c
188.184.9.235
# SUCCESS
Contributor

nbarrientos commented Feb 14, 2016

Hi again,

(It's getting quite amusing to speak to myself :))

It's also interesting to observe that the problem is only triggered when more than one (compressed) IPv6 nameservers are available and therefore UnconnectedUDP is used. I realized about this after trying to reproduce the problem by creating Resolv::DNS objects with manual configuration aiming to take resolv.conf out of the equation.

So first test case, two IPv6 nameservers with compressed addresses. As shown in previous comments this should trigger the bug and therefore fail.

# bin/jruby -r debug -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:1458:201:1000::5', '2001:1458:201:1100::5']).getaddress 'web.cern.ch'"
(rdb:1) c
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661: `Resolv::ResolvTimeout' (Resolv::ResolvTimeout)
        from -e:1:in `(root)'
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:            raise ResolvTimeout
(rdb:1) p from
["2001:1458:201:1000:0:0:0:5", 53]
(rdb:1) p @senders
{[["2001:1458:201:1000::5", 53], 22726]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x3dcf9961 @host="2001:1458:201:1000::5", @msg="X\xC6\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 13>, @port=53>}
(rdb:1) c
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661: `Resolv::ResolvTimeout' (Resolv::ResolvTimeout)
        from -e:1:in `(root)'
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:            raise ResolvTimeout
(rdb:1) c
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661: `Resolv::ResolvTimeout' (Resolv::ResolvTimeout)
        from -e:1:in `(root)'
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:661:            raise ResolvTimeout
.... and on and on ....
(rdb:1) q
Really quit? (y/n) y
# FAIL

Now only one compressed nameserver, note that Resolv::DNS::Requester::ConnectedUDP is used instead:

# bin/jruby -r debug -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => '2001:1458:201:1000::5').getaddress 'web.cern.ch'"Debug.rb
Emacs support available.

/root/jruby-1.7.24/lib/ruby/1.9/debug.rb:910 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag
/root/jruby-1.7.24/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:39:
(rdb:1) b /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
Set breakpoint 1 at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
(rdb:1) c
Breakpoint 1, request at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676:          if s = @senders[[from,msg.id]]
(rdb:1) p @senders
{[nil, 39392]=>#<Resolv::DNS::Requester::ConnectedUDP::Sender:0x36a70903 @msg="\x99\xE0\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @sock=#<UDPSocket:fd 13>, @data=#<Resolv::DNS::Name: web.cern.ch.>>}
(rdb:1) p from
nil
(rdb:1) c
188.184.9.235
# SUCCESS

It works!

Finally, the last test case for completeness' sake: more than one IPv6 nameserver without compressed addresses. This should also work as seen before (when I expanded the addresses in resolv.conf by hand).

# bin/jruby -r debug -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:1458:201:1000:0:0:0:5', '2001:1458:201:1100:0:0:0:5']).getaddress 'web.cern.ch'"
Debug.rb
Emacs support available.

/root/jruby-1.7.24/lib/ruby/1.9/debug.rb:910 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag
/root/jruby-1.7.24/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:39:
(rdb:1)  b /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
Set breakpoint 1 at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
(rdb:1) c
Breakpoint 1, request at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:676:          if s = @senders[[from,msg.id]]
(rdb:1) p from
["2001:1458:201:1000:0:0:0:5", 53]
(rdb:1) p @senders
{[["2001:1458:201:1000:0:0:0:5", 53], 54764]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x238048a @host="2001:1458:201:1000:0:0:0:5", @msg="\xD5\xEC\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 13>, @port=53>}
(rdb:1) c
188.184.9.235
# SUCCESS
@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Feb 14, 2016

Contributor

This could be a candidate for a patch, so the responsibility to compare addresses is delegated to IPAddr (instead of merely comparing strings):

--- lib/ruby/1.9/resolv.rb.orig 2016-02-14 12:16:06.916368212 +0100
+++ lib/ruby/1.9/resolv.rb      2016-02-14 12:30:17.925784003 +0100
@@ -2,6 +2,7 @@
 require 'fcntl'
 require 'timeout'
 require 'thread'
+require 'ipaddr'

 begin
   require 'securerandom'
@@ -728,13 +729,13 @@

         def recv_reply(readable_socks)
           reply, from = readable_socks[0].recvfrom(UDPSize)
-          return reply, [from[3],from[1]]
+          return reply, [IPAddr.new(from[3]),from[1]]
         end

         def sender(msg, data, host, port=Port)
           sock = @socks_hash[host.index(':') ? "::" : "0.0.0.0"]
           return nil if !sock
-          service = [host, port]
+          service = [IPAddr.new(host), port]
           id = DNS.allocate_request_id(host, port)
           request = msg.encode
           request[0,2] = [id].pack('n')

This way it does not matter if the address returned by recvfrom is compressed or expanded, as the one it has to be compared to is also an IPAddr object.

# bin/jruby -r debug -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:1458:201:1000::5', '2001:1458:201:1100::5']).getaddress 'web.cern.ch'"
Debug.rb
Emacs support available.
(rdb:1) b /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:677
Set breakpoint 1 at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:677
(rdb:1) c
Breakpoint 1, request at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:677
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:677:          if s = @senders[[from,msg.id]]
(rdb:1) l
[672, 681] in /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb
   672            begin
   673              msg = Message.decode(reply)
   674            rescue DecodeError
   675              next # broken DNS message ignored
   676            end
=> 677            if s = @senders[[from,msg.id]]
   678              break
   679            else
   680              # unexpected DNS message ignored
   681            end
(rdb:1) p from
[#<IPAddr: IPv6:2001:1458:0201:1000:0000:0000:0000:0005/ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff>, 53]
(rdb:1) p @senders
{[[#<IPAddr: IPv6:2001:1458:0201:1000:0000:0000:0000:0005/ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff>, 53], 30628]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x68c0dad5 @host="2001:1458:201:1000::5", @msg="w\xA4\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 13>, @port=53>}
(rdb:1) c
188.184.9.235
#
Contributor

nbarrientos commented Feb 14, 2016

This could be a candidate for a patch, so the responsibility to compare addresses is delegated to IPAddr (instead of merely comparing strings):

--- lib/ruby/1.9/resolv.rb.orig 2016-02-14 12:16:06.916368212 +0100
+++ lib/ruby/1.9/resolv.rb      2016-02-14 12:30:17.925784003 +0100
@@ -2,6 +2,7 @@
 require 'fcntl'
 require 'timeout'
 require 'thread'
+require 'ipaddr'

 begin
   require 'securerandom'
@@ -728,13 +729,13 @@

         def recv_reply(readable_socks)
           reply, from = readable_socks[0].recvfrom(UDPSize)
-          return reply, [from[3],from[1]]
+          return reply, [IPAddr.new(from[3]),from[1]]
         end

         def sender(msg, data, host, port=Port)
           sock = @socks_hash[host.index(':') ? "::" : "0.0.0.0"]
           return nil if !sock
-          service = [host, port]
+          service = [IPAddr.new(host), port]
           id = DNS.allocate_request_id(host, port)
           request = msg.encode
           request[0,2] = [id].pack('n')

This way it does not matter if the address returned by recvfrom is compressed or expanded, as the one it has to be compared to is also an IPAddr object.

# bin/jruby -r debug -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:1458:201:1000::5', '2001:1458:201:1100::5']).getaddress 'web.cern.ch'"
Debug.rb
Emacs support available.
(rdb:1) b /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:677
Set breakpoint 1 at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:677
(rdb:1) c
Breakpoint 1, request at /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:677
/root/jruby-1.7.24/lib/ruby/1.9/resolv.rb:677:          if s = @senders[[from,msg.id]]
(rdb:1) l
[672, 681] in /root/jruby-1.7.24/lib/ruby/1.9/resolv.rb
   672            begin
   673              msg = Message.decode(reply)
   674            rescue DecodeError
   675              next # broken DNS message ignored
   676            end
=> 677            if s = @senders[[from,msg.id]]
   678              break
   679            else
   680              # unexpected DNS message ignored
   681            end
(rdb:1) p from
[#<IPAddr: IPv6:2001:1458:0201:1000:0000:0000:0000:0005/ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff>, 53]
(rdb:1) p @senders
{[[#<IPAddr: IPv6:2001:1458:0201:1000:0000:0000:0000:0005/ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff>, 53], 30628]=>#<Resolv::DNS::Requester::UnconnectedUDP::Sender:0x68c0dad5 @host="2001:1458:201:1000::5", @msg="w\xA4\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03web\x04cern\x02ch\x00\x00\x01\x00\x01", @data=#<Resolv::DNS::Name: web.cern.ch.>, @sock=#<UDPSocket:fd 13>, @port=53>}
(rdb:1) c
188.184.9.235
#
@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Feb 14, 2016

Contributor

BTW, the IPv6 nameservers used above for all the examples are not accesible from the Internet so expect a ResolvTimeout to be raised anyway, masking the bug. It should be easy to reproduce the problem though using Google's public nameservers (2001:4860:4860::8888 and 2001:4860:4860::8844).

Contributor

nbarrientos commented Feb 14, 2016

BTW, the IPv6 nameservers used above for all the examples are not accesible from the Internet so expect a ResolvTimeout to be raised anyway, masking the bug. It should be easy to reproduce the problem though using Google's public nameservers (2001:4860:4860::8888 and 2001:4860:4860::8844).

@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Feb 14, 2016

Contributor

Now using Google's nameservers. Slightly different test environment though (OSX, Java8) but same results:

$ host web.cern.ch 2001:4860:4860::8888
Using domain server:
Name: 2001:4860:4860::8888
Address: 2001:4860:4860::8888#53
Aliases:

web.cern.ch has address 188.184.9.235

Unpatched:

$ bin/jruby -v
jruby 1.7.24 (1.9.3p551) 2016-01-20 bd68d85 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_74-b02 +jit [darwin-x86_64]
$ time bin/jruby -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:4860:4860::8888', '2001:4860:4860::8844']).getaddress 'web.cern.ch'"
Resolv::ResolvError: DNS result has no information for web.cern.ch
  getaddress at /Users/nacho/Downloads/jruby-1.7.24/lib/ruby/1.9/resolv.rb:373
      (root) at -e:1

real    2m41.820s
user    0m5.326s
sys 0m0.372s

Patched:

$ bin/jruby -v
jruby 1.7.24 (1.9.3p551) 2016-01-20 bd68d85 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_74-b02 +jit [darwin-x86_64]
$ time bin/jruby -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:4860:4860::8888', '2001:4860:4860::8844']).getaddress 'web.cern.ch'"
188.184.9.235

real    0m2.007s
user    0m4.815s
sys 0m0.215s

And unpatched again but with uncompressed addresses:

$ time bin/jruby -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:4860:4860:0:0:0:0:8888', '2001:4860:4860:0:0:0:0:8844']).getaddress 'web.cern.ch'"
188.184.9.235

real    0m1.644s
user    0m4.680s
sys 0m0.209s
Contributor

nbarrientos commented Feb 14, 2016

Now using Google's nameservers. Slightly different test environment though (OSX, Java8) but same results:

$ host web.cern.ch 2001:4860:4860::8888
Using domain server:
Name: 2001:4860:4860::8888
Address: 2001:4860:4860::8888#53
Aliases:

web.cern.ch has address 188.184.9.235

Unpatched:

$ bin/jruby -v
jruby 1.7.24 (1.9.3p551) 2016-01-20 bd68d85 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_74-b02 +jit [darwin-x86_64]
$ time bin/jruby -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:4860:4860::8888', '2001:4860:4860::8844']).getaddress 'web.cern.ch'"
Resolv::ResolvError: DNS result has no information for web.cern.ch
  getaddress at /Users/nacho/Downloads/jruby-1.7.24/lib/ruby/1.9/resolv.rb:373
      (root) at -e:1

real    2m41.820s
user    0m5.326s
sys 0m0.372s

Patched:

$ bin/jruby -v
jruby 1.7.24 (1.9.3p551) 2016-01-20 bd68d85 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_74-b02 +jit [darwin-x86_64]
$ time bin/jruby -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:4860:4860::8888', '2001:4860:4860::8844']).getaddress 'web.cern.ch'"
188.184.9.235

real    0m2.007s
user    0m4.815s
sys 0m0.215s

And unpatched again but with uncompressed addresses:

$ time bin/jruby -e "require 'resolv'; puts Resolv::DNS.new(:nameserver => ['2001:4860:4860:0:0:0:0:8888', '2001:4860:4860:0:0:0:0:8844']).getaddress 'web.cern.ch'"
188.184.9.235

real    0m1.644s
user    0m4.680s
sys 0m0.209s
@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Mar 1, 2016

Contributor

Hello?

Contributor

nbarrientos commented Mar 1, 2016

Hello?

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Mar 4, 2016

Member

Wow, you put a lot of research into this. I've got it in my queue to look at now.

Member

headius commented Mar 4, 2016

Wow, you put a lot of research into this. I've got it in my queue to look at now.

@headius headius added this to the JRuby 9.1.1.0 milestone Apr 20, 2016

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Apr 20, 2016

Member

This didn't make 9.1 but I'll try to make it a priority for 9.1.1.

Member

headius commented Apr 20, 2016

This didn't make 9.1 but I'll try to make it a priority for 9.1.1.

@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Apr 20, 2016

Contributor

Thanks.

Contributor

nbarrientos commented Apr 20, 2016

Thanks.

@headius headius modified the milestones: JRuby 9.1.1.0, JRuby 9.1.2.0 May 11, 2016

@enebo enebo modified the milestones: JRuby 9.1.2.0, JRuby 9.1.3.0 May 23, 2016

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Aug 15, 2016

Member

Ok, this got backburnered for a while, but I'm not quite sure what to do at this point. I need to know...

  1. Does this work on JRuby 9k? It should have the same resolv.rb as MRI and you said MRI works.
  2. Is your patch still appropriate for JRuby 1.7? I believe MRI no longer maintains 1.8.7 or 1.9.3 stdlib, so we can patch that one unilaterally. If we need to patch JRuby 9k, we'll want to work with MRI to upstream that patch.

Sorry this got left for so long!

Member

headius commented Aug 15, 2016

Ok, this got backburnered for a while, but I'm not quite sure what to do at this point. I need to know...

  1. Does this work on JRuby 9k? It should have the same resolv.rb as MRI and you said MRI works.
  2. Is your patch still appropriate for JRuby 1.7? I believe MRI no longer maintains 1.8.7 or 1.9.3 stdlib, so we can patch that one unilaterally. If we need to patch JRuby 9k, we'll want to work with MRI to upstream that patch.

Sorry this got left for so long!

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Nov 8, 2016

Member

No response since August 15...closing for now, but if it's still an issue and you can reproduce it on 9k, we need to get a reproduction script/repo from you.

Member

headius commented Nov 8, 2016

No response since August 15...closing for now, but if it's still an issue and you can reproduce it on 9k, we need to get a reproduction script/repo from you.

@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Nov 8, 2016

Contributor

Hi,

Sorry, I missed your update :(

Does this work on JRuby 9k? It should have the same resolv.rb as MRI and you said MRI works.

It does not, as stated on the third comment.

Is your patch still appropriate for JRuby 1.7? I believe MRI no longer maintains 1.8.7 or 1.9.3 stdlib, so we can patch that one unilaterally. If we need to patch JRuby 9k, we'll want to work with MRI to upstream that patch.

It is I believe, however I selfishly only care about 1.7 which is what we have to run as part of Puppetserer 1.x :)

we need to get a reproduction script/repo from you.

The way to reproduce the bug is explained above in detail, you need:

  • jRuby 1.7 or 9k.
  • A box with either IPv6 only or dual stack.
  • Two (or more) IPv6 nameservers in /etc/resolv.conf expressed in the compact form.

Demos (using the latest 9k that I could find):

MRI, compacted IPv6 nameservers: it works.

# cat /etc/resolv.conf 
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1100::5
nameserver 2001:1458:201:1000::5
# ruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
188.184.9.235
# ruby -v
ruby 2.0.0p598 (2014-11-13) [x86_64-linux]

jRuby 9.1.5.0, compacted nameservers, it hangs:

# cat /etc/resolv.conf 
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1100::5
nameserver 2001:1458:201:1000::5
# PATH=$PATH:jruby-9.1.5.0/bin/
# jruby -v
jruby 9.1.5.0 (2.3.1) 2016-09-07 036ce39 OpenJDK 64-Bit Server VM 24.111-b01 on 1.7.0_111-mockbuild_2016_07_27_10_50-b00 +jit [linux-x86_64]
# jruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
.... hangs forever, had to CTRL+C it ...

jRuby 9.1.5.0, expanded nameservers, it works:

# cat /etc/resolv.conf 
; generated by /usr/sbin/dhclient-script
search cern.ch.
;nameserver 2001:1458:201:1100::5
;nameserver 2001:1458:201:1000::5
nameserver 2001:1458:201:1000:0:0:0:5
nameserver 2001:1458:201:1100:0:0:0:5
# jruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
188.184.9.235

And for completeness, jRuby 1.7.26 and compacted nameservers, it hangs:

# cat /etc/resolv.conf 
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1100::5
nameserver 2001:1458:201:1000::5
# PATH=$PATH:jruby-1.7.26/bin/
# jruby -v
jruby 1.7.26 (1.9.3p551) 2016-08-26 69763b8 on OpenJDK 64-Bit Server VM 1.7.0_111-mockbuild_2016_07_27_10_50-b00 +jit [linux-amd64]
# jruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
... hangs...

Those nameservers cannot be used from the Internet, however this should be reproducible using Google's public IPv6 nameservers: 2001:4860:4860::8888 and 2001:4860:4860::8844.

Hope this helps.

Contributor

nbarrientos commented Nov 8, 2016

Hi,

Sorry, I missed your update :(

Does this work on JRuby 9k? It should have the same resolv.rb as MRI and you said MRI works.

It does not, as stated on the third comment.

Is your patch still appropriate for JRuby 1.7? I believe MRI no longer maintains 1.8.7 or 1.9.3 stdlib, so we can patch that one unilaterally. If we need to patch JRuby 9k, we'll want to work with MRI to upstream that patch.

It is I believe, however I selfishly only care about 1.7 which is what we have to run as part of Puppetserer 1.x :)

we need to get a reproduction script/repo from you.

The way to reproduce the bug is explained above in detail, you need:

  • jRuby 1.7 or 9k.
  • A box with either IPv6 only or dual stack.
  • Two (or more) IPv6 nameservers in /etc/resolv.conf expressed in the compact form.

Demos (using the latest 9k that I could find):

MRI, compacted IPv6 nameservers: it works.

# cat /etc/resolv.conf 
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1100::5
nameserver 2001:1458:201:1000::5
# ruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
188.184.9.235
# ruby -v
ruby 2.0.0p598 (2014-11-13) [x86_64-linux]

jRuby 9.1.5.0, compacted nameservers, it hangs:

# cat /etc/resolv.conf 
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1100::5
nameserver 2001:1458:201:1000::5
# PATH=$PATH:jruby-9.1.5.0/bin/
# jruby -v
jruby 9.1.5.0 (2.3.1) 2016-09-07 036ce39 OpenJDK 64-Bit Server VM 24.111-b01 on 1.7.0_111-mockbuild_2016_07_27_10_50-b00 +jit [linux-x86_64]
# jruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
.... hangs forever, had to CTRL+C it ...

jRuby 9.1.5.0, expanded nameservers, it works:

# cat /etc/resolv.conf 
; generated by /usr/sbin/dhclient-script
search cern.ch.
;nameserver 2001:1458:201:1100::5
;nameserver 2001:1458:201:1000::5
nameserver 2001:1458:201:1000:0:0:0:5
nameserver 2001:1458:201:1100:0:0:0:5
# jruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
188.184.9.235

And for completeness, jRuby 1.7.26 and compacted nameservers, it hangs:

# cat /etc/resolv.conf 
; generated by /usr/sbin/dhclient-script
search cern.ch.
nameserver 2001:1458:201:1100::5
nameserver 2001:1458:201:1000::5
# PATH=$PATH:jruby-1.7.26/bin/
# jruby -v
jruby 1.7.26 (1.9.3p551) 2016-08-26 69763b8 on OpenJDK 64-Bit Server VM 1.7.0_111-mockbuild_2016_07_27_10_50-b00 +jit [linux-amd64]
# jruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
... hangs...

Those nameservers cannot be used from the Internet, however this should be reproducible using Google's public IPv6 nameservers: 2001:4860:4860::8888 and 2001:4860:4860::8844.

Hope this helps.

@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Nov 29, 2016

Contributor

Could at least the issue be reopened? Thanks.

Contributor

nbarrientos commented Nov 29, 2016

Could at least the issue be reopened? Thanks.

@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Jan 13, 2017

Contributor

Hi,

Could this issue be reopened, please?

Contributor

nbarrientos commented Jan 13, 2017

Hi,

Could this issue be reopened, please?

@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos
Contributor

nbarrientos commented Feb 3, 2017

nbarrientos added a commit to nbarrientos/jruby that referenced this issue Feb 19, 2017

Use ipaddr to compare the sender's address
When resolving names using compressed IPv6-only DNS servers, it could
happen that when processing the response of a name query, the IP of the
sender is not compressed so using a simple string comparison is not
safe.

This patch encapsulates the from address and the address of the sender in
IPAddr objects so the comparison is safe and just works in all cases.

This fixes #3663.
@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos
Contributor

nbarrientos commented Feb 19, 2017

@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Feb 19, 2017

Contributor

And yes, this is still an issue in jRuby 9.1.7.0.

# grep nameserver /etc/resolv.conf 
nameserver 2001:1458:201:1100::5
nameserver 2001:1458:201:1100::5
# jruby -v
jruby 9.1.7.0 (2.3.1) 2017-01-11 68056ae OpenJDK 64-Bit Server VM 25.121-b13 on 1.8.0_121-b13 +jit [linux-x86_64]
# time jruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
Resolv::ResolvError: no address for web.cern.ch
  getaddress at /root/jruby-9.1.7.0/lib/ruby/stdlib/resolv.rb:100
  getaddress at /root/jruby-9.1.7.0/lib/ruby/stdlib/resolv.rb:50
      <main> at -e:1

real	2m44.461s
user	0m5.107s
sys	0m0.272s
# PATH=jruby-9.1.7.0-patched/bin/:$PATH
## jruby-9.1.7.0-patched contains
## https://patch-diff.githubusercontent.com/raw/jruby/jruby/pull/4496.patch
# jruby -v
jruby 9.1.7.0 (2.3.1) 2017-02-19 fffffff OpenJDK 64-Bit Server VM 25.121-b13 on 1.8.0_121-b13 +jit [linux-x86_64]
# time jruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
188.184.9.235

real	0m4.568s
user	0m4.309s
sys	0m0.212s
Contributor

nbarrientos commented Feb 19, 2017

And yes, this is still an issue in jRuby 9.1.7.0.

# grep nameserver /etc/resolv.conf 
nameserver 2001:1458:201:1100::5
nameserver 2001:1458:201:1100::5
# jruby -v
jruby 9.1.7.0 (2.3.1) 2017-01-11 68056ae OpenJDK 64-Bit Server VM 25.121-b13 on 1.8.0_121-b13 +jit [linux-x86_64]
# time jruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
Resolv::ResolvError: no address for web.cern.ch
  getaddress at /root/jruby-9.1.7.0/lib/ruby/stdlib/resolv.rb:100
  getaddress at /root/jruby-9.1.7.0/lib/ruby/stdlib/resolv.rb:50
      <main> at -e:1

real	2m44.461s
user	0m5.107s
sys	0m0.272s
# PATH=jruby-9.1.7.0-patched/bin/:$PATH
## jruby-9.1.7.0-patched contains
## https://patch-diff.githubusercontent.com/raw/jruby/jruby/pull/4496.patch
# jruby -v
jruby 9.1.7.0 (2.3.1) 2017-02-19 fffffff OpenJDK 64-Bit Server VM 25.121-b13 on 1.8.0_121-b13 +jit [linux-x86_64]
# time jruby -e "require 'resolv'; puts Resolv.getaddress 'web.cern.ch'"
188.184.9.235

real	0m4.568s
user	0m4.309s
sys	0m0.212s
@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Feb 23, 2017

Member

I commented on #4496 but I'll mention again here: I'd be more comfortable patching standard library if we could prove MRI exhibits the same problem.

I also apologize for missing all your activity! Much of your self-conversation here took place while @enebo and I were at FOSDEM and not monitoring issues as closely. We'll get this wrapped up for 9.1.8.0.

Member

headius commented Feb 23, 2017

I commented on #4496 but I'll mention again here: I'd be more comfortable patching standard library if we could prove MRI exhibits the same problem.

I also apologize for missing all your activity! Much of your self-conversation here took place while @enebo and I were at FOSDEM and not monitoring issues as closely. We'll get this wrapped up for 9.1.8.0.

@nbarrientos

This comment has been minimized.

Show comment
Hide comment
@nbarrientos

nbarrientos Feb 23, 2017

Contributor

No problem, thanks anyway for your time 😃

I've just replied on the merge request.

Contributor

nbarrientos commented Feb 23, 2017

No problem, thanks anyway for your time 😃

I've just replied on the merge request.

@headius headius closed this in #4496 Mar 1, 2017

headius added a commit to jruby/ruby that referenced this issue Mar 14, 2017

See jruby/jruby#3663 and jruby/jruby@d1a760e.
Use ipaddr to compare the sender's address
When resolving names using compressed IPv6-only DNS servers, it could
happen that when processing the response of a name query, the IP of the
sender is not compressed so using a simple string comparison is not
safe.

This patch encapsulates the from address and the address of the sender in
IPAddr objects so the comparison is safe and just works in all cases.

This fixes jruby/jruby#3663.

headius added a commit to jruby/ruby that referenced this issue Mar 14, 2017

See jruby/jruby#3663 and jruby/jruby@d1a760e.
Use ipaddr to compare the sender's address
When resolving names using compressed IPv6-only DNS servers, it could
happen that when processing the response of a name query, the IP of the
sender is not compressed so using a simple string comparison is not
safe.

This patch encapsulates the from address and the address of the sender in
IPAddr objects so the comparison is safe and just works in all cases.

This fixes jruby/jruby#3663.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment