Skip to content

Loading…

NoMethodError while running RubyDNS #20

Closed
Hengjie opened this Issue · 19 comments

2 participants

@Hengjie

Hi there,

I'm currently using RubyDNS on Heroku and I sometimes see this error:

/app/vendor/ruby-1.9.3/lib/ruby/1.9.1/resolv.rb:1510:in `get_labels': undefined method `ord' for nil:NilClass (NoMethodError) 
/app/vendor/ruby-1.9.3/lib/ruby/1.9.1/resolv.rb:1503:in `get_name' 
/app/vendor/ruby-1.9.3/lib/ruby/1.9.1/resolv.rb:1542:in `get_rr' 
/app/vendor/ruby-1.9.3/lib/ruby/1.9.1/resolv.rb:1418:in `block (2 levels) in decode' 
/app/vendor/ruby-1.9.3/lib/ruby/1.9.1/resolv.rb:1417:in `each' 
/app/vendor/ruby-1.9.3/lib/ruby/1.9.1/resolv.rb:1417:in `block in decode' 
/app/vendor/ruby-1.9.3/lib/ruby/1.9.1/resolv.rb:1438:in `initialize' 
/app/vendor/ruby-1.9.3/lib/ruby/1.9.1/resolv.rb:1402:in `new' 
/app/vendor/ruby-1.9.3/lib/ruby/1.9.1/resolv.rb:1402:in `decode' 
/app/vendor/bundle/ruby/1.9.1/gems/rubydns-0.6.5/lib/rubydns/message.rb:39:in `decode_message' 
/app/vendor/bundle/ruby/1.9.1/gems/rubydns-0.6.5/lib/rubydns/resolver.rb:191:in `receive_data' 

I'm not exactly sure why this happens. I don't know which nameserver or domain name is causing it yet (I'll be finding out shortly). But some insight how we could fix it would be appreciated.

Thanks for making rubydns!

@ioquatix
Owner

Hi

Looks like a bug in the message that is being received.

It might be corruption, or something else going on.

Can you check resolv.rb to find out what is nil and what it probably should be?

Resolv.rb is out of my control unfortunately, so it might be a bug in Ruby. But, I guess it is probably an issue in the incoming data. Is it UDP or TCP?

Kind regards,
Samuel

@Hengjie

The data is UDP. The relevant block of code is (line 191 is message = RubyDNS::decode_message(data)):

def receive_data(data)
  # Receiving response from remote DNS server...
  message = RubyDNS::decode_message(data)

  # The message id must match, and it can't be truncated:
  @request.process_response!(message)
rescue Resolv::DNS::DecodeError => error
  @request.process_response!(error)
end

https://github.com/ioquatix/rubydns/blob/master/lib/rubydns/resolver.rb#L191

I guess we could rescue it if we find that it's something that we can't fix. I'm rerunning my app now with a patch that should give me context (nameserver and domain name) as to what inputs may have caused it. Hopefully it's something that's repeatable.

Thanks for the quick reply Samuel.

@ioquatix
Owner

The error is deep in the bowels of resolve.rb as your backtrace shows. resolve.rb is standard part of Ruby. Probably it shouldn't throw an exception like that, but if the message is corrupt it might hit some weird edge case that wasn't considered by the resolve.rb developers. You should try to run pcap and debug the offending packet which is probably corrupt or malformed some how.

@Hengjie

Ah my bad, I thought you meant look at rubydns code. Yes, it looks like it's something in resolve.rb. I'll investigate further.

@ioquatix
Owner

A few bugs have been resolved in the latest release 0.6.6 - I don't know if these are relating to the issues you are having, but please feel free to cautiously try the new release.

@Hengjie
@ioquatix
Owner

Can you please give me an update on this issue? Thanks.

@Hengjie

I'm giving it a try now, should have results in about 24 hours.

@Hengjie

No it doesn't appear to resolve it. I had another look into my logs and it turns out everything ending with .255 fails.

e.g.

Oct 17 18:42:39 worker-2 app/worker.1:  NoMethodError on 190.153.142.255: undefined method `ord' for nil:NilClass 
Oct 17 20:41:17 worker-2 app/worker.1:  NoMethodError on 218.77.188.255: undefined method `ord' for nil:NilClass 
Oct 17 21:00:58 worker-2 app/worker.1:  NoMethodError on 222.211.159.255: undefined method `ord' for nil:NilClass 
@ioquatix
Owner

Can you please paste in a copy of the get_labels function, I don't have that specific version (running 2.x).

@Hengjie

Sorry, where (filepath) is get_labels located?

@ioquatix
Owner

/app/vendor/ruby-1.9.3/lib/ruby/1.9.1/resolv.rb:1510?

@Hengjie
        def get_labels(limit=nil)
          limit = @index if !limit || @index < limit
          d = []
          while true
            case @data[@index].ord
            when 0
              @index += 1
              return d
            when 192..255
              idx = self.get_unpack('n')[0] & 0x3fff
              if limit <= idx
                raise DecodeError.new("non-backward name pointer")
              end
              save_index = @index
              @index = idx
              d += self.get_labels(limit)
              @index = save_index
              return d
            else
              d << self.get_label
            end
          end
          return d
        end
@ioquatix
Owner

I've added a message dumping facility:

RubyDNS::log_bad_messages!("bad.log")

Call that before calling start_server and the bad message will be logged to that path. Don't run it for more than a few hours, try to collect some samples.

That is in the latest version of RubyDNS 0.6.7

@ioquatix
Owner

Did you get a chance to dump some bad messages?

@Hengjie

Sorry, no I haven't. I'm running it on Heroku so it's a bit hard to log the messages into a ephemeral storage. Is it possible for it to just send to std out?

@ioquatix
Owner

Pass $stdout to log_bad_messages!

@ioquatix
Owner

@Hengjie there was a patch recently applied that catches all StandardErrors and converts them into decode errors. This will likely affect the nature of the issues you are experiencing. Did you make any progress on getting copies of the bad messages?

@ioquatix ioquatix was assigned
@ioquatix
Owner

Closing due to inactivity, latest version should have resolved this issue.

@ioquatix ioquatix closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.