Make RDoc::Markup::ToHtml#parseable? use a real parser

[jeremyevans]

Currently, parseable? uses a very basic regexp to guess if text is ruby or not (so that it can be syntax highlighted). See https://github.com/rdoc/rdoc/blob/0494067cffba009e3dc9a3f4545980d5b61700bd/lib/rdoc/markup/to_html.rb#L385. This misses a lot of valid ruby code, and it also syntax highlights some invalid ruby code (making it harder to spot invalid code examples in the documentation).

Would you be open to using a real parser (if available) to determine whether or not text should be considered ruby? Here's what I'm using in hanna-nouveau as a monkey patch using the parser gem:

begin
  require 'parser/current'

  class RDoc::Markup::ToHtml
    def parseable? text
      parser = Parser::CurrentRuby.new
      parser.diagnostics.consumer = lambda{|d|}
      buffer = Parser::Source::Buffer.new('(string)')
      buffer.source = text
      parser.parse(buffer)
      true
    rescue
      false
    end
  end
rescue LoadError
end

This change increases the time for rake rdoc in Sequel from 127 seconds to 140 seconds, or about 10%, which seems reasonable considering the advantages.

[drbrain]

RDoc can't have any dependencies, but using ruby -c should be OK, except maybe for ruby.

Will Ripper work? Last I checked I couldn't get it to reject garbage files.

[jeremyevans]

Ripper.sexp appears to return nil when passed invalid syntax and non-nil when passed valid syntax, but I didn't do a comprehensive check. It's unlikely to be worse than the current parse guess, though. Ripper isn't available in 1.8, but I'm not sure how much RDoc cares about 1.8. We could always fallback to using the current parsing on 1.8 until 1.8 support is dropped.

[jeremyevans]

Doing some more testing with Ripper and real world data, it does seem way to liberal, returning non-nil for invalid syntax. I'll see if I can get a ruby -c based approach working.

[jeremyevans]

This works on ruby, jruby, and rubinius, but is too slow to be practical, increasing documentation build time one of my projects by about 7x on ruby, and probably by much, much more when using jruby or rubinius:

require 'open3'
def parseable? text
  ruby = File.join(*RbConfig::CONFIG.values_at("bindir", "RUBY_INSTALL_NAME"))
  pid = Open3.popen3(ruby, '-c') do |stdin, stdout, stderr, wait_thr|
    stdin.write(text)
    stdin.close
    stdout.close
    stderr.close
    wait_thr.value.exitstatus == 0
  end
end

I don't think using ruby -c is going to be a workable approach for performance reasons. We have to have something that executes in process in order to get decent performance. I'll do some research and see if there are any other options.

[jeremyevans]

This appears to work with all versions of ruby and is very fast:

def parseable? text
  eval("BEGIN {return true}\n#{text}")
rescue SyntaxError
  false
end

I'll send a pull request for it.